Apr 17 15:19:38 user cinder-volume[84596]: DEBUG oslo_concurrency.lockutils [-] Acquiring lock "singleton_lock" {{(pid=84596) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:312}} Apr 17 15:19:38 user cinder-volume[84596]: DEBUG oslo_concurrency.lockutils [-] Acquired lock "singleton_lock" {{(pid=84596) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:315}} Apr 17 15:19:38 user cinder-volume[84596]: DEBUG oslo_concurrency.lockutils [-] Releasing lock "singleton_lock" {{(pid=84596) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:333}} Apr 17 15:19:38 user cinder-volume[84596]: INFO dbcounter [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] Registered counter for database cinder Apr 17 15:19:38 user cinder-volume[84596]: DEBUG oslo_db.sqlalchemy.engines [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84596) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 15:19:38 user cinder-volume[84596]: DEBUG dbcounter [-] [84596] Writer thread running {{(pid=84596) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:102}} Apr 17 15:19:38 user cinder-volume[84596]: INFO cinder.rpc [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] Automatically selected cinder-scheduler objects version 1.39 as minimum service version. Apr 17 15:19:38 user cinder-volume[84596]: INFO cinder.rpc [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] Automatically selected cinder-scheduler RPC version 3.12 as minimum service version. Apr 17 15:19:38 user cinder-volume[84596]: DEBUG oslo_db.sqlalchemy.engines [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84596) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 15:19:39 user cinder-volume[84596]: INFO cinder.volume.manager [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] Determined volume DB was empty at startup. Apr 17 15:19:39 user cinder-volume[84596]: DEBUG cinder.volume.manager [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] Cinder Volume DB check: vol_db_empty=True {{(pid=84596) __init__ /opt/stack/cinder/cinder/volume/manager.py:290}} Apr 17 15:19:39 user cinder-volume[84596]: INFO cinder.volume.manager [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] Service not found for updating active_backend_id, assuming default for driver init. Apr 17 15:19:39 user cinder-volume[84596]: WARNING stevedore.named [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] Could not load cinder.keymgr.conf_key_mgr.ConfKeyManager Apr 17 15:19:39 user cinder-volume[84596]: WARNING castellan.key_manager [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] Deprecation Warning : cinder.keymgr.conf_key_mgr.ConfKeyManager is not a stevedore based driver, trying to load it as a class: stevedore.exception.NoMatches: No 'castellan.drivers' driver found, looking for 'cinder.keymgr.conf_key_mgr.ConfKeyManager' Apr 17 15:19:39 user cinder-volume[84596]: WARNING cinder.keymgr.conf_key_mgr [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] This key manager is insecure and is not recommended for production deployments: stevedore.exception.NoMatches: No 'castellan.drivers' driver found, looking for 'cinder.keymgr.conf_key_mgr.ConfKeyManager' Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_db.sqlalchemy.engines [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84596) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 15:19:39 user cinder-volume[84596]: INFO cinder.rpc [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] Automatically selected cinder-volume objects version 1.39 as minimum service version. Apr 17 15:19:39 user cinder-volume[84596]: INFO cinder.rpc [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] Automatically selected cinder-volume RPC version 3.18 as minimum service version. Apr 17 15:19:39 user cinder-volume[84596]: WARNING stevedore.named [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] Could not load cinder.keymgr.conf_key_mgr.ConfKeyManager Apr 17 15:19:39 user cinder-volume[84596]: WARNING castellan.key_manager [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] Deprecation Warning : cinder.keymgr.conf_key_mgr.ConfKeyManager is not a stevedore based driver, trying to load it as a class: stevedore.exception.NoMatches: No 'castellan.drivers' driver found, looking for 'cinder.keymgr.conf_key_mgr.ConfKeyManager' Apr 17 15:19:39 user cinder-volume[84596]: INFO cinder.volume.manager [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] Image-volume cache enabled for host user@joviandss-2. Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_db.sqlalchemy.engines [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84596) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 15:19:39 user cinder-volume[84596]: INFO oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] Starting 1 workers Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] Started child 84696 {{(pid=84596) _start_child /usr/local/lib/python3.10/dist-packages/oslo_service/service.py:575}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] Full set of CONF: {{(pid=84596) wait /usr/local/lib/python3.10/dist-packages/oslo_service/service.py:649}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] ******************************************************************************** {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2589}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] Configuration options gathered from: {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2590}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] command line args: ['--config-file', '/etc/cinder/cinder.conf'] {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2591}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] config files: ['/etc/cinder/cinder.conf'] {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2592}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] ================================================================================ {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2594}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] allow_availability_zone_fallback = False {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] allow_compression_on_image_upload = False {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] allowed_direct_url_schemes = [] {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] api_rate_limit = True {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] auth_strategy = keystone {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] az_cache_duration = 3600 {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] backend_availability_zone = None {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84696]: INFO cinder.service [-] Starting cinder-volume node (version 22.1.0) Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] backend_host = None {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] backend_name = None {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] backend_stats_polling_interval = 60 {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] backup_api_class = cinder.backup.api.API {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] backup_manager = cinder.backup.manager.BackupManager {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] backup_name_template = backup-%s {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] backup_use_same_host = False {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] backup_use_temp_snapshot = False {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] chap_password = **** {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] chap_password_len = 12 {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] chap_username = {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] chiscsi_conf = /etc/chelsio-iscsi/chiscsi.conf {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] cinder_internal_tenant_project_id = 2ef07a6150b7438d850c124674d4e245 {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] cinder_internal_tenant_user_id = 7e22f9ab9d1c42a29166bf831687159a {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] cloned_volume_same_az = True {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] cluster = None {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] compression_format = gzip {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] compute_api_class = cinder.compute.nova.API {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] config_dir = [] {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] config_file = ['/etc/cinder/cinder.conf'] {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] config_source = [] {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] consistencygroup_api_class = cinder.consistencygroup.api.API {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] control_exchange = openstack {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] debug = True {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] default_availability_zone = None {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] default_group_type = None {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] default_log_levels = ['amqp=WARN', 'amqplib=WARN', 'boto=WARN', 'qpid=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO'] {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] default_volume_type = joviandss-2 {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] driver_client_cert = None {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] driver_client_cert_key = None {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] driver_data_namespace = None {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] driver_ssl_cert_path = None {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] driver_ssl_cert_verify = False {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] driver_use_ssl = False {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] enable_force_upload = False {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] enable_new_services = True {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] enable_unsupported_driver = False {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] enabled_backends = ['joviandss-2'] {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] enforce_multipath_for_image_xfer = False {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] filter_function = None {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] glance_api_insecure = False {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] glance_api_servers = ['http://10.0.0.210/image'] {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] glance_api_ssl_compression = False {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] glance_ca_certificates_file = None {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] glance_catalog_info = image:glance:publicURL {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] glance_certfile = None {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] glance_core_properties = ['checksum', 'container_format', 'disk_format', 'image_name', 'image_id', 'min_disk', 'min_ram', 'name', 'size'] {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] glance_keyfile = None {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] glance_num_retries = 3 {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] glance_request_timeout = None {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] goodness_function = None {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] graceful_shutdown_timeout = 5 {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] group_api_class = cinder.group.api.API {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] host = user {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] iet_conf = /etc/iet/ietd.conf {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] image_compress_on_upload = True {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] image_conversion_address_space_limit = 1 {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] image_conversion_cpu_limit = 60 {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] image_conversion_dir = /opt/stack/data/cinder/conversion {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] image_conversion_disable = False {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] image_upload_use_cinder_backend = False {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] image_upload_use_internal_tenant = False {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] image_volume_cache_enabled = False {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] image_volume_cache_max_count = 0 {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] image_volume_cache_max_size_gb = 0 {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] init_host_max_objects_retrieval = 0 {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] instance_format = [instance: %(uuid)s] {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] instance_uuid_format = [instance: %(uuid)s] {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] iscsi_iotype = fileio {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] iscsi_target_flags = {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] iscsi_write_cache = on {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] iser_helper = tgtadm {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] iser_ip_address = 10.0.0.210 {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] iser_port = 3260 {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] iser_target_prefix = iqn.2010-10.org.openstack: {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] jovian_block_size = 64K {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] jovian_ignore_tpath = [] {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] jovian_pool = Pool-0 {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] jovian_recovery_delay = 60 {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] log_config_append = None {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] log_date_format = %Y-%m-%d %H:%M:%S {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] log_dir = None {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] log_file = None {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] log_options = True {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] log_rotate_interval = 1 {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] log_rotate_interval_type = days {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] log_rotation_type = none {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}} {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] logging_exception_prefix = ERROR %(name)s %(instance)s {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] logging_user_identity_format = %(user)s %(project)s %(domain)s %(system_scope)s %(user_domain)s %(project_domain)s {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] max_age = 0 {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] max_logfile_count = 30 {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] max_logfile_size_mb = 200 {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] max_over_subscription_ratio = 20.0 {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] message_reap_interval = 86400 {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] message_ttl = 2592000 {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] migration_create_volume_timeout_secs = 300 {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] monkey_patch = False {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] monkey_patch_modules = [] {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] my_ip = 10.0.0.210 {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] no_snapshot_gb_quota = False {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] num_iser_scan_tries = 3 {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] num_shell_tries = 3 {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] num_volume_device_scan_tries = 3 {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] nvmeof_conn_info_version = 1 {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] nvmet_ns_id = 10 {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] nvmet_port_id = 1 {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] osapi_max_limit = 1000 {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] osapi_volume_ext_list = [] {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] osapi_volume_extension = ['cinder.api.contrib.standard_extensions'] {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] osapi_volume_listen = 0.0.0.0 {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] osapi_volume_listen_port = 8776 {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] osapi_volume_use_ssl = False {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] osapi_volume_workers = 3 {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84696]: INFO cinder.volume.manager [None req-56c536ad-a672-42a0-ad8f-80bf0f9c4548 None None] Starting volume driver JovianISCSIDriver (1.0.2) Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] per_volume_size_limit = -1 {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] periodic_fuzzy_delay = 60 {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] periodic_interval = 60 {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] project_id_regex = [0-9a-f\-]+ {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] public_endpoint = None {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] publish_errors = False {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] quota_backup_gigabytes = 1000 {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] quota_backups = 10 {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] quota_consistencygroups = 10 {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] quota_driver = cinder.quota.DbQuotaDriver {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] quota_gigabytes = 1000 {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] quota_groups = 10 {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] quota_snapshots = 10 {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] quota_volumes = 10 {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] rate_limit_burst = 0 {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] rate_limit_except_level = CRITICAL {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] rate_limit_interval = 0 {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-56c536ad-a672-42a0-ad8f-80bf0f9c4548 None None] check pool {{(pid=84696) is_pool_exists /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:71}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] reinit_driver_count = 3 {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] replication_device = **** {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-56c536ad-a672-42a0-ad8f-80bf0f9c4548 None None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] report_discard_supported = False {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-56c536ad-a672-42a0-ad8f-80bf0f9c4548 None None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] report_interval = 10 {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] reservation_clean_interval = 86400 {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] reservation_expire = 86400 {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] reserved_image_namespaces = [] {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] reserved_percentage = 0 {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] resource_query_filters_file = /etc/cinder/resource_filters.json {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] rootwrap_config = /etc/cinder/rootwrap.conf {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] rpc_response_timeout = 120 {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] run_external_periodic_tasks = True {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] san_hosts = [] {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] scheduler_manager = cinder.scheduler.manager.SchedulerManager {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] scst_target_driver = iscsi {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] scst_target_iqn_name = None {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] service_down_time = 60 {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] snapshot_name_template = snapshot-%s {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] snapshot_same_host = True {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] split_loggers = False {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] ssh_hosts_key_file = /opt/stack/data/cinder/ssh_known_hosts {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] state_path = /opt/stack/data/cinder {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] storage_availability_zone = nova {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] storage_protocol = iSCSI {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] strict_ssh_host_key_policy = False {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] syslog_log_facility = LOG_USER {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] target_helper = lioadm {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] target_ip_address = 10.0.0.210 {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] target_port = 3260 {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] target_prefix = iqn.2010-10.org.openstack: {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] target_protocol = iscsi {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] target_secondary_ip_addresses = [] {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] trace_flags = None {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] transfer_api_class = cinder.transfer.api.API {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] transport_url = **** {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] until_refresh = 0 {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] use_chap_auth = False {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] use_default_quota_class = True {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] use_eventlog = False {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] use_journal = False {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] use_json = False {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] use_multipath_for_image_xfer = False {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] use_stderr = False {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] use_syslog = False {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] verify_glance_signatures = enabled {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] vmdk_allowed_types = ['streamOptimized', 'monolithicSparse'] {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] volume_api_class = cinder.volume.api.API {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] volume_backend_name = None {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] volume_clear = zero {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] volume_clear_ionice = None {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] volume_clear_size = 0 {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] volume_copy_blkio_cgroup_name = cinder-volume-copy {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] volume_copy_bps_limit = 0 {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] volume_dd_blocksize = 1M {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] volume_manager = cinder.volume.manager.VolumeManager {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] volume_name_template = volume-%s {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] volume_service_inithost_offload = False {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] volume_usage_audit_period = month {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] volumes_dir = /opt/stack/data/cinder/volumes {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] watch_log_file = False {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] zoning_mode = None {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2602}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] os_brick.lock_path = /opt/stack/data/cinder {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] oslo_middleware.http_basic_auth_user_file = /etc/htpasswd {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] oslo_versionedobjects.fatal_exception_format_errors = False {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] oslo_concurrency.disable_process_locking = False {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] oslo_concurrency.lock_path = /opt/stack/data/cinder {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] oslo_messaging_metrics.metrics_buffer_size = 1000 {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] oslo_messaging_metrics.metrics_enabled = False {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] oslo_messaging_metrics.metrics_process_name = {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] oslo_messaging_metrics.metrics_socket_file = /var/tmp/metrics_collector.sock {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] oslo_messaging_metrics.metrics_thread_stop_timeout = 10 {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] keystone_authtoken.admin_password = **** {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] keystone_authtoken.admin_tenant_name = admin {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] keystone_authtoken.admin_token = **** {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] keystone_authtoken.admin_user = None {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] keystone_authtoken.auth_admin_prefix = {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] keystone_authtoken.auth_host = 127.0.0.1 {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] keystone_authtoken.auth_port = 35357 {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] keystone_authtoken.auth_protocol = https {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] keystone_authtoken.auth_section = None {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] keystone_authtoken.auth_type = password {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] keystone_authtoken.auth_uri = None {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] keystone_authtoken.auth_version = None {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] keystone_authtoken.cache = None {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] keystone_authtoken.cafile = /opt/stack/data/ca-bundle.pem {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] keystone_authtoken.certfile = None {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] keystone_authtoken.delay_auth_decision = False {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] keystone_authtoken.enforce_token_bind = permissive {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] keystone_authtoken.http_connect_timeout = None {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] keystone_authtoken.http_request_max_retries = 3 {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] keystone_authtoken.identity_uri = None {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] keystone_authtoken.include_service_catalog = True {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] keystone_authtoken.insecure = False {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] keystone_authtoken.interface = public {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] keystone_authtoken.keyfile = None {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] keystone_authtoken.memcache_pool_conn_get_timeout = 10 {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] keystone_authtoken.memcache_pool_dead_retry = 300 {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] keystone_authtoken.memcache_pool_maxsize = 10 {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] keystone_authtoken.memcache_pool_socket_timeout = 3 {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] keystone_authtoken.memcache_pool_unused_timeout = 60 {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] keystone_authtoken.memcache_secret_key = **** {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] keystone_authtoken.memcache_security_strategy = None {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] keystone_authtoken.memcache_use_advanced_pool = True {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] keystone_authtoken.memcached_servers = ['localhost:11211'] {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] keystone_authtoken.region_name = None {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] keystone_authtoken.service_token_roles = ['service'] {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] keystone_authtoken.service_token_roles_required = False {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] keystone_authtoken.service_type = None {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] keystone_authtoken.token_cache_time = 300 {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] keystone_authtoken.www_authenticate_uri = None {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] cache.backend = dogpile.cache.null {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] cache.backend_argument = **** {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] cache.config_prefix = cache.oslo {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] cache.dead_timeout = 60.0 {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] cache.debug_cache_backend = False {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] cache.enable_retry_client = False {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] cache.enable_socket_keepalive = False {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] cache.enabled = False {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] cache.expiration_time = 600 {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] cache.hashclient_retry_attempts = 2 {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] cache.hashclient_retry_delay = 1.0 {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] cache.memcache_dead_retry = 300 {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] cache.memcache_password = {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] cache.memcache_pool_connection_get_timeout = 10 {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] cache.memcache_pool_flush_on_reconnect = False {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] cache.memcache_pool_maxsize = 10 {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] cache.memcache_pool_unused_timeout = 60 {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] cache.memcache_sasl_enabled = False {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] cache.memcache_servers = ['localhost:11211'] {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] cache.memcache_socket_timeout = 1.0 {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] cache.memcache_username = {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] cache.proxies = [] {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] cache.retry_attempts = 2 {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] cache.retry_delay = 0.0 {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] cache.socket_keepalive_count = 1 {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] cache.socket_keepalive_idle = 1 {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] cache.socket_keepalive_interval = 1 {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] cache.tls_allowed_ciphers = None {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] cache.tls_cafile = None {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] cache.tls_certfile = None {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] cache.tls_enabled = False {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] cache.tls_keyfile = None {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] coordination.backend_url = etcd3+http://10.0.0.210:2379 {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] database.backend = sqlalchemy {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] database.connection = **** {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] database.connection_debug = 0 {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] database.connection_parameters = {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] database.connection_recycle_time = 3600 {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] database.connection_trace = False {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] database.db_inc_retry_interval = True {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] database.db_max_retries = 20 {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] database.db_max_retry_interval = 10 {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] database.db_retry_interval = 1 {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] database.max_overflow = 50 {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] database.max_pool_size = 5 {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] database.max_retries = 10 {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] database.mysql_enable_ndb = False {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] database.mysql_sql_mode = TRADITIONAL {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] database.mysql_wsrep_sync_wait = None {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] database.pool_timeout = None {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] database.retry_interval = 10 {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] database.slave_connection = **** {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] database.sqlite_synchronous = True {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] database.use_db_reconnect = False {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] oslo_policy.enforce_new_defaults = False {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] oslo_policy.enforce_scope = False {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] oslo_policy.policy_default_rule = default {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] oslo_policy.policy_dirs = ['policy.d'] {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] oslo_policy.policy_file = policy.yaml {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] oslo_policy.remote_content_type = application/x-www-form-urlencoded {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] oslo_policy.remote_ssl_ca_crt_file = None {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] oslo_policy.remote_ssl_client_crt_file = None {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] oslo_policy.remote_ssl_client_key_file = None {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] oslo_policy.remote_ssl_verify_server_crt = False {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] profiler.connection_string = messaging:// {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] profiler.enabled = False {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] profiler.es_doc_type = notification {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] profiler.es_scroll_size = 10000 {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] profiler.es_scroll_time = 2m {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] profiler.filter_error_trace = False {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] profiler.hmac_keys = SECRET_KEY {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] profiler.sentinel_service_name = mymaster {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] profiler.socket_timeout = 0.1 {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] profiler.trace_sqlalchemy = False {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] privsep_osbrick.capabilities = [21] {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] privsep_osbrick.group = None {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] privsep_osbrick.helper_command = None {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] privsep_osbrick.logger_name = os_brick.privileged {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] privsep_osbrick.thread_pool_size = 12 {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] privsep_osbrick.user = None {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] cinder_sys_admin.capabilities = [0, 1, 2, 3, 12, 21] {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] cinder_sys_admin.group = None {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] cinder_sys_admin.helper_command = None {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] cinder_sys_admin.logger_name = oslo_privsep.daemon {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] cinder_sys_admin.thread_pool_size = 12 {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] cinder_sys_admin.user = None {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] service_user.auth_section = None {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] service_user.auth_type = None {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] service_user.cafile = None {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] service_user.certfile = None {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] service_user.collect_timing = False {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] service_user.insecure = False {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] service_user.keyfile = None {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] service_user.send_service_user_token = False {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] service_user.split_loggers = False {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] service_user.timeout = None {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] backend_defaults.backend_availability_zone = None {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] backend_defaults.backend_native_threads_pool_size = 20 {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] backend_defaults.backup_use_temp_snapshot = False {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] backend_defaults.chap_password = **** {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] backend_defaults.chap_username = {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] backend_defaults.chiscsi_conf = /etc/chelsio-iscsi/chiscsi.conf {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] backend_defaults.driver_client_cert = None {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] backend_defaults.driver_client_cert_key = None {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] backend_defaults.driver_data_namespace = None {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] backend_defaults.driver_ssl_cert_path = None {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] backend_defaults.driver_ssl_cert_verify = False {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] backend_defaults.driver_use_ssl = False {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] backend_defaults.enable_unsupported_driver = False {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] backend_defaults.enforce_multipath_for_image_xfer = False {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] backend_defaults.extra_capabilities = {} {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] backend_defaults.filter_function = None {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] backend_defaults.goodness_function = None {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] backend_defaults.iet_conf = /etc/iet/ietd.conf {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] backend_defaults.image_upload_use_cinder_backend = False {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] backend_defaults.image_upload_use_internal_tenant = False {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] backend_defaults.image_volume_cache_enabled = False {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] backend_defaults.image_volume_cache_max_count = 0 {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] backend_defaults.image_volume_cache_max_size_gb = 0 {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] backend_defaults.iscsi_iotype = fileio {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] backend_defaults.iscsi_target_flags = {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] backend_defaults.iscsi_write_cache = on {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] backend_defaults.iser_helper = tgtadm {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] backend_defaults.iser_ip_address = 10.0.0.210 {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] backend_defaults.iser_port = 3260 {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] backend_defaults.iser_target_prefix = iqn.2010-10.org.openstack: {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] backend_defaults.max_over_subscription_ratio = 20.0 {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] backend_defaults.num_iser_scan_tries = 3 {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] backend_defaults.num_shell_tries = 3 {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] backend_defaults.num_volume_device_scan_tries = 3 {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] backend_defaults.nvmeof_conn_info_version = 1 {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] backend_defaults.nvmet_ns_id = 10 {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] backend_defaults.nvmet_port_id = 1 {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] backend_defaults.replication_device = **** {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] backend_defaults.report_discard_supported = False {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] backend_defaults.reserved_percentage = 0 {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] backend_defaults.san_api_port = None {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] backend_defaults.san_clustername = {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] backend_defaults.san_ip = {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] backend_defaults.san_is_local = False {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] backend_defaults.san_login = admin {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] backend_defaults.san_password = **** {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] backend_defaults.san_private_key = {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] backend_defaults.san_ssh_port = 22 {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] backend_defaults.san_thin_provision = True {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] backend_defaults.scst_target_driver = iscsi {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] backend_defaults.scst_target_iqn_name = None {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] backend_defaults.ssh_conn_timeout = 30 {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] backend_defaults.ssh_max_pool_conn = 5 {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] backend_defaults.ssh_min_pool_conn = 1 {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] backend_defaults.storage_protocol = iSCSI {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] backend_defaults.suppress_requests_ssl_warnings = False {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] backend_defaults.target_helper = tgtadm {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] backend_defaults.target_ip_address = 10.0.0.210 {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] backend_defaults.target_port = 3260 {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] backend_defaults.target_prefix = iqn.2010-10.org.openstack: {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] backend_defaults.target_protocol = iscsi {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] backend_defaults.target_secondary_ip_addresses = [] {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] backend_defaults.trace_flags = None {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] backend_defaults.unique_fqdn_network = True {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] backend_defaults.use_chap_auth = False {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] backend_defaults.use_multipath_for_image_xfer = False {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] backend_defaults.volume_backend_name = None {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] backend_defaults.volume_clear = zero {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] backend_defaults.volume_clear_ionice = None {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] backend_defaults.volume_clear_size = 0 {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] backend_defaults.volume_copy_blkio_cgroup_name = cinder-volume-copy {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] backend_defaults.volume_copy_bps_limit = 0 {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] backend_defaults.volume_dd_blocksize = 1M {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] backend_defaults.volume_driver = cinder.volume.drivers.lvm.LVMVolumeDriver {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] backend_defaults.volumes_dir = /opt/stack/data/cinder/volumes {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] oslo_reports.file_event_handler = None {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] oslo_reports.file_event_handler_interval = 1 {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] oslo_reports.log_dir = None {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] joviandss-2.backend_availability_zone = None {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] joviandss-2.backend_host = None {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] joviandss-2.backend_native_threads_pool_size = None {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] joviandss-2.backup_use_temp_snapshot = None {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] joviandss-2.chap_password = **** {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] joviandss-2.chap_username = None {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] joviandss-2.chiscsi_conf = None {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] joviandss-2.driver_client_cert = None {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] joviandss-2.driver_client_cert_key = None {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] joviandss-2.driver_data_namespace = None {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] joviandss-2.driver_ssl_cert_path = None {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] joviandss-2.driver_ssl_cert_verify = False {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] joviandss-2.driver_use_ssl = True {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] joviandss-2.enable_unsupported_driver = None {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] joviandss-2.enforce_multipath_for_image_xfer = None {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] joviandss-2.extra_capabilities = None {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] joviandss-2.filter_function = None {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] joviandss-2.goodness_function = None {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] joviandss-2.iet_conf = None {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] joviandss-2.image_upload_use_cinder_backend = None {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] joviandss-2.image_upload_use_internal_tenant = None {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] joviandss-2.image_volume_cache_enabled = True {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] joviandss-2.image_volume_cache_max_count = None {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] joviandss-2.image_volume_cache_max_size_gb = None {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] joviandss-2.iscsi_iotype = None {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] joviandss-2.iscsi_target_flags = None {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] joviandss-2.iscsi_write_cache = None {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] joviandss-2.iser_helper = None {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] joviandss-2.iser_ip_address = None {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] joviandss-2.iser_port = None {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] joviandss-2.iser_target_prefix = None {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] joviandss-2.max_over_subscription_ratio = None {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] joviandss-2.num_iser_scan_tries = None {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] joviandss-2.num_shell_tries = None {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] joviandss-2.num_volume_device_scan_tries = None {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] joviandss-2.nvmeof_conn_info_version = None {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] joviandss-2.nvmet_ns_id = None {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] joviandss-2.nvmet_port_id = None {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] joviandss-2.replication_device = **** {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] joviandss-2.report_discard_supported = None {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] joviandss-2.reserved_percentage = None {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] joviandss-2.scst_target_driver = None {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] joviandss-2.scst_target_iqn_name = None {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] joviandss-2.storage_protocol = None {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] joviandss-2.suppress_requests_ssl_warnings = None {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] joviandss-2.target_helper = None {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] joviandss-2.target_ip_address = None {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] joviandss-2.target_port = 3260 {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] joviandss-2.target_prefix = None {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] joviandss-2.target_protocol = None {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] joviandss-2.target_secondary_ip_addresses = None {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] joviandss-2.trace_flags = None {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] joviandss-2.unique_fqdn_network = None {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] joviandss-2.use_chap_auth = None {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] joviandss-2.use_multipath_for_image_xfer = None {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] joviandss-2.volume_backend_name = joviandss-2 {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] joviandss-2.volume_clear = None {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] joviandss-2.volume_clear_ionice = None {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] joviandss-2.volume_clear_size = None {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] joviandss-2.volume_copy_blkio_cgroup_name = None {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] joviandss-2.volume_copy_bps_limit = None {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] joviandss-2.volume_dd_blocksize = None {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] joviandss-2.volume_driver = cinder.volume.drivers.open_e.iscsi.JovianISCSIDriver {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] joviandss-2.volumes_dir = None {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] oslo_messaging_rabbit.amqp_auto_delete = False {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] oslo_messaging_rabbit.amqp_durable_queues = False {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] oslo_messaging_rabbit.conn_pool_min_size = 2 {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] oslo_messaging_rabbit.conn_pool_ttl = 1200 {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] oslo_messaging_rabbit.direct_mandatory_flag = True {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] oslo_messaging_rabbit.enable_cancel_on_failover = False {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] oslo_messaging_rabbit.heartbeat_in_pthread = False {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] oslo_messaging_rabbit.heartbeat_rate = 2 {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] oslo_messaging_rabbit.heartbeat_timeout_threshold = 60 {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] oslo_messaging_rabbit.kombu_compression = None {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] oslo_messaging_rabbit.kombu_failover_strategy = round-robin {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] oslo_messaging_rabbit.kombu_missing_consumer_retry_timeout = 60 {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] oslo_messaging_rabbit.kombu_reconnect_delay = 1.0 {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] oslo_messaging_rabbit.rabbit_ha_queues = False {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] oslo_messaging_rabbit.rabbit_interval_max = 30 {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] oslo_messaging_rabbit.rabbit_login_method = AMQPLAIN {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] oslo_messaging_rabbit.rabbit_qos_prefetch_count = 0 {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] oslo_messaging_rabbit.rabbit_quorum_delivery_limit = 0 {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] oslo_messaging_rabbit.rabbit_quorum_queue = False {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] oslo_messaging_rabbit.rabbit_quroum_max_memory_bytes = 0 {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] oslo_messaging_rabbit.rabbit_quroum_max_memory_length = 0 {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] oslo_messaging_rabbit.rabbit_retry_backoff = 2 {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] oslo_messaging_rabbit.rabbit_retry_interval = 1 {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] oslo_messaging_rabbit.rabbit_transient_queues_ttl = 1800 {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] oslo_messaging_rabbit.rpc_conn_pool_size = 30 {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] oslo_messaging_rabbit.ssl = False {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] oslo_messaging_rabbit.ssl_ca_file = {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] oslo_messaging_rabbit.ssl_cert_file = {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] oslo_messaging_rabbit.ssl_enforce_fips_mode = False {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] oslo_messaging_rabbit.ssl_key_file = {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] oslo_messaging_rabbit.ssl_version = {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] oslo_messaging_notifications.driver = [] {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] oslo_messaging_notifications.retry = -1 {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] oslo_messaging_notifications.topics = ['notifications'] {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] oslo_messaging_notifications.transport_url = **** {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] key_manager.backend = cinder.keymgr.conf_key_mgr.ConfKeyManager {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] key_manager.fixed_key = **** {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] barbican.auth_endpoint = http://localhost/identity/v3 {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] barbican.barbican_api_version = None {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] barbican.barbican_endpoint = None {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] barbican.barbican_endpoint_type = public {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] barbican.barbican_region_name = None {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] barbican.cafile = None {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] barbican.certfile = None {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] barbican.collect_timing = False {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] barbican.insecure = False {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] barbican.keyfile = None {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] barbican.number_of_retries = 60 {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] barbican.retry_delay = 1 {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] barbican.send_service_user_token = False {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] barbican.split_loggers = False {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] barbican.timeout = None {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] barbican.verify_ssl = True {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] barbican.verify_ssl_path = None {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] barbican_service_user.auth_section = None {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] barbican_service_user.auth_type = None {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] barbican_service_user.cafile = None {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] barbican_service_user.certfile = None {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] barbican_service_user.collect_timing = False {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] barbican_service_user.insecure = False {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] barbican_service_user.keyfile = None {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] barbican_service_user.split_loggers = False {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] barbican_service_user.timeout = None {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] vault.approle_role_id = None {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] vault.approle_secret_id = None {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] vault.cafile = None {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] vault.certfile = None {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] vault.collect_timing = False {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] vault.insecure = False {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] vault.keyfile = None {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] vault.kv_mountpoint = secret {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] vault.kv_version = 2 {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] vault.namespace = None {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] vault.root_token_id = None {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] vault.split_loggers = False {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] vault.ssl_ca_crt_file = None {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] vault.timeout = None {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] vault.use_ssl = False {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] vault.vault_url = http://127.0.0.1:8200 {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2609}} Apr 17 15:19:39 user cinder-volume[84596]: DEBUG oslo_service.service [None req-5e39fe2c-d874-4b5e-a95f-78327510af12 None None] ******************************************************************************** {{(pid=84596) log_opt_values /usr/local/lib/python3.10/dist-packages/oslo_config/cfg.py:2613}} Apr 17 15:19:40 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-56c536ad-a672-42a0-ad8f-80bf0f9c4548 None None] Geting {'code': 200, 'error': None, 'data': {'available': '113926905856', 'status': 30, 'import_status': {'import_steps': {'san_setup': True, 'vip_setup': True, 'zfs_import': True, 'mount_system_volume': True, 'nas_setup': True}, 'import_successful': True}, 'scan': None, 'encryption': {'enabled': False}, 'deduplication_rate': '1.00', 'sysvol_upgrade': 'not_needed', 'vdevs': [{'name': 'wwn-0x6000c296b793bf526a107a9d823e947a', 'iostats': {'read': '0', 'write': '0', 'chksum': '0'}, 'disks': [{'origin': 'local', 'slot': '', 'led': 'off', 'name': 'sdn', 'iostats': {'read': '0', 'write': '0', 'chksum': '0'}, 'alias': '', 'health': 'ONLINE', 'sn': '6000c296b793bf526a107a9d823e947a', 'trim_data': {'status': 'trim_unsupported', 'progress': None, 'trim_time': None}, 'path': 'pci-0000:03:00.0-scsi-0:0:14:0', 'model': 'Virtual disk', 'id': 'wwn-0x6000c296b793bf526a107a9d823e947a', 'size': 128849018880}], 'health': 'ONLINE', 'vdev_replacings': [], 'vdev_spares': [], 'type': ''}], 'id': '14181317334440137330', 'health': 'ONLINE', 'iostats': {'read': '0', 'write': '0', 'chksum': '0'}, 'operation': 'none', 'size': '127775277056', 'autotrim': False, 'name': 'Pool-0'}} from GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:19:40 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-56c536ad-a672-42a0-ad8f-80bf0f9c4548 None None] check pool {{(pid=84696) is_pool_exists /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:71}} Apr 17 15:19:40 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-56c536ad-a672-42a0-ad8f-80bf0f9c4548 None None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:19:40 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-56c536ad-a672-42a0-ad8f-80bf0f9c4548 None None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:19:40 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-56c536ad-a672-42a0-ad8f-80bf0f9c4548 None None] Geting {'code': 200, 'error': None, 'data': {'available': '113926905856', 'status': 30, 'import_status': {'import_steps': {'san_setup': True, 'vip_setup': True, 'zfs_import': True, 'mount_system_volume': True, 'nas_setup': True}, 'import_successful': True}, 'scan': None, 'encryption': {'enabled': False}, 'deduplication_rate': '1.00', 'sysvol_upgrade': 'not_needed', 'vdevs': [{'name': 'wwn-0x6000c296b793bf526a107a9d823e947a', 'iostats': {'read': '0', 'write': '0', 'chksum': '0'}, 'disks': [{'origin': 'local', 'slot': '', 'led': 'off', 'name': 'sdn', 'iostats': {'read': '0', 'write': '0', 'chksum': '0'}, 'alias': '', 'health': 'ONLINE', 'sn': '6000c296b793bf526a107a9d823e947a', 'trim_data': {'status': 'trim_unsupported', 'progress': None, 'trim_time': None}, 'path': 'pci-0000:03:00.0-scsi-0:0:14:0', 'model': 'Virtual disk', 'id': 'wwn-0x6000c296b793bf526a107a9d823e947a', 'size': 128849018880}], 'health': 'ONLINE', 'vdev_replacings': [], 'vdev_spares': [], 'type': ''}], 'id': '14181317334440137330', 'health': 'ONLINE', 'iostats': {'read': '0', 'write': '0', 'chksum': '0'}, 'operation': 'none', 'size': '127775277056', 'autotrim': False, 'name': 'Pool-0'}} from GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:19:40 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-56c536ad-a672-42a0-ad8f-80bf0f9c4548 None None] Updating volume stats {{(pid=84696) _update_volume_stats /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:724}} Apr 17 15:19:40 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-56c536ad-a672-42a0-ad8f-80bf0f9c4548 None None] Get pool Pool-0 fsprops {{(pid=84696) get_pool_stats /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:894}} Apr 17 15:19:40 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-56c536ad-a672-42a0-ad8f-80bf0f9c4548 None None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:19:40 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-56c536ad-a672-42a0-ad8f-80bf0f9c4548 None None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:19:41 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-56c536ad-a672-42a0-ad8f-80bf0f9c4548 None None] Geting {'code': 200, 'error': None, 'data': {'available': '113926905856', 'status': 30, 'import_status': {'import_steps': {'san_setup': True, 'vip_setup': True, 'zfs_import': True, 'mount_system_volume': True, 'nas_setup': True}, 'import_successful': True}, 'scan': None, 'encryption': {'enabled': False}, 'deduplication_rate': '1.00', 'sysvol_upgrade': 'not_needed', 'vdevs': [{'name': 'wwn-0x6000c296b793bf526a107a9d823e947a', 'iostats': {'read': '0', 'write': '0', 'chksum': '0'}, 'disks': [{'origin': 'local', 'slot': '', 'led': 'off', 'name': 'sdn', 'iostats': {'read': '0', 'write': '0', 'chksum': '0'}, 'alias': '', 'health': 'ONLINE', 'sn': '6000c296b793bf526a107a9d823e947a', 'trim_data': {'status': 'trim_unsupported', 'progress': None, 'trim_time': None}, 'path': 'pci-0000:03:00.0-scsi-0:0:14:0', 'model': 'Virtual disk', 'id': 'wwn-0x6000c296b793bf526a107a9d823e947a', 'size': 128849018880}], 'health': 'ONLINE', 'vdev_replacings': [], 'vdev_spares': [], 'type': ''}], 'id': '14181317334440137330', 'health': 'ONLINE', 'iostats': {'read': '0', 'write': '0', 'chksum': '0'}, 'operation': 'none', 'size': '127775277056', 'autotrim': False, 'name': 'Pool-0'}} from GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:19:41 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-56c536ad-a672-42a0-ad8f-80bf0f9c4548 None None] Total capacity: 119, Free 106. {{(pid=84696) _update_volume_stats /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:757}} Apr 17 15:19:41 user cinder-volume[84696]: INFO cinder.volume.driver [None req-56c536ad-a672-42a0-ad8f-80bf0f9c4548 None None] Driver hasn't implemented _init_vendor_properties() Apr 17 15:19:41 user cinder-volume[84696]: DEBUG cinder.volume.driver [None req-56c536ad-a672-42a0-ad8f-80bf0f9c4548 None None] Initialized capabilities list: {'vendor_name': 'Open-E', 'driver_version': '1.0.2', 'storage_protocol': 'iSCSI', 'total_capacity_gb': 119, 'free_capacity_gb': 106, 'reserved_percentage': 0, 'volume_backend_name': 'joviandss-2', 'QoS_support': False, 'location_info': 'JovianISCSIDriver:1:Pool-0', 'multiattach': True, 'properties': {'thin_provisioning': {'title': 'Thin Provisioning', 'description': 'Sets thin provisioning.', 'type': 'boolean'}, 'compression': {'title': 'Compression', 'description': 'Enables compression.', 'type': 'boolean'}, 'qos': {'title': 'QoS', 'description': 'Enables QoS.', 'type': 'boolean'}, 'replication_enabled': {'title': 'Replication', 'description': 'Enables replication.', 'type': 'boolean'}}}. {{(pid=84696) init_capabilities /opt/stack/cinder/cinder/volume/driver.py:832}} Apr 17 15:19:41 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-56c536ad-a672-42a0-ad8f-80bf0f9c4548 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 15:19:41 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writer thread running {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:102}} Apr 17 15:19:41 user cinder-volume[84696]: DEBUG cinder.db.sqlalchemy.api [None req-56c536ad-a672-42a0-ad8f-80bf0f9c4548 None None] Building query based on filter {{(pid=84696) _process_snaps_filters /opt/stack/cinder/cinder/db/sqlalchemy/api.py:3885}} Apr 17 15:19:41 user cinder-volume[84696]: DEBUG cinder.image.image_utils [None req-56c536ad-a672-42a0-ad8f-80bf0f9c4548 None None] Configuration image_conversion_dir is None or the path doesn't exist. {{(pid=84696) cleanup_temporary_file /opt/stack/cinder/cinder/image/image_utils.py:1146}} Apr 17 15:19:41 user cinder-volume[84696]: INFO cinder.keymgr.migration [None req-bd7c0580-d77f-40bb-9cb1-452189fbda32 None None] Not migrating encryption keys because the ConfKeyManager is still in use. Apr 17 15:19:41 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-56c536ad-a672-42a0-ad8f-80bf0f9c4548 None None] Updating volume stats {{(pid=84696) _update_volume_stats /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:724}} Apr 17 15:19:41 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-56c536ad-a672-42a0-ad8f-80bf0f9c4548 None None] Get pool Pool-0 fsprops {{(pid=84696) get_pool_stats /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:894}} Apr 17 15:19:41 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-56c536ad-a672-42a0-ad8f-80bf0f9c4548 None None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:19:41 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-56c536ad-a672-42a0-ad8f-80bf0f9c4548 None None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:19:42 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-56c536ad-a672-42a0-ad8f-80bf0f9c4548 None None] Geting {'code': 200, 'error': None, 'data': {'available': '113926905856', 'status': 30, 'import_status': {'import_steps': {'san_setup': True, 'vip_setup': True, 'zfs_import': True, 'mount_system_volume': True, 'nas_setup': True}, 'import_successful': True}, 'scan': None, 'encryption': {'enabled': False}, 'deduplication_rate': '1.00', 'sysvol_upgrade': 'not_needed', 'vdevs': [{'name': 'wwn-0x6000c296b793bf526a107a9d823e947a', 'iostats': {'read': '0', 'write': '0', 'chksum': '0'}, 'disks': [{'origin': 'local', 'slot': '', 'led': 'off', 'name': 'sdn', 'iostats': {'read': '0', 'write': '0', 'chksum': '0'}, 'alias': '', 'health': 'ONLINE', 'sn': '6000c296b793bf526a107a9d823e947a', 'trim_data': {'status': 'trim_unsupported', 'progress': None, 'trim_time': None}, 'path': 'pci-0000:03:00.0-scsi-0:0:14:0', 'model': 'Virtual disk', 'id': 'wwn-0x6000c296b793bf526a107a9d823e947a', 'size': 128849018880}], 'health': 'ONLINE', 'vdev_replacings': [], 'vdev_spares': [], 'type': ''}], 'id': '14181317334440137330', 'health': 'ONLINE', 'iostats': {'read': '0', 'write': '0', 'chksum': '0'}, 'operation': 'none', 'size': '127775277056', 'autotrim': False, 'name': 'Pool-0'}} from GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:19:42 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-56c536ad-a672-42a0-ad8f-80bf0f9c4548 None None] Total capacity: 119, Free 106. {{(pid=84696) _update_volume_stats /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:757}} Apr 17 15:19:42 user cinder-volume[84696]: DEBUG cinder.manager [None req-56c536ad-a672-42a0-ad8f-80bf0f9c4548 None None] Notifying Schedulers of capabilities ... {{(pid=84696) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 17 15:19:42 user cinder-volume[84696]: INFO cinder.volume.manager [None req-56c536ad-a672-42a0-ad8f-80bf0f9c4548 None None] Driver initialization completed successfully. Apr 17 15:19:42 user cinder-volume[84696]: INFO cinder.manager [None req-56c536ad-a672-42a0-ad8f-80bf0f9c4548 None None] Initiating service 2 cleanup Apr 17 15:19:42 user cinder-volume[84696]: INFO cinder.manager [None req-56c536ad-a672-42a0-ad8f-80bf0f9c4548 None None] Service 2 cleanup completed. Apr 17 15:19:42 user cinder-volume[84696]: DEBUG cinder.service [None req-56c536ad-a672-42a0-ad8f-80bf0f9c4548 None None] Creating RPC server for service cinder-volume {{(pid=84696) start /opt/stack/cinder/cinder/service.py:238}} Apr 17 15:19:42 user cinder-volume[84696]: DEBUG cinder.service [None req-56c536ad-a672-42a0-ad8f-80bf0f9c4548 None None] Pinning object versions for RPC server serializer to 1.39 {{(pid=84696) start /opt/stack/cinder/cinder/service.py:244}} Apr 17 15:19:42 user cinder-volume[84696]: INFO cinder.volume.manager [None req-56c536ad-a672-42a0-ad8f-80bf0f9c4548 None None] Initializing RPC dependent components of volume driver JovianISCSIDriver (1.0.2) Apr 17 15:19:42 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-56c536ad-a672-42a0-ad8f-80bf0f9c4548 None None] Updating volume stats {{(pid=84696) _update_volume_stats /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:724}} Apr 17 15:19:42 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-56c536ad-a672-42a0-ad8f-80bf0f9c4548 None None] Get pool Pool-0 fsprops {{(pid=84696) get_pool_stats /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:894}} Apr 17 15:19:42 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-56c536ad-a672-42a0-ad8f-80bf0f9c4548 None None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:19:42 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-56c536ad-a672-42a0-ad8f-80bf0f9c4548 None None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:19:43 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-56c536ad-a672-42a0-ad8f-80bf0f9c4548 None None] Geting {'code': 200, 'error': None, 'data': {'available': '113926905856', 'status': 30, 'import_status': {'import_steps': {'san_setup': True, 'vip_setup': True, 'zfs_import': True, 'mount_system_volume': True, 'nas_setup': True}, 'import_successful': True}, 'scan': None, 'encryption': {'enabled': False}, 'deduplication_rate': '1.00', 'sysvol_upgrade': 'not_needed', 'vdevs': [{'name': 'wwn-0x6000c296b793bf526a107a9d823e947a', 'iostats': {'read': '0', 'write': '0', 'chksum': '0'}, 'disks': [{'origin': 'local', 'slot': '', 'led': 'off', 'name': 'sdn', 'iostats': {'read': '0', 'write': '0', 'chksum': '0'}, 'alias': '', 'health': 'ONLINE', 'sn': '6000c296b793bf526a107a9d823e947a', 'trim_data': {'status': 'trim_unsupported', 'progress': None, 'trim_time': None}, 'path': 'pci-0000:03:00.0-scsi-0:0:14:0', 'model': 'Virtual disk', 'id': 'wwn-0x6000c296b793bf526a107a9d823e947a', 'size': 128849018880}], 'health': 'ONLINE', 'vdev_replacings': [], 'vdev_spares': [], 'type': ''}], 'id': '14181317334440137330', 'health': 'ONLINE', 'iostats': {'read': '0', 'write': '0', 'chksum': '0'}, 'operation': 'none', 'size': '127775277056', 'autotrim': False, 'name': 'Pool-0'}} from GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:19:43 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-56c536ad-a672-42a0-ad8f-80bf0f9c4548 None None] Total capacity: 119, Free 106. {{(pid=84696) _update_volume_stats /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:757}} Apr 17 15:19:43 user cinder-volume[84696]: INFO cinder.volume.manager [None req-56c536ad-a672-42a0-ad8f-80bf0f9c4548 None None] Driver post RPC initialization completed successfully. Apr 17 15:19:49 user cinder-volume[84596]: DEBUG dbcounter [-] [84596] Writing DB stats cinder:SELECT=16,cinder:INSERT=1 {{(pid=84596) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:19:53 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=6 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:19:53 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=23,cinder:INSERT=1,cinder:UPDATE=1 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:20:13 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=4 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:20:23 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:UPDATE=3,cinder:SELECT=5 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:20:43 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=5,cinder:UPDATE=1 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:21:13 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=9,cinder:UPDATE=2 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:21:15 user cinder-volume[84696]: DEBUG oslo_service.periodic_task [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Running periodic task VolumeManager.publish_service_capabilities {{(pid=84696) run_periodic_tasks /usr/local/lib/python3.10/dist-packages/oslo_service/periodic_task.py:210}} Apr 17 15:21:15 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Updating volume stats {{(pid=84696) _update_volume_stats /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:724}} Apr 17 15:21:15 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Get pool Pool-0 fsprops {{(pid=84696) get_pool_stats /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:894}} Apr 17 15:21:15 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:21:15 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:21:16 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Geting {'code': 200, 'error': None, 'data': {'available': '113926905856', 'status': 30, 'import_status': {'import_steps': {'san_setup': True, 'vip_setup': True, 'zfs_import': True, 'mount_system_volume': True, 'nas_setup': True}, 'import_successful': True}, 'scan': None, 'encryption': {'enabled': False}, 'deduplication_rate': '1.00', 'sysvol_upgrade': 'not_needed', 'vdevs': [{'name': 'wwn-0x6000c296b793bf526a107a9d823e947a', 'iostats': {'read': '0', 'write': '0', 'chksum': '0'}, 'disks': [{'origin': 'local', 'slot': '', 'led': 'off', 'name': 'sdn', 'iostats': {'read': '0', 'write': '0', 'chksum': '0'}, 'alias': '', 'health': 'ONLINE', 'sn': '6000c296b793bf526a107a9d823e947a', 'trim_data': {'status': 'trim_unsupported', 'progress': None, 'trim_time': None}, 'path': 'pci-0000:03:00.0-scsi-0:0:14:0', 'model': 'Virtual disk', 'id': 'wwn-0x6000c296b793bf526a107a9d823e947a', 'size': 128849018880}], 'health': 'ONLINE', 'vdev_replacings': [], 'vdev_spares': [], 'type': ''}], 'id': '14181317334440137330', 'health': 'ONLINE', 'iostats': {'read': '0', 'write': '0', 'chksum': '0'}, 'operation': 'none', 'size': '127775277056', 'autotrim': False, 'name': 'Pool-0'}} from GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:21:16 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Total capacity: 119, Free 106. {{(pid=84696) _update_volume_stats /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:757}} Apr 17 15:21:16 user cinder-volume[84696]: DEBUG cinder.manager [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Notifying Schedulers of capabilities ... {{(pid=84696) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 17 15:21:23 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=5,cinder:UPDATE=3 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:21:43 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=3,cinder:UPDATE=2 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:22:03 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=1 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:22:03 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=8,cinder:UPDATE=2 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:22:15 user cinder-volume[84696]: DEBUG oslo_service.periodic_task [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Running periodic task VolumeManager.publish_service_capabilities {{(pid=84696) run_periodic_tasks /usr/local/lib/python3.10/dist-packages/oslo_service/periodic_task.py:210}} Apr 17 15:22:15 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Updating volume stats {{(pid=84696) _update_volume_stats /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:724}} Apr 17 15:22:15 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Get pool Pool-0 fsprops {{(pid=84696) get_pool_stats /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:894}} Apr 17 15:22:15 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:22:15 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:22:16 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Geting {'code': 200, 'error': None, 'data': {'available': '113926918144', 'status': 30, 'import_status': {'import_steps': {'san_setup': True, 'vip_setup': True, 'zfs_import': True, 'mount_system_volume': True, 'nas_setup': True}, 'import_successful': True}, 'scan': None, 'encryption': {'enabled': False}, 'deduplication_rate': '1.00', 'sysvol_upgrade': 'not_needed', 'vdevs': [{'name': 'wwn-0x6000c296b793bf526a107a9d823e947a', 'iostats': {'read': '0', 'write': '0', 'chksum': '0'}, 'disks': [{'origin': 'local', 'slot': '', 'led': 'off', 'name': 'sdn', 'iostats': {'read': '0', 'write': '0', 'chksum': '0'}, 'alias': '', 'health': 'ONLINE', 'sn': '6000c296b793bf526a107a9d823e947a', 'trim_data': {'status': 'trim_unsupported', 'progress': None, 'trim_time': None}, 'path': 'pci-0000:03:00.0-scsi-0:0:14:0', 'model': 'Virtual disk', 'id': 'wwn-0x6000c296b793bf526a107a9d823e947a', 'size': 128849018880}], 'health': 'ONLINE', 'vdev_replacings': [], 'vdev_spares': [], 'type': ''}], 'id': '14181317334440137330', 'health': 'ONLINE', 'iostats': {'read': '0', 'write': '0', 'chksum': '0'}, 'operation': 'none', 'size': '127775277056', 'autotrim': False, 'name': 'Pool-0'}} from GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:22:16 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Total capacity: 119, Free 106. {{(pid=84696) _update_volume_stats /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:757}} Apr 17 15:22:16 user cinder-volume[84696]: DEBUG cinder.manager [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Notifying Schedulers of capabilities ... {{(pid=84696) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 17 15:22:33 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=4,cinder:UPDATE=2 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:22:33 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=5,cinder:UPDATE=1 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:22:53 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=3,cinder:UPDATE=1 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:23:13 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=7,cinder:UPDATE=2 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:23:15 user cinder-volume[84696]: DEBUG oslo_service.periodic_task [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Running periodic task VolumeManager.publish_service_capabilities {{(pid=84696) run_periodic_tasks /usr/local/lib/python3.10/dist-packages/oslo_service/periodic_task.py:210}} Apr 17 15:23:15 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Updating volume stats {{(pid=84696) _update_volume_stats /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:724}} Apr 17 15:23:15 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Get pool Pool-0 fsprops {{(pid=84696) get_pool_stats /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:894}} Apr 17 15:23:15 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:23:15 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:23:16 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Geting {'code': 200, 'error': None, 'data': {'available': '113926905856', 'status': 30, 'import_status': {'import_steps': {'san_setup': True, 'vip_setup': True, 'zfs_import': True, 'mount_system_volume': True, 'nas_setup': True}, 'import_successful': True}, 'scan': None, 'encryption': {'enabled': False}, 'deduplication_rate': '1.00', 'sysvol_upgrade': 'not_needed', 'vdevs': [{'name': 'wwn-0x6000c296b793bf526a107a9d823e947a', 'iostats': {'read': '0', 'write': '0', 'chksum': '0'}, 'disks': [{'origin': 'local', 'slot': '', 'led': 'off', 'name': 'sdn', 'iostats': {'read': '0', 'write': '0', 'chksum': '0'}, 'alias': '', 'health': 'ONLINE', 'sn': '6000c296b793bf526a107a9d823e947a', 'trim_data': {'status': 'trim_unsupported', 'progress': None, 'trim_time': None}, 'path': 'pci-0000:03:00.0-scsi-0:0:14:0', 'model': 'Virtual disk', 'id': 'wwn-0x6000c296b793bf526a107a9d823e947a', 'size': 128849018880}], 'health': 'ONLINE', 'vdev_replacings': [], 'vdev_spares': [], 'type': ''}], 'id': '14181317334440137330', 'health': 'ONLINE', 'iostats': {'read': '0', 'write': '0', 'chksum': '0'}, 'operation': 'none', 'size': '127775277056', 'autotrim': False, 'name': 'Pool-0'}} from GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:23:16 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Total capacity: 119, Free 106. {{(pid=84696) _update_volume_stats /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:757}} Apr 17 15:23:16 user cinder-volume[84696]: DEBUG cinder.manager [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Notifying Schedulers of capabilities ... {{(pid=84696) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 17 15:23:23 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=5,cinder:UPDATE=2 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:23:26 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-05b47da9-26a7-4ab0-966e-6fa0f3f58ac6 tempest-VolumesTestJSON-1329991747 None] Volume reschedule parameters: True retry: {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 15:23:26 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-05b47da9-26a7-4ab0-966e-6fa0f3f58ac6 tempest-VolumesTestJSON-1329991747 None] Flow 'volume_create_manager' (5d6232ff-60e3-4f16-8e4f-d1cd781e094b) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:23:26 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-05b47da9-26a7-4ab0-966e-6fa0f3f58ac6 tempest-VolumesTestJSON-1329991747 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (056ee0bd-42e3-41f7-beb5-fc24cab63130) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:23:26 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-05b47da9-26a7-4ab0-966e-6fa0f3f58ac6 tempest-VolumesTestJSON-1329991747 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (056ee0bd-42e3-41f7-beb5-fc24cab63130) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:23:26Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesTestJSON-volume-420411441',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=c5f04e05-b556-4cb3-84c8-590aa69695b0,launched_at=None,metadata={Type='work'},migration_status=None,multiattach=False,previous_status=None,project_id='01ad54f0fc8a4200a62264f0f9c33107',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:23:26Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:23:26Z,use_quota=True,user_id='e8b77c4bc3b84389b237889bcd55f2af',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:23:26 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-05b47da9-26a7-4ab0-966e-6fa0f3f58ac6 tempest-VolumesTestJSON-1329991747 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (78dd7c26-a3f7-4963-83fd-2572b6664d38) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:23:26 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-05b47da9-26a7-4ab0-966e-6fa0f3f58ac6 tempest-VolumesTestJSON-1329991747 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (78dd7c26-a3f7-4963-83fd-2572b6664d38) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:23:26 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-05b47da9-26a7-4ab0-966e-6fa0f3f58ac6 tempest-VolumesTestJSON-1329991747 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (0d1ca77d-e22e-4699-99bd-63aa3c47b78d) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:23:26 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-05b47da9-26a7-4ab0-966e-6fa0f3f58ac6 tempest-VolumesTestJSON-1329991747 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (0d1ca77d-e22e-4699-99bd-63aa3c47b78d) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': 'c5f04e05-b556-4cb3-84c8-590aa69695b0', 'volume_name': 'volume-c5f04e05-b556-4cb3-84c8-590aa69695b0', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:23:26 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-05b47da9-26a7-4ab0-966e-6fa0f3f58ac6 tempest-VolumesTestJSON-1329991747 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (c8dd2519-e146-4663-bc2c-bddbf7b16b19) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:23:26 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-05b47da9-26a7-4ab0-966e-6fa0f3f58ac6 tempest-VolumesTestJSON-1329991747 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (c8dd2519-e146-4663-bc2c-bddbf7b16b19) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:23:26 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-05b47da9-26a7-4ab0-966e-6fa0f3f58ac6 tempest-VolumesTestJSON-1329991747 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (8244fa12-ca59-4c56-a907-9942ef0ed736) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:23:26 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-05b47da9-26a7-4ab0-966e-6fa0f3f58ac6 tempest-VolumesTestJSON-1329991747 None] Volume c5f04e05-b556-4cb3-84c8-590aa69695b0: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-c5f04e05-b556-4cb3-84c8-590aa69695b0', 'volume_size': 1} Apr 17 15:23:26 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-05b47da9-26a7-4ab0-966e-6fa0f3f58ac6 tempest-VolumesTestJSON-1329991747 None] creating volume v_c5f04e05-b556-4cb3-84c8-590aa69695b0. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 15:23:26 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-05b47da9-26a7-4ab0-966e-6fa0f3f58ac6 tempest-VolumesTestJSON-1329991747 None] create volume {'name': 'v_c5f04e05-b556-4cb3-84c8-590aa69695b0', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 15:23:26 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-05b47da9-26a7-4ab0-966e-6fa0f3f58ac6 tempest-VolumesTestJSON-1329991747 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:23:26 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-05b47da9-26a7-4ab0-966e-6fa0f3f58ac6 tempest-VolumesTestJSON-1329991747 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:23:26 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-773249cb-bb44-4155-b636-73558e12aaff tempest-VolumeQuotasNegativeTestJSON-2121978028 None] Volume reschedule parameters: True retry: {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 15:23:26 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-773249cb-bb44-4155-b636-73558e12aaff tempest-VolumeQuotasNegativeTestJSON-2121978028 None] Flow 'volume_create_manager' (3294ef7f-919b-4f57-a583-e4a4cf0dba1a) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:23:26 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-773249cb-bb44-4155-b636-73558e12aaff tempest-VolumeQuotasNegativeTestJSON-2121978028 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (67ed183d-dea1-48c7-a249-3e45fb55d082) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:23:26 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-773249cb-bb44-4155-b636-73558e12aaff tempest-VolumeQuotasNegativeTestJSON-2121978028 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (67ed183d-dea1-48c7-a249-3e45fb55d082) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:23:26Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumeQuotasNegativeTestJSON-Volume-492980821',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=90683e1b-58ef-42b7-8ebf-85f478ece679,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='ff79d745ac3b45da90c1cca9fcba6b9c',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:23:26Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:23:26Z,use_quota=True,user_id='6ce046152381477e89b03a5d253a0047',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:23:26 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-773249cb-bb44-4155-b636-73558e12aaff tempest-VolumeQuotasNegativeTestJSON-2121978028 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (3940309b-aeb8-4bb4-940b-c7ca57f82485) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:23:26 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-773249cb-bb44-4155-b636-73558e12aaff tempest-VolumeQuotasNegativeTestJSON-2121978028 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (3940309b-aeb8-4bb4-940b-c7ca57f82485) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:23:26 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-773249cb-bb44-4155-b636-73558e12aaff tempest-VolumeQuotasNegativeTestJSON-2121978028 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (518b8f46-7d76-4bb4-8234-78ec343d48fe) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:23:26 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-773249cb-bb44-4155-b636-73558e12aaff tempest-VolumeQuotasNegativeTestJSON-2121978028 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (518b8f46-7d76-4bb4-8234-78ec343d48fe) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': '90683e1b-58ef-42b7-8ebf-85f478ece679', 'volume_name': 'volume-90683e1b-58ef-42b7-8ebf-85f478ece679', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:23:26 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-773249cb-bb44-4155-b636-73558e12aaff tempest-VolumeQuotasNegativeTestJSON-2121978028 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (0205e102-7ea9-4a18-b4b0-f56e08ab3006) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:23:26 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-773249cb-bb44-4155-b636-73558e12aaff tempest-VolumeQuotasNegativeTestJSON-2121978028 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (0205e102-7ea9-4a18-b4b0-f56e08ab3006) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:23:26 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-773249cb-bb44-4155-b636-73558e12aaff tempest-VolumeQuotasNegativeTestJSON-2121978028 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (ad87c2dd-ed6a-4a39-a3bd-8421499c794d) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:23:26 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-773249cb-bb44-4155-b636-73558e12aaff tempest-VolumeQuotasNegativeTestJSON-2121978028 None] Volume 90683e1b-58ef-42b7-8ebf-85f478ece679: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-90683e1b-58ef-42b7-8ebf-85f478ece679', 'volume_size': 1} Apr 17 15:23:26 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-773249cb-bb44-4155-b636-73558e12aaff tempest-VolumeQuotasNegativeTestJSON-2121978028 None] creating volume v_90683e1b-58ef-42b7-8ebf-85f478ece679. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 15:23:26 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-773249cb-bb44-4155-b636-73558e12aaff tempest-VolumeQuotasNegativeTestJSON-2121978028 None] create volume {'name': 'v_90683e1b-58ef-42b7-8ebf-85f478ece679', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 15:23:26 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-773249cb-bb44-4155-b636-73558e12aaff tempest-VolumeQuotasNegativeTestJSON-2121978028 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:23:26 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-773249cb-bb44-4155-b636-73558e12aaff tempest-VolumeQuotasNegativeTestJSON-2121978028 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:23:26 user cinder-volume[84696]: INFO cinder.volume.manager [None req-0dec2be9-e768-4b8e-805b-db306db439de tempest-GroupSnapshotsTest-1926061586 None] Group tempest-GroupSnapshotsTest-Group-593988090: creating Apr 17 15:23:26 user cinder-volume[84696]: INFO cinder.volume.manager [None req-0dec2be9-e768-4b8e-805b-db306db439de tempest-GroupSnapshotsTest-1926061586 None] Group tempest-GroupSnapshotsTest-Group-593988090: created successfully Apr 17 15:23:26 user cinder-volume[84696]: INFO cinder.volume.manager [None req-0dec2be9-e768-4b8e-805b-db306db439de tempest-GroupSnapshotsTest-1926061586 None] Create group completed successfully. Apr 17 15:23:27 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-05b47da9-26a7-4ab0-966e-6fa0f3f58ac6 tempest-VolumesTestJSON-1329991747 None] Geting {'code': 201, 'error': None, 'data': {'full_name': 'Pool-0/v_c5f04e05-b556-4cb3-84c8-590aa69695b0', 'is_clone': False, 'name': 'v_c5f04e05-b556-4cb3-84c8-590aa69695b0'}} from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:23:27 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-05b47da9-26a7-4ab0-966e-6fa0f3f58ac6 tempest-VolumesTestJSON-1329991747 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (8244fa12-ca59-4c56-a907-9942ef0ed736) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'volume_name': 'volume-c5f04e05-b556-4cb3-84c8-590aa69695b0', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:23:27 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-05b47da9-26a7-4ab0-966e-6fa0f3f58ac6 tempest-VolumesTestJSON-1329991747 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeOnFinishTask;volume:create, create.end' (e7940da6-dc43-4970-9f7b-e8c2567b1f92) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:23:27 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-773249cb-bb44-4155-b636-73558e12aaff tempest-VolumeQuotasNegativeTestJSON-2121978028 None] Geting {'code': 201, 'error': None, 'data': {'full_name': 'Pool-0/v_90683e1b-58ef-42b7-8ebf-85f478ece679', 'is_clone': False, 'name': 'v_90683e1b-58ef-42b7-8ebf-85f478ece679'}} from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:23:27 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-05b47da9-26a7-4ab0-966e-6fa0f3f58ac6 tempest-VolumesTestJSON-1329991747 None] Volume volume-c5f04e05-b556-4cb3-84c8-590aa69695b0 (c5f04e05-b556-4cb3-84c8-590aa69695b0): created successfully Apr 17 15:23:27 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-05b47da9-26a7-4ab0-966e-6fa0f3f58ac6 tempest-VolumesTestJSON-1329991747 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeOnFinishTask;volume:create, create.end' (e7940da6-dc43-4970-9f7b-e8c2567b1f92) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:23:27 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-05b47da9-26a7-4ab0-966e-6fa0f3f58ac6 tempest-VolumesTestJSON-1329991747 None] Flow 'volume_create_manager' (5d6232ff-60e3-4f16-8e4f-d1cd781e094b) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:23:27 user cinder-volume[84696]: INFO cinder.volume.manager [None req-05b47da9-26a7-4ab0-966e-6fa0f3f58ac6 tempest-VolumesTestJSON-1329991747 None] Created volume successfully. Apr 17 15:23:27 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-773249cb-bb44-4155-b636-73558e12aaff tempest-VolumeQuotasNegativeTestJSON-2121978028 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (ad87c2dd-ed6a-4a39-a3bd-8421499c794d) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'volume_name': 'volume-90683e1b-58ef-42b7-8ebf-85f478ece679', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:23:27 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-773249cb-bb44-4155-b636-73558e12aaff tempest-VolumeQuotasNegativeTestJSON-2121978028 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeOnFinishTask;volume:create, create.end' (9446504b-98d2-4e89-b182-e40c9bde70ab) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:23:27 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-773249cb-bb44-4155-b636-73558e12aaff tempest-VolumeQuotasNegativeTestJSON-2121978028 None] Volume volume-90683e1b-58ef-42b7-8ebf-85f478ece679 (90683e1b-58ef-42b7-8ebf-85f478ece679): created successfully Apr 17 15:23:27 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-773249cb-bb44-4155-b636-73558e12aaff tempest-VolumeQuotasNegativeTestJSON-2121978028 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeOnFinishTask;volume:create, create.end' (9446504b-98d2-4e89-b182-e40c9bde70ab) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:23:27 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-773249cb-bb44-4155-b636-73558e12aaff tempest-VolumeQuotasNegativeTestJSON-2121978028 None] Flow 'volume_create_manager' (3294ef7f-919b-4f57-a583-e4a4cf0dba1a) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:23:27 user cinder-volume[84696]: INFO cinder.volume.manager [None req-773249cb-bb44-4155-b636-73558e12aaff tempest-VolumeQuotasNegativeTestJSON-2121978028 None] Created volume successfully. Apr 17 15:23:28 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-ff28d70d-3957-4e48-97d2-741aa067698b tempest-GroupSnapshotsTest-1926061586 None] Volume reschedule parameters: True retry: {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 15:23:28 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-ff28d70d-3957-4e48-97d2-741aa067698b tempest-GroupSnapshotsTest-1926061586 None] Flow 'volume_create_manager' (170b23e9-34e9-4d92-a86a-80fe23904fd8) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:23:28 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-ff28d70d-3957-4e48-97d2-741aa067698b tempest-GroupSnapshotsTest-1926061586 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (7d86f57c-faf9-4f89-9162-d8d2c39eccf8) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:23:28 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-ff28d70d-3957-4e48-97d2-741aa067698b tempest-GroupSnapshotsTest-1926061586 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (7d86f57c-faf9-4f89-9162-d8d2c39eccf8) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:23:28Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-volume-1843290826',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=203550cd-a3a4-4ff5-bfe6-6ed2607e334b,host='user@joviandss-2#joviandss-2',id=74f7ff47-86dd-433a-8a61-f46278efe3fd,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='6cae12f3c2614d6fb32f21db60fec8d3',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:23:28Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:23:28Z,use_quota=True,user_id='560c850c324949d7a214e7a59dd65f8d',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(a40cb290-464c-4f83-9d6b-adf225cc2e0c),volume_type_id=a40cb290-464c-4f83-9d6b-adf225cc2e0c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:23:28 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-ff28d70d-3957-4e48-97d2-741aa067698b tempest-GroupSnapshotsTest-1926061586 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (70af0944-4de8-46bf-b139-8d68e0967235) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:23:28 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-ff28d70d-3957-4e48-97d2-741aa067698b tempest-GroupSnapshotsTest-1926061586 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (70af0944-4de8-46bf-b139-8d68e0967235) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:23:28 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-ff28d70d-3957-4e48-97d2-741aa067698b tempest-GroupSnapshotsTest-1926061586 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (15059ea3-d784-4c88-868e-3d15720ad89a) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:23:28 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-ff28d70d-3957-4e48-97d2-741aa067698b tempest-GroupSnapshotsTest-1926061586 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (15059ea3-d784-4c88-868e-3d15720ad89a) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': '74f7ff47-86dd-433a-8a61-f46278efe3fd', 'volume_name': 'volume-74f7ff47-86dd-433a-8a61-f46278efe3fd', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:23:28 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-ff28d70d-3957-4e48-97d2-741aa067698b tempest-GroupSnapshotsTest-1926061586 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (ff86f68c-6c02-4abe-a016-3166e069541c) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:23:28 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-ff28d70d-3957-4e48-97d2-741aa067698b tempest-GroupSnapshotsTest-1926061586 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (ff86f68c-6c02-4abe-a016-3166e069541c) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:23:28 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-ff28d70d-3957-4e48-97d2-741aa067698b tempest-GroupSnapshotsTest-1926061586 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (60a56bba-7100-4637-9d72-1e945fbf46b7) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:23:28 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-ff28d70d-3957-4e48-97d2-741aa067698b tempest-GroupSnapshotsTest-1926061586 None] Volume 74f7ff47-86dd-433a-8a61-f46278efe3fd: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-74f7ff47-86dd-433a-8a61-f46278efe3fd', 'volume_size': 1} Apr 17 15:23:28 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-ff28d70d-3957-4e48-97d2-741aa067698b tempest-GroupSnapshotsTest-1926061586 None] creating volume v_74f7ff47-86dd-433a-8a61-f46278efe3fd. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 15:23:28 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-ff28d70d-3957-4e48-97d2-741aa067698b tempest-GroupSnapshotsTest-1926061586 None] create volume {'name': 'v_74f7ff47-86dd-433a-8a61-f46278efe3fd', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 15:23:28 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-ff28d70d-3957-4e48-97d2-741aa067698b tempest-GroupSnapshotsTest-1926061586 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:23:28 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-ff28d70d-3957-4e48-97d2-741aa067698b tempest-GroupSnapshotsTest-1926061586 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:23:29 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-7cf58784-5a09-4291-b5b9-ad092863a300 tempest-VolumesTestJSON-1329991747 None] Volume reschedule parameters: True retry: {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 15:23:29 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-7cf58784-5a09-4291-b5b9-ad092863a300 tempest-VolumesTestJSON-1329991747 None] Flow 'volume_create_manager' (74cdd190-9b5c-49cf-b73d-b498ab2c2320) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:23:29 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-7cf58784-5a09-4291-b5b9-ad092863a300 tempest-VolumesTestJSON-1329991747 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (480192f0-37d5-4ed1-9dee-df6f9c09cb01) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:23:29 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-7cf58784-5a09-4291-b5b9-ad092863a300 tempest-VolumesTestJSON-1329991747 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (480192f0-37d5-4ed1-9dee-df6f9c09cb01) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:23:29Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesTestJSON-volume-238641164',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=f8f9aea9-6773-4928-9e8e-153c3796301b,launched_at=None,metadata={Type='work'},migration_status=None,multiattach=False,previous_status=None,project_id='01ad54f0fc8a4200a62264f0f9c33107',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:23:29Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:23:29Z,use_quota=True,user_id='e8b77c4bc3b84389b237889bcd55f2af',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:23:29 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-7cf58784-5a09-4291-b5b9-ad092863a300 tempest-VolumesTestJSON-1329991747 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (542a72f3-07a2-43cb-b2ff-8143517f990f) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:23:29 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-7cf58784-5a09-4291-b5b9-ad092863a300 tempest-VolumesTestJSON-1329991747 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (542a72f3-07a2-43cb-b2ff-8143517f990f) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:23:29 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-7cf58784-5a09-4291-b5b9-ad092863a300 tempest-VolumesTestJSON-1329991747 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (c05dc9c1-f356-4c78-8078-41faee090ce8) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:23:29 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-7cf58784-5a09-4291-b5b9-ad092863a300 tempest-VolumesTestJSON-1329991747 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (c05dc9c1-f356-4c78-8078-41faee090ce8) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': 'f8f9aea9-6773-4928-9e8e-153c3796301b', 'volume_name': 'volume-f8f9aea9-6773-4928-9e8e-153c3796301b', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:23:29 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-7cf58784-5a09-4291-b5b9-ad092863a300 tempest-VolumesTestJSON-1329991747 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (5c8f236c-0a55-4a8a-b47e-a0b1da0c6948) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:23:29 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-7cf58784-5a09-4291-b5b9-ad092863a300 tempest-VolumesTestJSON-1329991747 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (5c8f236c-0a55-4a8a-b47e-a0b1da0c6948) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:23:29 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-7cf58784-5a09-4291-b5b9-ad092863a300 tempest-VolumesTestJSON-1329991747 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (2e81da61-388c-409b-9a56-f19dedcec4d0) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:23:29 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-7cf58784-5a09-4291-b5b9-ad092863a300 tempest-VolumesTestJSON-1329991747 None] Volume f8f9aea9-6773-4928-9e8e-153c3796301b: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-f8f9aea9-6773-4928-9e8e-153c3796301b', 'volume_size': 1} Apr 17 15:23:29 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-7cf58784-5a09-4291-b5b9-ad092863a300 tempest-VolumesTestJSON-1329991747 None] creating volume v_f8f9aea9-6773-4928-9e8e-153c3796301b. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 15:23:29 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-7cf58784-5a09-4291-b5b9-ad092863a300 tempest-VolumesTestJSON-1329991747 None] create volume {'name': 'v_f8f9aea9-6773-4928-9e8e-153c3796301b', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 15:23:29 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-7cf58784-5a09-4291-b5b9-ad092863a300 tempest-VolumesTestJSON-1329991747 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:23:29 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-7cf58784-5a09-4291-b5b9-ad092863a300 tempest-VolumesTestJSON-1329991747 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:23:29 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-ff28d70d-3957-4e48-97d2-741aa067698b tempest-GroupSnapshotsTest-1926061586 None] Geting {'code': 201, 'error': None, 'data': {'full_name': 'Pool-0/v_74f7ff47-86dd-433a-8a61-f46278efe3fd', 'is_clone': False, 'name': 'v_74f7ff47-86dd-433a-8a61-f46278efe3fd'}} from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:23:29 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-ff28d70d-3957-4e48-97d2-741aa067698b tempest-GroupSnapshotsTest-1926061586 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (60a56bba-7100-4637-9d72-1e945fbf46b7) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'volume_name': 'volume-74f7ff47-86dd-433a-8a61-f46278efe3fd', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:23:29 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-ff28d70d-3957-4e48-97d2-741aa067698b tempest-GroupSnapshotsTest-1926061586 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeOnFinishTask;volume:create, create.end' (c0776812-1173-4c36-bacf-22fa78ec022e) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:23:29 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-ff28d70d-3957-4e48-97d2-741aa067698b tempest-GroupSnapshotsTest-1926061586 None] Volume volume-74f7ff47-86dd-433a-8a61-f46278efe3fd (74f7ff47-86dd-433a-8a61-f46278efe3fd): created successfully Apr 17 15:23:29 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-ff28d70d-3957-4e48-97d2-741aa067698b tempest-GroupSnapshotsTest-1926061586 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeOnFinishTask;volume:create, create.end' (c0776812-1173-4c36-bacf-22fa78ec022e) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:23:29 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-ff28d70d-3957-4e48-97d2-741aa067698b tempest-GroupSnapshotsTest-1926061586 None] Flow 'volume_create_manager' (170b23e9-34e9-4d92-a86a-80fe23904fd8) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:23:29 user cinder-volume[84696]: INFO cinder.volume.manager [None req-ff28d70d-3957-4e48-97d2-741aa067698b tempest-GroupSnapshotsTest-1926061586 None] Created volume successfully. Apr 17 15:23:29 user cinder-volume[84696]: DEBUG cinder.coordination [None req-bf4316d4-7553-4c02-9fad-8509117f5729 tempest-VolumeQuotasNegativeTestJSON-2121978028 None] Acquiring lock "cinder-90683e1b-58ef-42b7-8ebf-85f478ece679-delete_volume" by "delete_volume" {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:192}} Apr 17 15:23:29 user cinder-volume[84696]: DEBUG cinder.coordination [None req-bf4316d4-7553-4c02-9fad-8509117f5729 tempest-VolumeQuotasNegativeTestJSON-2121978028 None] Lock "cinder-90683e1b-58ef-42b7-8ebf-85f478ece679-delete_volume" acquired by "delete_volume" :: waited 0.011s {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:196}} Apr 17 15:23:29 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-bf4316d4-7553-4c02-9fad-8509117f5729 tempest-VolumeQuotasNegativeTestJSON-2121978028 None] remove_export for volume: 90683e1b-58ef-42b7-8ebf-85f478ece679. {{(pid=84696) remove_export /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:718}} Apr 17 15:23:29 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-bf4316d4-7553-4c02-9fad-8509117f5729 tempest-VolumeQuotasNegativeTestJSON-2121978028 None] remove export {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:912}} Apr 17 15:23:29 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-bf4316d4-7553-4c02-9fad-8509117f5729 tempest-VolumeQuotasNegativeTestJSON-2121978028 None] detach volume:Volume(_name_id=None,admin_metadata={},attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:23:26Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumeQuotasNegativeTestJSON-Volume-492980821',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=90683e1b-58ef-42b7-8ebf-85f478ece679,launched_at=2023-04-17T15:23:28Z,metadata={},migration_status=None,multiattach=False,previous_status='available',project_id='ff79d745ac3b45da90c1cca9fcba6b9c',provider_auth='CHAP PeZBvMjF ZQFBVbYW6aOcvd',provider_geometry=None,provider_id=None,provider_location='172.16.0.220:3260,1 iqn.2010-10.org.openstack:90683e1b-58ef-42b7-8ebf-85f478ece679 0',replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:23:26Z,service_uuid='e628352e-c4cf-40eb-8389-cce40663026a',shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='deleting',terminated_at=2023-04-17T15:23:30Z,updated_at=2023-04-17T15:23:30Z,use_quota=True,user_id='6ce046152381477e89b03a5d253a0047',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c) from target:iqn.2010-10.org.openstack:90683e1b-58ef-42b7-8ebf-85f478ece679. {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:913}} Apr 17 15:23:29 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-bf4316d4-7553-4c02-9fad-8509117f5729 tempest-VolumeQuotasNegativeTestJSON-2121978028 None] detach volume v_90683e1b-58ef-42b7-8ebf-85f478ece679 from target iqn.2010-10.org.openstack:90683e1b-58ef-42b7-8ebf-85f478ece679 {{(pid=84696) detach_target_vol /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:606}} Apr 17 15:23:29 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-bf4316d4-7553-4c02-9fad-8509117f5729 tempest-VolumeQuotasNegativeTestJSON-2121978028 None] Sending pool request DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:90683e1b-58ef-42b7-8ebf-85f478ece679/luns/v_90683e1b-58ef-42b7-8ebf-85f478ece679 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:23:29 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-bf4316d4-7553-4c02-9fad-8509117f5729 tempest-VolumeQuotasNegativeTestJSON-2121978028 None] Sending DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:90683e1b-58ef-42b7-8ebf-85f478ece679/luns/v_90683e1b-58ef-42b7-8ebf-85f478ece679 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:23:29 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-bf4316d4-7553-4c02-9fad-8509117f5729 tempest-VolumeQuotasNegativeTestJSON-2121978028 None] Geting {'code': 404, 'error': {'class': 'opene.exceptions.ItemNotFoundError', 'message': 'Target iqn.2010-10.org.openstack:90683e1b-58ef-42b7-8ebf-85f478ece679 not exists.', 'url': 'http://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:90683e1b-58ef-42b7-8ebf-85f478ece679/luns/v_90683e1b-58ef-42b7-8ebf-85f478ece679'}, 'data': None} from DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:90683e1b-58ef-42b7-8ebf-85f478ece679/luns/v_90683e1b-58ef-42b7-8ebf-85f478ece679 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:23:29 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-bf4316d4-7553-4c02-9fad-8509117f5729 tempest-VolumeQuotasNegativeTestJSON-2121978028 None] failed to remove resource iqn.2010-10.org.openstack:90683e1b-58ef-42b7-8ebf-85f478ece679 because of JDSS resource v_90683e1b-58ef-42b7-8ebf-85f478ece679 DNE. {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:920}} Apr 17 15:23:29 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-bf4316d4-7553-4c02-9fad-8509117f5729 tempest-VolumeQuotasNegativeTestJSON-2121978028 None] delete target: iqn.2010-10.org.openstack:90683e1b-58ef-42b7-8ebf-85f478ece679 {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:930}} Apr 17 15:23:29 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-bf4316d4-7553-4c02-9fad-8509117f5729 tempest-VolumeQuotasNegativeTestJSON-2121978028 None] delete target iqn.2010-10.org.openstack:90683e1b-58ef-42b7-8ebf-85f478ece679 {{(pid=84696) delete_target /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:442}} Apr 17 15:23:29 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-bf4316d4-7553-4c02-9fad-8509117f5729 tempest-VolumeQuotasNegativeTestJSON-2121978028 None] Sending pool request DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:90683e1b-58ef-42b7-8ebf-85f478ece679 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:23:29 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-bf4316d4-7553-4c02-9fad-8509117f5729 tempest-VolumeQuotasNegativeTestJSON-2121978028 None] Sending DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:90683e1b-58ef-42b7-8ebf-85f478ece679 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:23:30 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-bf4316d4-7553-4c02-9fad-8509117f5729 tempest-VolumeQuotasNegativeTestJSON-2121978028 None] Geting {'code': 404, 'error': {'class': 'opene.exceptions.ItemNotFoundError', 'message': 'Target iqn.2010-10.org.openstack:90683e1b-58ef-42b7-8ebf-85f478ece679 not exists.', 'url': 'http://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:90683e1b-58ef-42b7-8ebf-85f478ece679'}, 'data': None} from DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:90683e1b-58ef-42b7-8ebf-85f478ece679 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:23:30 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-bf4316d4-7553-4c02-9fad-8509117f5729 tempest-VolumeQuotasNegativeTestJSON-2121978028 None] failed to remove resource iqn.2010-10.org.openstack:90683e1b-58ef-42b7-8ebf-85f478ece679 because of JDSS resource iqn.2010-10.org.openstack:90683e1b-58ef-42b7-8ebf-85f478ece679 DNE. {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:935}} Apr 17 15:23:30 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-bf4316d4-7553-4c02-9fad-8509117f5729 tempest-VolumeQuotasNegativeTestJSON-2121978028 None] deleating volume v_90683e1b-58ef-42b7-8ebf-85f478ece679 {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 15:23:30 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-bf4316d4-7553-4c02-9fad-8509117f5729 tempest-VolumeQuotasNegativeTestJSON-2121978028 None] get snapshots for volume v_90683e1b-58ef-42b7-8ebf-85f478ece679 {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 15:23:30 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-bf4316d4-7553-4c02-9fad-8509117f5729 tempest-VolumeQuotasNegativeTestJSON-2121978028 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_90683e1b-58ef-42b7-8ebf-85f478ece679/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:23:30 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-bf4316d4-7553-4c02-9fad-8509117f5729 tempest-VolumeQuotasNegativeTestJSON-2121978028 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_90683e1b-58ef-42b7-8ebf-85f478ece679/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:23:30 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-7cf58784-5a09-4291-b5b9-ad092863a300 tempest-VolumesTestJSON-1329991747 None] Geting {'code': 201, 'error': None, 'data': {'full_name': 'Pool-0/v_f8f9aea9-6773-4928-9e8e-153c3796301b', 'is_clone': False, 'name': 'v_f8f9aea9-6773-4928-9e8e-153c3796301b'}} from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:23:30 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-7cf58784-5a09-4291-b5b9-ad092863a300 tempest-VolumesTestJSON-1329991747 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (2e81da61-388c-409b-9a56-f19dedcec4d0) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'volume_name': 'volume-f8f9aea9-6773-4928-9e8e-153c3796301b', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:23:30 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-7cf58784-5a09-4291-b5b9-ad092863a300 tempest-VolumesTestJSON-1329991747 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeOnFinishTask;volume:create, create.end' (6fabdee1-0fd1-4a9d-9588-2680345b67fc) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:23:30 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-7cf58784-5a09-4291-b5b9-ad092863a300 tempest-VolumesTestJSON-1329991747 None] Volume volume-f8f9aea9-6773-4928-9e8e-153c3796301b (f8f9aea9-6773-4928-9e8e-153c3796301b): created successfully Apr 17 15:23:30 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-7cf58784-5a09-4291-b5b9-ad092863a300 tempest-VolumesTestJSON-1329991747 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeOnFinishTask;volume:create, create.end' (6fabdee1-0fd1-4a9d-9588-2680345b67fc) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:23:30 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-7cf58784-5a09-4291-b5b9-ad092863a300 tempest-VolumesTestJSON-1329991747 None] Flow 'volume_create_manager' (74cdd190-9b5c-49cf-b73d-b498ab2c2320) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:23:30 user cinder-volume[84696]: INFO cinder.volume.manager [None req-7cf58784-5a09-4291-b5b9-ad092863a300 tempest-VolumesTestJSON-1329991747 None] Created volume successfully. Apr 17 15:23:30 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-bf4316d4-7553-4c02-9fad-8509117f5729 tempest-VolumeQuotasNegativeTestJSON-2121978028 None] Geting {'code': 200, 'error': None, 'data': {'results': 0, 'entries': []}} from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_90683e1b-58ef-42b7-8ebf-85f478ece679/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:23:30 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-bf4316d4-7553-4c02-9fad-8509117f5729 tempest-VolumeQuotasNegativeTestJSON-2121978028 None] get volume v_90683e1b-58ef-42b7-8ebf-85f478ece679 info {{(pid=84696) get_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:239}} Apr 17 15:23:30 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-bf4316d4-7553-4c02-9fad-8509117f5729 tempest-VolumeQuotasNegativeTestJSON-2121978028 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_90683e1b-58ef-42b7-8ebf-85f478ece679 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:23:30 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-bf4316d4-7553-4c02-9fad-8509117f5729 tempest-VolumeQuotasNegativeTestJSON-2121978028 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_90683e1b-58ef-42b7-8ebf-85f478ece679 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:23:30 user cinder-volume[84696]: INFO cinder.volume.manager [None req-3fb2d295-b8cf-4f2a-adbc-90d9a0f4416d tempest-GroupSnapshotsTest-1926061586 None] GroupSnapshot e790b696-22cc-4fac-8b67-57499e6de8b3: creating. Apr 17 15:23:30 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-bf4316d4-7553-4c02-9fad-8509117f5729 tempest-VolumeQuotasNegativeTestJSON-2121978028 None] Geting {'code': 200, 'error': None, 'data': {'origin': None, 'relatime': None, 'acltype': None, 'vscan': None, 'full_name': 'Pool-0/v_90683e1b-58ef-42b7-8ebf-85f478ece679', 'userrefs': None, 'primarycache': 'all', 'logbias': 'latency', 'creation': '1681746400', 'sync': 'always', 'is_clone': False, 'dedup': 'off', 'sharenfs': None, 'receive_resume_token': None, 'volsize': '1073741824', 'referenced': '57344', 'sharesmb': None, 'createtxg': '187793', 'reservation': '0', 'scontext': None, 'mountpoint': None, 'casesensitivity': None, 'guid': '12766039077361853775', 'usedbyrefreservation': '0', 'dnodesize': None, 'written': '57344', 'logicalused': '28672', 'compressratio': '1.00', 'rootcontext': 'none', 'default_scsi_id': '2bda165e5267987e', 'type': 'volume', 'compression': 'lz4', 'snapdir': None, 'overlay': None, 'encryption': 'off', 'xattr': None, 'volmode': 'default', 'copies': '1', 'snapshot_limit': '18446744073709551615', 'aclinherit': None, 'defcontext': 'none', 'readonly': 'off', 'version': None, 'recordsize': None, 'filesystem_limit': None, 'mounted': None, 'mlslabel': 'none', 'secondarycache': 'all', 'refreservation': '0', 'available': '113926529024', 'san:volume_id': '2bda165e5267987e32f6179e6743cc7ead48d666', 'encryptionroot': None, 'exec': None, 'refquota': None, 'refcompressratio': '1.00', 'quota': None, 'utf8only': None, 'keylocation': 'none', 'snapdev': 'hidden', 'snapshot_count': '18446744073709551615', 'fscontext': 'none', 'clones': None, 'canmount': None, 'keystatus': None, 'atime': None, 'usedbysnapshots': '0', 'normalization': None, 'usedbychildren': '0', 'volblocksize': '131072', 'usedbydataset': '57344', 'objsetid': '122885', 'name': 'v_90683e1b-58ef-42b7-8ebf-85f478ece679', 'defer_destroy': None, 'pbkdf2iters': '0', 'checksum': 'on', 'redundant_metadata': 'all', 'filesystem_count': None, 'devices': None, 'keyformat': 'none', 'setuid': None, 'used': '57344', 'logicalreferenced': '28672', 'context': 'none', 'zoned': None, 'nbmand': None}} from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_90683e1b-58ef-42b7-8ebf-85f478ece679 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:23:30 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-bf4316d4-7553-4c02-9fad-8509117f5729 tempest-VolumeQuotasNegativeTestJSON-2121978028 None] delete volume:v_90683e1b-58ef-42b7-8ebf-85f478ece679 recursively children:{'force_umount': True} {{(pid=84696) delete_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:335}} Apr 17 15:23:30 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-bf4316d4-7553-4c02-9fad-8509117f5729 tempest-VolumeQuotasNegativeTestJSON-2121978028 None] Sending pool request DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_90683e1b-58ef-42b7-8ebf-85f478ece679 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:23:30 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-bf4316d4-7553-4c02-9fad-8509117f5729 tempest-VolumeQuotasNegativeTestJSON-2121978028 None] Sending DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_90683e1b-58ef-42b7-8ebf-85f478ece679 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:23:30 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-3fb2d295-b8cf-4f2a-adbc-90d9a0f4416d tempest-GroupSnapshotsTest-1926061586 None] Group snapshot e790b696-22cc-4fac-8b67-57499e6de8b3: creating. {{(pid=84696) create_group_snapshot /opt/stack/cinder/cinder/volume/manager.py:4129}} Apr 17 15:23:30 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-3fb2d295-b8cf-4f2a-adbc-90d9a0f4416d tempest-GroupSnapshotsTest-1926061586 None] create snapshot f815ac77-8ec1-4c5c-a0a6-a00cb2779806 for volume 74f7ff47-86dd-433a-8a61-f46278efe3fd {{(pid=84696) create_snapshot /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:625}} Apr 17 15:23:30 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-3fb2d295-b8cf-4f2a-adbc-90d9a0f4416d tempest-GroupSnapshotsTest-1926061586 None] cloning v_74f7ff47-86dd-433a-8a61-f46278efe3fd to s_f815ac77-8ec1-4c5c-a0a6-a00cb2779806 {{(pid=84696) _clone_object /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:456}} Apr 17 15:23:30 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-3fb2d295-b8cf-4f2a-adbc-90d9a0f4416d tempest-GroupSnapshotsTest-1926061586 None] create snapshot s_f815ac77-8ec1-4c5c-a0a6-a00cb2779806 {{(pid=84696) create_snapshot /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:635}} Apr 17 15:23:30 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-3fb2d295-b8cf-4f2a-adbc-90d9a0f4416d tempest-GroupSnapshotsTest-1926061586 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_74f7ff47-86dd-433a-8a61-f46278efe3fd/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:23:30 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-3fb2d295-b8cf-4f2a-adbc-90d9a0f4416d tempest-GroupSnapshotsTest-1926061586 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_74f7ff47-86dd-433a-8a61-f46278efe3fd/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:23:32 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-088501df-e36c-4cae-82f0-c986294f5da9 tempest-VolumesTestJSON-1329991747 None] Volume reschedule parameters: True retry: {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 15:23:32 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-088501df-e36c-4cae-82f0-c986294f5da9 tempest-VolumesTestJSON-1329991747 None] Flow 'volume_create_manager' (e8e2b8aa-ccb8-4e2b-abe5-71885b436904) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:23:32 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-088501df-e36c-4cae-82f0-c986294f5da9 tempest-VolumesTestJSON-1329991747 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (431292b0-213e-44c2-9d3e-b4f240c68fab) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:23:32 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-088501df-e36c-4cae-82f0-c986294f5da9 tempest-VolumesTestJSON-1329991747 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (431292b0-213e-44c2-9d3e-b4f240c68fab) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:23:32Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesTestJSON-volume-1560547090',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=e03a489c-cc11-4ae5-bf0d-f270a6a5ee02,launched_at=None,metadata={Type='work'},migration_status=None,multiattach=False,previous_status=None,project_id='01ad54f0fc8a4200a62264f0f9c33107',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:23:32Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:23:32Z,use_quota=True,user_id='e8b77c4bc3b84389b237889bcd55f2af',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:23:32 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-088501df-e36c-4cae-82f0-c986294f5da9 tempest-VolumesTestJSON-1329991747 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (8fda280c-ed3e-4a88-94cf-b5ccec9d89e2) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:23:32 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-088501df-e36c-4cae-82f0-c986294f5da9 tempest-VolumesTestJSON-1329991747 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (8fda280c-ed3e-4a88-94cf-b5ccec9d89e2) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:23:32 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-088501df-e36c-4cae-82f0-c986294f5da9 tempest-VolumesTestJSON-1329991747 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (3736f5d9-0f1c-40f7-872d-e7470e4d256b) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:23:32 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-088501df-e36c-4cae-82f0-c986294f5da9 tempest-VolumesTestJSON-1329991747 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (3736f5d9-0f1c-40f7-872d-e7470e4d256b) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': 'e03a489c-cc11-4ae5-bf0d-f270a6a5ee02', 'volume_name': 'volume-e03a489c-cc11-4ae5-bf0d-f270a6a5ee02', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:23:32 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-088501df-e36c-4cae-82f0-c986294f5da9 tempest-VolumesTestJSON-1329991747 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (29732109-9bee-4f57-afac-fba5163265fb) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:23:32 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-088501df-e36c-4cae-82f0-c986294f5da9 tempest-VolumesTestJSON-1329991747 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (29732109-9bee-4f57-afac-fba5163265fb) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:23:32 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-088501df-e36c-4cae-82f0-c986294f5da9 tempest-VolumesTestJSON-1329991747 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (20753b12-ccb1-4fa8-950c-e42d7ad30316) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:23:32 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-088501df-e36c-4cae-82f0-c986294f5da9 tempest-VolumesTestJSON-1329991747 None] Volume e03a489c-cc11-4ae5-bf0d-f270a6a5ee02: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-e03a489c-cc11-4ae5-bf0d-f270a6a5ee02', 'volume_size': 1} Apr 17 15:23:32 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-088501df-e36c-4cae-82f0-c986294f5da9 tempest-VolumesTestJSON-1329991747 None] creating volume v_e03a489c-cc11-4ae5-bf0d-f270a6a5ee02. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 15:23:32 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-088501df-e36c-4cae-82f0-c986294f5da9 tempest-VolumesTestJSON-1329991747 None] create volume {'name': 'v_e03a489c-cc11-4ae5-bf0d-f270a6a5ee02', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 15:23:32 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-088501df-e36c-4cae-82f0-c986294f5da9 tempest-VolumesTestJSON-1329991747 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:23:32 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-088501df-e36c-4cae-82f0-c986294f5da9 tempest-VolumesTestJSON-1329991747 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:23:32 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-3fb2d295-b8cf-4f2a-adbc-90d9a0f4416d tempest-GroupSnapshotsTest-1926061586 None] Geting {'code': 200, 'error': None, 'data': {'name': 's_f815ac77-8ec1-4c5c-a0a6-a00cb2779806'}} from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_74f7ff47-86dd-433a-8a61-f46278efe3fd/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:23:32 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-3fb2d295-b8cf-4f2a-adbc-90d9a0f4416d tempest-GroupSnapshotsTest-1926061586 None] create volume s_f815ac77-8ec1-4c5c-a0a6-a00cb2779806 from snapshot s_f815ac77-8ec1-4c5c-a0a6-a00cb2779806 {{(pid=84696) create_volume_from_snapshot /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:674}} Apr 17 15:23:32 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-3fb2d295-b8cf-4f2a-adbc-90d9a0f4416d tempest-GroupSnapshotsTest-1926061586 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_74f7ff47-86dd-433a-8a61-f46278efe3fd/clone {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:23:32 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-3fb2d295-b8cf-4f2a-adbc-90d9a0f4416d tempest-GroupSnapshotsTest-1926061586 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_74f7ff47-86dd-433a-8a61-f46278efe3fd/clone {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:23:33 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-088501df-e36c-4cae-82f0-c986294f5da9 tempest-VolumesTestJSON-1329991747 None] Geting {'code': 201, 'error': None, 'data': {'full_name': 'Pool-0/v_e03a489c-cc11-4ae5-bf0d-f270a6a5ee02', 'is_clone': False, 'name': 'v_e03a489c-cc11-4ae5-bf0d-f270a6a5ee02'}} from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:23:33 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-088501df-e36c-4cae-82f0-c986294f5da9 tempest-VolumesTestJSON-1329991747 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (20753b12-ccb1-4fa8-950c-e42d7ad30316) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'volume_name': 'volume-e03a489c-cc11-4ae5-bf0d-f270a6a5ee02', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:23:33 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-088501df-e36c-4cae-82f0-c986294f5da9 tempest-VolumesTestJSON-1329991747 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeOnFinishTask;volume:create, create.end' (f2298e3f-8ce8-4b04-8d5d-92c662d48926) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:23:33 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-088501df-e36c-4cae-82f0-c986294f5da9 tempest-VolumesTestJSON-1329991747 None] Volume volume-e03a489c-cc11-4ae5-bf0d-f270a6a5ee02 (e03a489c-cc11-4ae5-bf0d-f270a6a5ee02): created successfully Apr 17 15:23:33 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-088501df-e36c-4cae-82f0-c986294f5da9 tempest-VolumesTestJSON-1329991747 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeOnFinishTask;volume:create, create.end' (f2298e3f-8ce8-4b04-8d5d-92c662d48926) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:23:33 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-088501df-e36c-4cae-82f0-c986294f5da9 tempest-VolumesTestJSON-1329991747 None] Flow 'volume_create_manager' (e8e2b8aa-ccb8-4e2b-abe5-71885b436904) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:23:33 user cinder-volume[84696]: INFO cinder.volume.manager [None req-088501df-e36c-4cae-82f0-c986294f5da9 tempest-VolumesTestJSON-1329991747 None] Created volume successfully. Apr 17 15:23:33 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-3fb2d295-b8cf-4f2a-adbc-90d9a0f4416d tempest-GroupSnapshotsTest-1926061586 None] Geting {'code': 200, 'error': None, 'data': {'origin': 'Pool-0/v_74f7ff47-86dd-433a-8a61-f46278efe3fd@s_f815ac77-8ec1-4c5c-a0a6-a00cb2779806', 'is_clone': True, 'full_name': 'Pool-0/s_f815ac77-8ec1-4c5c-a0a6-a00cb2779806', 'name': 's_f815ac77-8ec1-4c5c-a0a6-a00cb2779806'}} from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_74f7ff47-86dd-433a-8a61-f46278efe3fd/clone {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:23:33 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-3fb2d295-b8cf-4f2a-adbc-90d9a0f4416d tempest-GroupSnapshotsTest-1926061586 None] Sending pool request PUT to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/s_f815ac77-8ec1-4c5c-a0a6-a00cb2779806/properties {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:23:33 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-3fb2d295-b8cf-4f2a-adbc-90d9a0f4416d tempest-GroupSnapshotsTest-1926061586 None] Sending PUT to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/s_f815ac77-8ec1-4c5c-a0a6-a00cb2779806/properties {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:23:33 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-3fb2d295-b8cf-4f2a-adbc-90d9a0f4416d tempest-GroupSnapshotsTest-1926061586 None] Geting {'code': 201, 'error': None, 'data': None} from PUT to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/s_f815ac77-8ec1-4c5c-a0a6-a00cb2779806/properties {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:23:33 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-3fb2d295-b8cf-4f2a-adbc-90d9a0f4416d tempest-GroupSnapshotsTest-1926061586 None] volume s_f815ac77-8ec1-4c5c-a0a6-a00cb2779806 properties updated {{(pid=84696) modify_property_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:301}} Apr 17 15:23:33 user cinder-volume[84696]: INFO cinder.volume.manager [None req-3fb2d295-b8cf-4f2a-adbc-90d9a0f4416d tempest-GroupSnapshotsTest-1926061586 None] group_snapshot e790b696-22cc-4fac-8b67-57499e6de8b3: created successfully Apr 17 15:23:34 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-b4d0b80e-1b78-415e-ab7c-90da14a8eb7b tempest-GroupSnapshotsTest-1926061586 None] create volume 3b30abbe-d7f1-4e56-9991-a082bc596387 from snapshot f815ac77-8ec1-4c5c-a0a6-a00cb2779806 {{(pid=84696) create_volume_from_snapshot /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:574}} Apr 17 15:23:34 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-b4d0b80e-1b78-415e-ab7c-90da14a8eb7b tempest-GroupSnapshotsTest-1926061586 None] cloning s_f815ac77-8ec1-4c5c-a0a6-a00cb2779806 to v_3b30abbe-d7f1-4e56-9991-a082bc596387 {{(pid=84696) _clone_object /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:456}} Apr 17 15:23:34 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-b4d0b80e-1b78-415e-ab7c-90da14a8eb7b tempest-GroupSnapshotsTest-1926061586 None] create snapshot v_3b30abbe-d7f1-4e56-9991-a082bc596387 {{(pid=84696) create_snapshot /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:635}} Apr 17 15:23:34 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-b4d0b80e-1b78-415e-ab7c-90da14a8eb7b tempest-GroupSnapshotsTest-1926061586 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/s_f815ac77-8ec1-4c5c-a0a6-a00cb2779806/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:23:34 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-b4d0b80e-1b78-415e-ab7c-90da14a8eb7b tempest-GroupSnapshotsTest-1926061586 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/s_f815ac77-8ec1-4c5c-a0a6-a00cb2779806/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:23:35 user cinder-volume[84696]: DEBUG cinder.coordination [None req-21eb6b89-a742-42f1-9d2f-4bcf12a902bc tempest-VolumesTestJSON-1329991747 None] Acquiring lock "cinder-e03a489c-cc11-4ae5-bf0d-f270a6a5ee02-delete_volume" by "delete_volume" {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:192}} Apr 17 15:23:35 user cinder-volume[84696]: DEBUG cinder.coordination [None req-21eb6b89-a742-42f1-9d2f-4bcf12a902bc tempest-VolumesTestJSON-1329991747 None] Lock "cinder-e03a489c-cc11-4ae5-bf0d-f270a6a5ee02-delete_volume" acquired by "delete_volume" :: waited 0.011s {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:196}} Apr 17 15:23:35 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-21eb6b89-a742-42f1-9d2f-4bcf12a902bc tempest-VolumesTestJSON-1329991747 None] remove_export for volume: e03a489c-cc11-4ae5-bf0d-f270a6a5ee02. {{(pid=84696) remove_export /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:718}} Apr 17 15:23:35 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-21eb6b89-a742-42f1-9d2f-4bcf12a902bc tempest-VolumesTestJSON-1329991747 None] remove export {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:912}} Apr 17 15:23:35 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-21eb6b89-a742-42f1-9d2f-4bcf12a902bc tempest-VolumesTestJSON-1329991747 None] detach volume:Volume(_name_id=None,admin_metadata={},attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:23:32Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesTestJSON-volume-1560547090',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=e03a489c-cc11-4ae5-bf0d-f270a6a5ee02,launched_at=2023-04-17T15:23:33Z,metadata={Type='work'},migration_status=None,multiattach=False,previous_status=None,project_id='01ad54f0fc8a4200a62264f0f9c33107',provider_auth='CHAP SJLxBhDQ TEweczirtfDL8k',provider_geometry=None,provider_id=None,provider_location='172.16.0.220:3260,1 iqn.2010-10.org.openstack:e03a489c-cc11-4ae5-bf0d-f270a6a5ee02 0',replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:23:32Z,service_uuid='e628352e-c4cf-40eb-8389-cce40663026a',shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='deleting',terminated_at=2023-04-17T15:23:35Z,updated_at=2023-04-17T15:23:35Z,use_quota=True,user_id='e8b77c4bc3b84389b237889bcd55f2af',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c) from target:iqn.2010-10.org.openstack:e03a489c-cc11-4ae5-bf0d-f270a6a5ee02. {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:913}} Apr 17 15:23:35 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-21eb6b89-a742-42f1-9d2f-4bcf12a902bc tempest-VolumesTestJSON-1329991747 None] detach volume v_e03a489c-cc11-4ae5-bf0d-f270a6a5ee02 from target iqn.2010-10.org.openstack:e03a489c-cc11-4ae5-bf0d-f270a6a5ee02 {{(pid=84696) detach_target_vol /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:606}} Apr 17 15:23:35 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-21eb6b89-a742-42f1-9d2f-4bcf12a902bc tempest-VolumesTestJSON-1329991747 None] Sending pool request DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:e03a489c-cc11-4ae5-bf0d-f270a6a5ee02/luns/v_e03a489c-cc11-4ae5-bf0d-f270a6a5ee02 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:23:35 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-21eb6b89-a742-42f1-9d2f-4bcf12a902bc tempest-VolumesTestJSON-1329991747 None] Sending DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:e03a489c-cc11-4ae5-bf0d-f270a6a5ee02/luns/v_e03a489c-cc11-4ae5-bf0d-f270a6a5ee02 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:23:35 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-21eb6b89-a742-42f1-9d2f-4bcf12a902bc tempest-VolumesTestJSON-1329991747 None] Geting {'code': 404, 'error': {'class': 'opene.exceptions.ItemNotFoundError', 'message': 'Target iqn.2010-10.org.openstack:e03a489c-cc11-4ae5-bf0d-f270a6a5ee02 not exists.', 'url': 'http://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:e03a489c-cc11-4ae5-bf0d-f270a6a5ee02/luns/v_e03a489c-cc11-4ae5-bf0d-f270a6a5ee02'}, 'data': None} from DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:e03a489c-cc11-4ae5-bf0d-f270a6a5ee02/luns/v_e03a489c-cc11-4ae5-bf0d-f270a6a5ee02 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:23:35 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-21eb6b89-a742-42f1-9d2f-4bcf12a902bc tempest-VolumesTestJSON-1329991747 None] failed to remove resource iqn.2010-10.org.openstack:e03a489c-cc11-4ae5-bf0d-f270a6a5ee02 because of JDSS resource v_e03a489c-cc11-4ae5-bf0d-f270a6a5ee02 DNE. {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:920}} Apr 17 15:23:35 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-21eb6b89-a742-42f1-9d2f-4bcf12a902bc tempest-VolumesTestJSON-1329991747 None] delete target: iqn.2010-10.org.openstack:e03a489c-cc11-4ae5-bf0d-f270a6a5ee02 {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:930}} Apr 17 15:23:35 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-21eb6b89-a742-42f1-9d2f-4bcf12a902bc tempest-VolumesTestJSON-1329991747 None] delete target iqn.2010-10.org.openstack:e03a489c-cc11-4ae5-bf0d-f270a6a5ee02 {{(pid=84696) delete_target /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:442}} Apr 17 15:23:35 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-21eb6b89-a742-42f1-9d2f-4bcf12a902bc tempest-VolumesTestJSON-1329991747 None] Sending pool request DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:e03a489c-cc11-4ae5-bf0d-f270a6a5ee02 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:23:35 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-21eb6b89-a742-42f1-9d2f-4bcf12a902bc tempest-VolumesTestJSON-1329991747 None] Sending DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:e03a489c-cc11-4ae5-bf0d-f270a6a5ee02 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:23:35 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-21eb6b89-a742-42f1-9d2f-4bcf12a902bc tempest-VolumesTestJSON-1329991747 None] Geting {'code': 404, 'error': {'class': 'opene.exceptions.ItemNotFoundError', 'message': 'Target iqn.2010-10.org.openstack:e03a489c-cc11-4ae5-bf0d-f270a6a5ee02 not exists.', 'url': 'http://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:e03a489c-cc11-4ae5-bf0d-f270a6a5ee02'}, 'data': None} from DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:e03a489c-cc11-4ae5-bf0d-f270a6a5ee02 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:23:35 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-21eb6b89-a742-42f1-9d2f-4bcf12a902bc tempest-VolumesTestJSON-1329991747 None] failed to remove resource iqn.2010-10.org.openstack:e03a489c-cc11-4ae5-bf0d-f270a6a5ee02 because of JDSS resource iqn.2010-10.org.openstack:e03a489c-cc11-4ae5-bf0d-f270a6a5ee02 DNE. {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:935}} Apr 17 15:23:35 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-21eb6b89-a742-42f1-9d2f-4bcf12a902bc tempest-VolumesTestJSON-1329991747 None] deleating volume v_e03a489c-cc11-4ae5-bf0d-f270a6a5ee02 {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 15:23:35 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-21eb6b89-a742-42f1-9d2f-4bcf12a902bc tempest-VolumesTestJSON-1329991747 None] get snapshots for volume v_e03a489c-cc11-4ae5-bf0d-f270a6a5ee02 {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 15:23:35 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-21eb6b89-a742-42f1-9d2f-4bcf12a902bc tempest-VolumesTestJSON-1329991747 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_e03a489c-cc11-4ae5-bf0d-f270a6a5ee02/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:23:35 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-21eb6b89-a742-42f1-9d2f-4bcf12a902bc tempest-VolumesTestJSON-1329991747 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_e03a489c-cc11-4ae5-bf0d-f270a6a5ee02/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:23:36 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-b4d0b80e-1b78-415e-ab7c-90da14a8eb7b tempest-GroupSnapshotsTest-1926061586 None] Geting {'code': 200, 'error': None, 'data': {'name': 'v_3b30abbe-d7f1-4e56-9991-a082bc596387'}} from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/s_f815ac77-8ec1-4c5c-a0a6-a00cb2779806/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:23:36 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-b4d0b80e-1b78-415e-ab7c-90da14a8eb7b tempest-GroupSnapshotsTest-1926061586 None] create volume v_3b30abbe-d7f1-4e56-9991-a082bc596387 from snapshot v_3b30abbe-d7f1-4e56-9991-a082bc596387 {{(pid=84696) create_volume_from_snapshot /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:674}} Apr 17 15:23:36 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-b4d0b80e-1b78-415e-ab7c-90da14a8eb7b tempest-GroupSnapshotsTest-1926061586 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/s_f815ac77-8ec1-4c5c-a0a6-a00cb2779806/clone {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:23:36 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-b4d0b80e-1b78-415e-ab7c-90da14a8eb7b tempest-GroupSnapshotsTest-1926061586 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/s_f815ac77-8ec1-4c5c-a0a6-a00cb2779806/clone {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:23:36 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-21eb6b89-a742-42f1-9d2f-4bcf12a902bc tempest-VolumesTestJSON-1329991747 None] Geting {'code': 200, 'error': None, 'data': {'results': 0, 'entries': []}} from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_e03a489c-cc11-4ae5-bf0d-f270a6a5ee02/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:23:36 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-21eb6b89-a742-42f1-9d2f-4bcf12a902bc tempest-VolumesTestJSON-1329991747 None] get volume v_e03a489c-cc11-4ae5-bf0d-f270a6a5ee02 info {{(pid=84696) get_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:239}} Apr 17 15:23:36 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-21eb6b89-a742-42f1-9d2f-4bcf12a902bc tempest-VolumesTestJSON-1329991747 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_e03a489c-cc11-4ae5-bf0d-f270a6a5ee02 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:23:36 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-21eb6b89-a742-42f1-9d2f-4bcf12a902bc tempest-VolumesTestJSON-1329991747 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_e03a489c-cc11-4ae5-bf0d-f270a6a5ee02 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:23:36 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-21eb6b89-a742-42f1-9d2f-4bcf12a902bc tempest-VolumesTestJSON-1329991747 None] Geting {'code': 200, 'error': None, 'data': {'origin': None, 'relatime': None, 'acltype': None, 'vscan': None, 'full_name': 'Pool-0/v_e03a489c-cc11-4ae5-bf0d-f270a6a5ee02', 'userrefs': None, 'primarycache': 'all', 'logbias': 'latency', 'creation': '1681746406', 'sync': 'always', 'is_clone': False, 'dedup': 'off', 'sharenfs': None, 'receive_resume_token': None, 'volsize': '1073741824', 'referenced': '57344', 'sharesmb': None, 'createtxg': '187813', 'reservation': '0', 'scontext': None, 'mountpoint': None, 'casesensitivity': None, 'guid': '12985110274713030039', 'usedbyrefreservation': '0', 'dnodesize': None, 'written': '57344', 'logicalused': '28672', 'compressratio': '1.00', 'rootcontext': 'none', 'default_scsi_id': 'd34293ff49c6c9a5', 'type': 'volume', 'compression': 'lz4', 'snapdir': None, 'overlay': None, 'encryption': 'off', 'xattr': None, 'volmode': 'default', 'copies': '1', 'snapshot_limit': '18446744073709551615', 'aclinherit': None, 'defcontext': 'none', 'readonly': 'off', 'version': None, 'recordsize': None, 'filesystem_limit': None, 'mounted': None, 'mlslabel': 'none', 'secondarycache': 'all', 'refreservation': '0', 'available': '113926479872', 'san:volume_id': 'd34293ff49c6c9a5baed7ff4f041d58ac06b34da', 'encryptionroot': None, 'exec': None, 'refquota': None, 'refcompressratio': '1.00', 'quota': None, 'utf8only': None, 'keylocation': 'none', 'snapdev': 'hidden', 'snapshot_count': '18446744073709551615', 'fscontext': 'none', 'clones': None, 'canmount': None, 'keystatus': None, 'atime': None, 'usedbysnapshots': '0', 'normalization': None, 'usedbychildren': '0', 'volblocksize': '131072', 'usedbydataset': '57344', 'objsetid': '122688', 'name': 'v_e03a489c-cc11-4ae5-bf0d-f270a6a5ee02', 'defer_destroy': None, 'pbkdf2iters': '0', 'checksum': 'on', 'redundant_metadata': 'all', 'filesystem_count': None, 'devices': None, 'keyformat': 'none', 'setuid': None, 'used': '57344', 'logicalreferenced': '28672', 'context': 'none', 'zoned': None, 'nbmand': None}} from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_e03a489c-cc11-4ae5-bf0d-f270a6a5ee02 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:23:36 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-21eb6b89-a742-42f1-9d2f-4bcf12a902bc tempest-VolumesTestJSON-1329991747 None] delete volume:v_e03a489c-cc11-4ae5-bf0d-f270a6a5ee02 recursively children:{'force_umount': True} {{(pid=84696) delete_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:335}} Apr 17 15:23:36 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-21eb6b89-a742-42f1-9d2f-4bcf12a902bc tempest-VolumesTestJSON-1329991747 None] Sending pool request DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_e03a489c-cc11-4ae5-bf0d-f270a6a5ee02 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:23:36 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-21eb6b89-a742-42f1-9d2f-4bcf12a902bc tempest-VolumesTestJSON-1329991747 None] Sending DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_e03a489c-cc11-4ae5-bf0d-f270a6a5ee02 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:23:36 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-bf4316d4-7553-4c02-9fad-8509117f5729 tempest-VolumeQuotasNegativeTestJSON-2121978028 None] Geting {'code': 204} from DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_90683e1b-58ef-42b7-8ebf-85f478ece679 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:23:36 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-bf4316d4-7553-4c02-9fad-8509117f5729 tempest-VolumeQuotasNegativeTestJSON-2121978028 None] volume v_90683e1b-58ef-42b7-8ebf-85f478ece679 deleted {{(pid=84696) delete_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:346}} Apr 17 15:23:36 user cinder-volume[84696]: WARNING cinder.quota [None req-bf4316d4-7553-4c02-9fad-8509117f5729 tempest-VolumeQuotasNegativeTestJSON-2121978028 None] Deprecated: Default quota for resource: volumes_joviandss-2 is set by the default quota flag: quota_volumes_joviandss-2, it is now deprecated. Please use the default quota class for default quota. Apr 17 15:23:36 user cinder-volume[84696]: WARNING cinder.quota [None req-bf4316d4-7553-4c02-9fad-8509117f5729 tempest-VolumeQuotasNegativeTestJSON-2121978028 None] Deprecated: Default quota for resource: gigabytes_joviandss-2 is set by the default quota flag: quota_gigabytes_joviandss-2, it is now deprecated. Please use the default quota class for default quota. Apr 17 15:23:36 user cinder-volume[84696]: DEBUG cinder.quota [None req-bf4316d4-7553-4c02-9fad-8509117f5729 tempest-VolumeQuotasNegativeTestJSON-2121978028 None] Created reservations ['65146948-2185-42ed-a45e-b56432ade911', '5a31e0fc-c2e0-4a66-9b93-b0d3b25b38fa', 'eee83985-c01e-41ca-9b2f-21808cedab9b', '61e9fcb8-a3d2-4877-b9f2-1a4bd3c4af04'] {{(pid=84696) reserve /opt/stack/cinder/cinder/quota.py:755}} Apr 17 15:23:36 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-bf4316d4-7553-4c02-9fad-8509117f5729 tempest-VolumeQuotasNegativeTestJSON-2121978028 None] Updating volume stats {{(pid=84696) _update_volume_stats /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:724}} Apr 17 15:23:36 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-bf4316d4-7553-4c02-9fad-8509117f5729 tempest-VolumeQuotasNegativeTestJSON-2121978028 None] Get pool Pool-0 fsprops {{(pid=84696) get_pool_stats /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:894}} Apr 17 15:23:36 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-bf4316d4-7553-4c02-9fad-8509117f5729 tempest-VolumeQuotasNegativeTestJSON-2121978028 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:23:36 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-bf4316d4-7553-4c02-9fad-8509117f5729 tempest-VolumeQuotasNegativeTestJSON-2121978028 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:23:37 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-b4d0b80e-1b78-415e-ab7c-90da14a8eb7b tempest-GroupSnapshotsTest-1926061586 None] Geting {'code': 200, 'error': None, 'data': {'origin': 'Pool-0/s_f815ac77-8ec1-4c5c-a0a6-a00cb2779806@v_3b30abbe-d7f1-4e56-9991-a082bc596387', 'is_clone': True, 'full_name': 'Pool-0/v_3b30abbe-d7f1-4e56-9991-a082bc596387', 'name': 'v_3b30abbe-d7f1-4e56-9991-a082bc596387'}} from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/s_f815ac77-8ec1-4c5c-a0a6-a00cb2779806/clone {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:23:37 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-b4d0b80e-1b78-415e-ab7c-90da14a8eb7b tempest-GroupSnapshotsTest-1926061586 None] get volume v_3b30abbe-d7f1-4e56-9991-a082bc596387 info {{(pid=84696) get_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:239}} Apr 17 15:23:37 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-b4d0b80e-1b78-415e-ab7c-90da14a8eb7b tempest-GroupSnapshotsTest-1926061586 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_3b30abbe-d7f1-4e56-9991-a082bc596387 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:23:37 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-b4d0b80e-1b78-415e-ab7c-90da14a8eb7b tempest-GroupSnapshotsTest-1926061586 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_3b30abbe-d7f1-4e56-9991-a082bc596387 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:23:37 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-bf4316d4-7553-4c02-9fad-8509117f5729 tempest-VolumeQuotasNegativeTestJSON-2121978028 None] Geting {'code': 200, 'error': None, 'data': {'available': '113926479872', 'status': 30, 'import_status': {'import_steps': {'san_setup': True, 'vip_setup': True, 'zfs_import': True, 'mount_system_volume': True, 'nas_setup': True}, 'import_successful': True}, 'scan': None, 'encryption': {'enabled': False}, 'deduplication_rate': '1.00', 'sysvol_upgrade': 'not_needed', 'vdevs': [{'name': 'wwn-0x6000c296b793bf526a107a9d823e947a', 'iostats': {'read': '0', 'write': '0', 'chksum': '0'}, 'disks': [{'origin': 'local', 'slot': '', 'led': 'off', 'name': 'sdn', 'iostats': {'read': '0', 'write': '0', 'chksum': '0'}, 'alias': '', 'health': 'ONLINE', 'sn': '6000c296b793bf526a107a9d823e947a', 'trim_data': {'status': 'trim_unsupported', 'progress': None, 'trim_time': None}, 'path': 'pci-0000:03:00.0-scsi-0:0:14:0', 'model': 'Virtual disk', 'id': 'wwn-0x6000c296b793bf526a107a9d823e947a', 'size': 128849018880}], 'health': 'ONLINE', 'vdev_replacings': [], 'vdev_spares': [], 'type': ''}], 'id': '14181317334440137330', 'health': 'ONLINE', 'iostats': {'read': '0', 'write': '0', 'chksum': '0'}, 'operation': 'none', 'size': '127775277056', 'autotrim': False, 'name': 'Pool-0'}} from GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:23:37 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-bf4316d4-7553-4c02-9fad-8509117f5729 tempest-VolumeQuotasNegativeTestJSON-2121978028 None] Total capacity: 119, Free 106. {{(pid=84696) _update_volume_stats /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:757}} Apr 17 15:23:37 user cinder-volume[84696]: DEBUG cinder.manager [None req-bf4316d4-7553-4c02-9fad-8509117f5729 tempest-VolumeQuotasNegativeTestJSON-2121978028 None] Notifying Schedulers of capabilities ... {{(pid=84696) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 17 15:23:37 user cinder-volume[84696]: INFO cinder.volume.manager [None req-bf4316d4-7553-4c02-9fad-8509117f5729 tempest-VolumeQuotasNegativeTestJSON-2121978028 None] Deleted volume successfully. Apr 17 15:23:37 user cinder-volume[84696]: DEBUG cinder.coordination [None req-bf4316d4-7553-4c02-9fad-8509117f5729 tempest-VolumeQuotasNegativeTestJSON-2121978028 None] Lock "cinder-90683e1b-58ef-42b7-8ebf-85f478ece679-delete_volume" released by "delete_volume" :: held 7.647s {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:207}} Apr 17 15:23:37 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-b4d0b80e-1b78-415e-ab7c-90da14a8eb7b tempest-GroupSnapshotsTest-1926061586 None] Geting {'code': 200, 'error': None, 'data': {'origin': 'Pool-0/s_f815ac77-8ec1-4c5c-a0a6-a00cb2779806@v_3b30abbe-d7f1-4e56-9991-a082bc596387', 'relatime': None, 'acltype': None, 'vscan': None, 'full_name': 'Pool-0/v_3b30abbe-d7f1-4e56-9991-a082bc596387', 'userrefs': None, 'primarycache': 'all', 'logbias': 'latency', 'creation': '1681746410', 'sync': 'always', 'is_clone': True, 'dedup': 'off', 'sharenfs': None, 'receive_resume_token': None, 'volsize': '1073741824', 'referenced': '57344', 'sharesmb': None, 'createtxg': '187820', 'reservation': '0', 'scontext': None, 'mountpoint': None, 'casesensitivity': None, 'guid': '1574029390680739449', 'usedbyrefreservation': '0', 'dnodesize': None, 'written': '0', 'logicalused': '0', 'compressratio': '1.00', 'rootcontext': 'none', 'default_scsi_id': '62b97efd26da3fb0', 'type': 'volume', 'compression': 'lz4', 'snapdir': None, 'overlay': None, 'encryption': 'off', 'xattr': None, 'volmode': 'default', 'copies': '1', 'snapshot_limit': '18446744073709551615', 'aclinherit': None, 'defcontext': 'none', 'readonly': 'off', 'version': None, 'recordsize': None, 'filesystem_limit': None, 'mounted': None, 'mlslabel': 'none', 'secondarycache': 'all', 'refreservation': '0', 'available': '113926479872', 'used': '0', 'encryptionroot': None, 'exec': None, 'refquota': None, 'refcompressratio': '1.00', 'quota': None, 'utf8only': None, 'keylocation': 'none', 'snapdev': 'hidden', 'snapshot_count': '18446744073709551615', 'fscontext': 'none', 'clones': None, 'canmount': None, 'keystatus': None, 'atime': None, 'usedbysnapshots': '0', 'normalization': None, 'usedbychildren': '0', 'volblocksize': '131072', 'usedbydataset': '0', 'objsetid': '122894', 'name': 'v_3b30abbe-d7f1-4e56-9991-a082bc596387', 'defer_destroy': None, 'pbkdf2iters': '0', 'checksum': 'on', 'redundant_metadata': 'all', 'filesystem_count': None, 'devices': None, 'keyformat': 'none', 'setuid': None, 'logicalreferenced': '28672', 'context': 'none', 'zoned': None, 'nbmand': None}} from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_3b30abbe-d7f1-4e56-9991-a082bc596387 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:23:37 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=74,cinder:UPDATE=22,cinder:DELETE=3,cinder:INSERT=1 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:23:37 user cinder-volume[84696]: INFO cinder.volume.manager [None req-b4d0b80e-1b78-415e-ab7c-90da14a8eb7b tempest-GroupSnapshotsTest-1926061586 None] Create group from source-snapshot-e790b696-22cc-4fac-8b67-57499e6de8b3 completed successfully. Apr 17 15:23:38 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-8991a11b-9d09-425a-8267-3a06c24ef669 tempest-GroupSnapshotsTest-1926061586 None] remove_export for volume: 3b30abbe-d7f1-4e56-9991-a082bc596387. {{(pid=84696) remove_export /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:718}} Apr 17 15:23:38 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-8991a11b-9d09-425a-8267-3a06c24ef669 tempest-GroupSnapshotsTest-1926061586 None] remove export {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:912}} Apr 17 15:23:38 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-8991a11b-9d09-425a-8267-3a06c24ef669 tempest-GroupSnapshotsTest-1926061586 None] detach volume:Volume(_name_id=None,admin_metadata={},attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:23:38Z,deleted=False,deleted_at=None,display_description=None,display_name=None,ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=ae8aa7f8-08a9-46a8-ab4e-645c6629023c,host='user@joviandss-2#joviandss-2',id=3b30abbe-d7f1-4e56-9991-a082bc596387,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='6cae12f3c2614d6fb32f21db60fec8d3',provider_auth='CHAP AWSvRVYb ofSDlsY85xt7Rh',provider_geometry=None,provider_id=None,provider_location='172.16.0.220:3260,1 iqn.2010-10.org.openstack:3b30abbe-d7f1-4e56-9991-a082bc596387 0',replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=None,service_uuid=None,shared_targets=True,size=1,snapshot_id=f815ac77-8ec1-4c5c-a0a6-a00cb2779806,snapshots=,source_volid=None,status='deleting',terminated_at=None,updated_at=2023-04-17T15:23:39Z,use_quota=True,user_id='560c850c324949d7a214e7a59dd65f8d',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(a40cb290-464c-4f83-9d6b-adf225cc2e0c),volume_type_id=a40cb290-464c-4f83-9d6b-adf225cc2e0c) from target:iqn.2010-10.org.openstack:3b30abbe-d7f1-4e56-9991-a082bc596387. {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:913}} Apr 17 15:23:38 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-8991a11b-9d09-425a-8267-3a06c24ef669 tempest-GroupSnapshotsTest-1926061586 None] detach volume v_3b30abbe-d7f1-4e56-9991-a082bc596387 from target iqn.2010-10.org.openstack:3b30abbe-d7f1-4e56-9991-a082bc596387 {{(pid=84696) detach_target_vol /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:606}} Apr 17 15:23:38 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-8991a11b-9d09-425a-8267-3a06c24ef669 tempest-GroupSnapshotsTest-1926061586 None] Sending pool request DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:3b30abbe-d7f1-4e56-9991-a082bc596387/luns/v_3b30abbe-d7f1-4e56-9991-a082bc596387 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:23:38 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-8991a11b-9d09-425a-8267-3a06c24ef669 tempest-GroupSnapshotsTest-1926061586 None] Sending DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:3b30abbe-d7f1-4e56-9991-a082bc596387/luns/v_3b30abbe-d7f1-4e56-9991-a082bc596387 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:23:38 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-8991a11b-9d09-425a-8267-3a06c24ef669 tempest-GroupSnapshotsTest-1926061586 None] Geting {'code': 404, 'error': {'class': 'opene.exceptions.ItemNotFoundError', 'message': 'Target iqn.2010-10.org.openstack:3b30abbe-d7f1-4e56-9991-a082bc596387 not exists.', 'url': 'http://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:3b30abbe-d7f1-4e56-9991-a082bc596387/luns/v_3b30abbe-d7f1-4e56-9991-a082bc596387'}, 'data': None} from DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:3b30abbe-d7f1-4e56-9991-a082bc596387/luns/v_3b30abbe-d7f1-4e56-9991-a082bc596387 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:23:38 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-8991a11b-9d09-425a-8267-3a06c24ef669 tempest-GroupSnapshotsTest-1926061586 None] failed to remove resource iqn.2010-10.org.openstack:3b30abbe-d7f1-4e56-9991-a082bc596387 because of JDSS resource v_3b30abbe-d7f1-4e56-9991-a082bc596387 DNE. {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:920}} Apr 17 15:23:38 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-8991a11b-9d09-425a-8267-3a06c24ef669 tempest-GroupSnapshotsTest-1926061586 None] delete target: iqn.2010-10.org.openstack:3b30abbe-d7f1-4e56-9991-a082bc596387 {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:930}} Apr 17 15:23:38 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-8991a11b-9d09-425a-8267-3a06c24ef669 tempest-GroupSnapshotsTest-1926061586 None] delete target iqn.2010-10.org.openstack:3b30abbe-d7f1-4e56-9991-a082bc596387 {{(pid=84696) delete_target /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:442}} Apr 17 15:23:38 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-8991a11b-9d09-425a-8267-3a06c24ef669 tempest-GroupSnapshotsTest-1926061586 None] Sending pool request DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:3b30abbe-d7f1-4e56-9991-a082bc596387 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:23:38 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-8991a11b-9d09-425a-8267-3a06c24ef669 tempest-GroupSnapshotsTest-1926061586 None] Sending DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:3b30abbe-d7f1-4e56-9991-a082bc596387 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:23:38 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-8991a11b-9d09-425a-8267-3a06c24ef669 tempest-GroupSnapshotsTest-1926061586 None] Geting {'code': 404, 'error': {'class': 'opene.exceptions.ItemNotFoundError', 'message': 'Target iqn.2010-10.org.openstack:3b30abbe-d7f1-4e56-9991-a082bc596387 not exists.', 'url': 'http://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:3b30abbe-d7f1-4e56-9991-a082bc596387'}, 'data': None} from DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:3b30abbe-d7f1-4e56-9991-a082bc596387 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:23:38 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-8991a11b-9d09-425a-8267-3a06c24ef669 tempest-GroupSnapshotsTest-1926061586 None] failed to remove resource iqn.2010-10.org.openstack:3b30abbe-d7f1-4e56-9991-a082bc596387 because of JDSS resource iqn.2010-10.org.openstack:3b30abbe-d7f1-4e56-9991-a082bc596387 DNE. {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:935}} Apr 17 15:23:38 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-8991a11b-9d09-425a-8267-3a06c24ef669 tempest-GroupSnapshotsTest-1926061586 None] deleating volume v_3b30abbe-d7f1-4e56-9991-a082bc596387 {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 15:23:38 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-8991a11b-9d09-425a-8267-3a06c24ef669 tempest-GroupSnapshotsTest-1926061586 None] get snapshots for volume v_3b30abbe-d7f1-4e56-9991-a082bc596387 {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 15:23:38 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-8991a11b-9d09-425a-8267-3a06c24ef669 tempest-GroupSnapshotsTest-1926061586 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_3b30abbe-d7f1-4e56-9991-a082bc596387/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:23:38 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-8991a11b-9d09-425a-8267-3a06c24ef669 tempest-GroupSnapshotsTest-1926061586 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_3b30abbe-d7f1-4e56-9991-a082bc596387/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:23:39 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-8991a11b-9d09-425a-8267-3a06c24ef669 tempest-GroupSnapshotsTest-1926061586 None] Geting {'code': 200, 'error': None, 'data': {'results': 0, 'entries': []}} from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_3b30abbe-d7f1-4e56-9991-a082bc596387/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:23:39 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-8991a11b-9d09-425a-8267-3a06c24ef669 tempest-GroupSnapshotsTest-1926061586 None] get volume v_3b30abbe-d7f1-4e56-9991-a082bc596387 info {{(pid=84696) get_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:239}} Apr 17 15:23:39 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-8991a11b-9d09-425a-8267-3a06c24ef669 tempest-GroupSnapshotsTest-1926061586 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_3b30abbe-d7f1-4e56-9991-a082bc596387 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:23:39 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-8991a11b-9d09-425a-8267-3a06c24ef669 tempest-GroupSnapshotsTest-1926061586 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_3b30abbe-d7f1-4e56-9991-a082bc596387 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:23:39 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-8991a11b-9d09-425a-8267-3a06c24ef669 tempest-GroupSnapshotsTest-1926061586 None] Geting {'code': 200, 'error': None, 'data': {'origin': 'Pool-0/s_f815ac77-8ec1-4c5c-a0a6-a00cb2779806@v_3b30abbe-d7f1-4e56-9991-a082bc596387', 'relatime': None, 'acltype': None, 'vscan': None, 'full_name': 'Pool-0/v_3b30abbe-d7f1-4e56-9991-a082bc596387', 'userrefs': None, 'primarycache': 'all', 'logbias': 'latency', 'creation': '1681746410', 'sync': 'always', 'is_clone': True, 'dedup': 'off', 'sharenfs': None, 'receive_resume_token': None, 'volsize': '1073741824', 'referenced': '57344', 'sharesmb': None, 'createtxg': '187820', 'reservation': '0', 'scontext': None, 'mountpoint': None, 'casesensitivity': None, 'guid': '1574029390680739449', 'usedbyrefreservation': '0', 'dnodesize': None, 'written': '0', 'logicalused': '0', 'compressratio': '1.00', 'rootcontext': 'none', 'default_scsi_id': '62b97efd26da3fb0', 'type': 'volume', 'compression': 'lz4', 'snapdir': None, 'overlay': None, 'encryption': 'off', 'xattr': None, 'volmode': 'default', 'copies': '1', 'snapshot_limit': '18446744073709551615', 'aclinherit': None, 'defcontext': 'none', 'readonly': 'off', 'version': None, 'recordsize': None, 'filesystem_limit': None, 'mounted': None, 'mlslabel': 'none', 'secondarycache': 'all', 'refreservation': '0', 'available': '113926549504', 'san:volume_id': '62b97efd26da3fb0673c088f37a05bb9177137b5', 'encryptionroot': None, 'exec': None, 'refquota': None, 'refcompressratio': '1.00', 'quota': None, 'utf8only': None, 'keylocation': 'none', 'snapdev': 'hidden', 'snapshot_count': '18446744073709551615', 'fscontext': 'none', 'clones': None, 'canmount': None, 'keystatus': None, 'atime': None, 'usedbysnapshots': '0', 'normalization': None, 'usedbychildren': '0', 'volblocksize': '131072', 'usedbydataset': '0', 'objsetid': '122894', 'name': 'v_3b30abbe-d7f1-4e56-9991-a082bc596387', 'defer_destroy': None, 'pbkdf2iters': '0', 'checksum': 'on', 'redundant_metadata': 'all', 'filesystem_count': None, 'devices': None, 'keyformat': 'none', 'setuid': None, 'used': '0', 'logicalreferenced': '28672', 'context': 'none', 'zoned': None, 'nbmand': None}} from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_3b30abbe-d7f1-4e56-9991-a082bc596387 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:23:39 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-8991a11b-9d09-425a-8267-3a06c24ef669 tempest-GroupSnapshotsTest-1926061586 None] delete snapshot v_3b30abbe-d7f1-4e56-9991-a082bc596387 of volume s_f815ac77-8ec1-4c5c-a0a6-a00cb2779806 {{(pid=84696) delete_snapshot /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:751}} Apr 17 15:23:39 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-8991a11b-9d09-425a-8267-3a06c24ef669 tempest-GroupSnapshotsTest-1926061586 None] Sending pool request DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/s_f815ac77-8ec1-4c5c-a0a6-a00cb2779806/snapshots/v_3b30abbe-d7f1-4e56-9991-a082bc596387 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:23:39 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-8991a11b-9d09-425a-8267-3a06c24ef669 tempest-GroupSnapshotsTest-1926061586 None] Sending DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/s_f815ac77-8ec1-4c5c-a0a6-a00cb2779806/snapshots/v_3b30abbe-d7f1-4e56-9991-a082bc596387 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:23:39 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-21eb6b89-a742-42f1-9d2f-4bcf12a902bc tempest-VolumesTestJSON-1329991747 None] Geting {'code': 204} from DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_e03a489c-cc11-4ae5-bf0d-f270a6a5ee02 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:23:39 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-21eb6b89-a742-42f1-9d2f-4bcf12a902bc tempest-VolumesTestJSON-1329991747 None] volume v_e03a489c-cc11-4ae5-bf0d-f270a6a5ee02 deleted {{(pid=84696) delete_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:346}} Apr 17 15:23:39 user cinder-volume[84696]: DEBUG cinder.quota [None req-21eb6b89-a742-42f1-9d2f-4bcf12a902bc tempest-VolumesTestJSON-1329991747 None] Created reservations ['d6ffa2ad-5124-4a61-875e-cc684b32f0b8', '76d4b094-199d-47c3-bf37-3fd88aab9764', 'df4316b2-8eb9-42e6-85e5-cc48a968a115', '36382b1e-ebc8-4108-8da9-0312e25cfe8e'] {{(pid=84696) reserve /opt/stack/cinder/cinder/quota.py:755}} Apr 17 15:23:39 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=63,cinder:UPDATE=28,cinder:DELETE=3,cinder:INSERT=6 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:23:39 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-21eb6b89-a742-42f1-9d2f-4bcf12a902bc tempest-VolumesTestJSON-1329991747 None] Updating volume stats {{(pid=84696) _update_volume_stats /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:724}} Apr 17 15:23:39 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-21eb6b89-a742-42f1-9d2f-4bcf12a902bc tempest-VolumesTestJSON-1329991747 None] Get pool Pool-0 fsprops {{(pid=84696) get_pool_stats /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:894}} Apr 17 15:23:39 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-21eb6b89-a742-42f1-9d2f-4bcf12a902bc tempest-VolumesTestJSON-1329991747 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:23:39 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-21eb6b89-a742-42f1-9d2f-4bcf12a902bc tempest-VolumesTestJSON-1329991747 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:23:39 user cinder-volume[84696]: DEBUG cinder.coordination [None req-a3d4b2c6-26c9-497e-8932-f55143203bdd tempest-VolumesTestJSON-1329991747 None] Acquiring lock "cinder-f8f9aea9-6773-4928-9e8e-153c3796301b-delete_volume" by "delete_volume" {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:192}} Apr 17 15:23:39 user cinder-volume[84696]: DEBUG cinder.coordination [None req-a3d4b2c6-26c9-497e-8932-f55143203bdd tempest-VolumesTestJSON-1329991747 None] Lock "cinder-f8f9aea9-6773-4928-9e8e-153c3796301b-delete_volume" acquired by "delete_volume" :: waited 0.011s {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:196}} Apr 17 15:23:39 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-a3d4b2c6-26c9-497e-8932-f55143203bdd tempest-VolumesTestJSON-1329991747 None] remove_export for volume: f8f9aea9-6773-4928-9e8e-153c3796301b. {{(pid=84696) remove_export /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:718}} Apr 17 15:23:39 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-a3d4b2c6-26c9-497e-8932-f55143203bdd tempest-VolumesTestJSON-1329991747 None] remove export {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:912}} Apr 17 15:23:39 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-a3d4b2c6-26c9-497e-8932-f55143203bdd tempest-VolumesTestJSON-1329991747 None] detach volume:Volume(_name_id=None,admin_metadata={},attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:23:29Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesTestJSON-volume-238641164',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=f8f9aea9-6773-4928-9e8e-153c3796301b,launched_at=2023-04-17T15:23:31Z,metadata={Type='work'},migration_status=None,multiattach=False,previous_status=None,project_id='01ad54f0fc8a4200a62264f0f9c33107',provider_auth='CHAP LNwBZIKu TqGi3pBKl5fEUt',provider_geometry=None,provider_id=None,provider_location='172.16.0.220:3260,1 iqn.2010-10.org.openstack:f8f9aea9-6773-4928-9e8e-153c3796301b 0',replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:23:29Z,service_uuid='e628352e-c4cf-40eb-8389-cce40663026a',shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='deleting',terminated_at=2023-04-17T15:23:40Z,updated_at=2023-04-17T15:23:40Z,use_quota=True,user_id='e8b77c4bc3b84389b237889bcd55f2af',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c) from target:iqn.2010-10.org.openstack:f8f9aea9-6773-4928-9e8e-153c3796301b. {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:913}} Apr 17 15:23:39 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-a3d4b2c6-26c9-497e-8932-f55143203bdd tempest-VolumesTestJSON-1329991747 None] detach volume v_f8f9aea9-6773-4928-9e8e-153c3796301b from target iqn.2010-10.org.openstack:f8f9aea9-6773-4928-9e8e-153c3796301b {{(pid=84696) detach_target_vol /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:606}} Apr 17 15:23:39 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-a3d4b2c6-26c9-497e-8932-f55143203bdd tempest-VolumesTestJSON-1329991747 None] Sending pool request DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:f8f9aea9-6773-4928-9e8e-153c3796301b/luns/v_f8f9aea9-6773-4928-9e8e-153c3796301b {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:23:39 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-a3d4b2c6-26c9-497e-8932-f55143203bdd tempest-VolumesTestJSON-1329991747 None] Sending DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:f8f9aea9-6773-4928-9e8e-153c3796301b/luns/v_f8f9aea9-6773-4928-9e8e-153c3796301b {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:23:39 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-a3d4b2c6-26c9-497e-8932-f55143203bdd tempest-VolumesTestJSON-1329991747 None] Geting {'code': 404, 'error': {'class': 'opene.exceptions.ItemNotFoundError', 'message': 'Target iqn.2010-10.org.openstack:f8f9aea9-6773-4928-9e8e-153c3796301b not exists.', 'url': 'http://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:f8f9aea9-6773-4928-9e8e-153c3796301b/luns/v_f8f9aea9-6773-4928-9e8e-153c3796301b'}, 'data': None} from DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:f8f9aea9-6773-4928-9e8e-153c3796301b/luns/v_f8f9aea9-6773-4928-9e8e-153c3796301b {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:23:39 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-a3d4b2c6-26c9-497e-8932-f55143203bdd tempest-VolumesTestJSON-1329991747 None] failed to remove resource iqn.2010-10.org.openstack:f8f9aea9-6773-4928-9e8e-153c3796301b because of JDSS resource v_f8f9aea9-6773-4928-9e8e-153c3796301b DNE. {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:920}} Apr 17 15:23:39 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-a3d4b2c6-26c9-497e-8932-f55143203bdd tempest-VolumesTestJSON-1329991747 None] delete target: iqn.2010-10.org.openstack:f8f9aea9-6773-4928-9e8e-153c3796301b {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:930}} Apr 17 15:23:39 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-a3d4b2c6-26c9-497e-8932-f55143203bdd tempest-VolumesTestJSON-1329991747 None] delete target iqn.2010-10.org.openstack:f8f9aea9-6773-4928-9e8e-153c3796301b {{(pid=84696) delete_target /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:442}} Apr 17 15:23:39 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-a3d4b2c6-26c9-497e-8932-f55143203bdd tempest-VolumesTestJSON-1329991747 None] Sending pool request DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:f8f9aea9-6773-4928-9e8e-153c3796301b {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:23:39 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-a3d4b2c6-26c9-497e-8932-f55143203bdd tempest-VolumesTestJSON-1329991747 None] Sending DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:f8f9aea9-6773-4928-9e8e-153c3796301b {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:23:39 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-a3d4b2c6-26c9-497e-8932-f55143203bdd tempest-VolumesTestJSON-1329991747 None] Geting {'code': 404, 'error': {'class': 'opene.exceptions.ItemNotFoundError', 'message': 'Target iqn.2010-10.org.openstack:f8f9aea9-6773-4928-9e8e-153c3796301b not exists.', 'url': 'http://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:f8f9aea9-6773-4928-9e8e-153c3796301b'}, 'data': None} from DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:f8f9aea9-6773-4928-9e8e-153c3796301b {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:23:39 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-a3d4b2c6-26c9-497e-8932-f55143203bdd tempest-VolumesTestJSON-1329991747 None] failed to remove resource iqn.2010-10.org.openstack:f8f9aea9-6773-4928-9e8e-153c3796301b because of JDSS resource iqn.2010-10.org.openstack:f8f9aea9-6773-4928-9e8e-153c3796301b DNE. {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:935}} Apr 17 15:23:39 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-a3d4b2c6-26c9-497e-8932-f55143203bdd tempest-VolumesTestJSON-1329991747 None] deleating volume v_f8f9aea9-6773-4928-9e8e-153c3796301b {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 15:23:39 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-a3d4b2c6-26c9-497e-8932-f55143203bdd tempest-VolumesTestJSON-1329991747 None] get snapshots for volume v_f8f9aea9-6773-4928-9e8e-153c3796301b {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 15:23:39 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-a3d4b2c6-26c9-497e-8932-f55143203bdd tempest-VolumesTestJSON-1329991747 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_f8f9aea9-6773-4928-9e8e-153c3796301b/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:23:39 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-a3d4b2c6-26c9-497e-8932-f55143203bdd tempest-VolumesTestJSON-1329991747 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_f8f9aea9-6773-4928-9e8e-153c3796301b/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:23:40 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-21eb6b89-a742-42f1-9d2f-4bcf12a902bc tempest-VolumesTestJSON-1329991747 None] Geting {'code': 200, 'error': None, 'data': {'available': '113926549504', 'status': 30, 'import_status': {'import_steps': {'san_setup': True, 'vip_setup': True, 'zfs_import': True, 'mount_system_volume': True, 'nas_setup': True}, 'import_successful': True}, 'scan': None, 'encryption': {'enabled': False}, 'deduplication_rate': '1.00', 'sysvol_upgrade': 'not_needed', 'vdevs': [{'name': 'wwn-0x6000c296b793bf526a107a9d823e947a', 'iostats': {'read': '0', 'write': '0', 'chksum': '0'}, 'disks': [{'origin': 'local', 'slot': '', 'led': 'off', 'name': 'sdn', 'iostats': {'read': '0', 'write': '0', 'chksum': '0'}, 'alias': '', 'health': 'ONLINE', 'sn': '6000c296b793bf526a107a9d823e947a', 'trim_data': {'status': 'trim_unsupported', 'progress': None, 'trim_time': None}, 'path': 'pci-0000:03:00.0-scsi-0:0:14:0', 'model': 'Virtual disk', 'id': 'wwn-0x6000c296b793bf526a107a9d823e947a', 'size': 128849018880}], 'health': 'ONLINE', 'vdev_replacings': [], 'vdev_spares': [], 'type': ''}], 'id': '14181317334440137330', 'health': 'ONLINE', 'iostats': {'read': '0', 'write': '0', 'chksum': '0'}, 'operation': 'none', 'size': '127775277056', 'autotrim': False, 'name': 'Pool-0'}} from GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:23:40 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-21eb6b89-a742-42f1-9d2f-4bcf12a902bc tempest-VolumesTestJSON-1329991747 None] Total capacity: 119, Free 106. {{(pid=84696) _update_volume_stats /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:757}} Apr 17 15:23:40 user cinder-volume[84696]: DEBUG cinder.manager [None req-21eb6b89-a742-42f1-9d2f-4bcf12a902bc tempest-VolumesTestJSON-1329991747 None] Notifying Schedulers of capabilities ... {{(pid=84696) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 17 15:23:40 user cinder-volume[84696]: INFO cinder.volume.manager [None req-21eb6b89-a742-42f1-9d2f-4bcf12a902bc tempest-VolumesTestJSON-1329991747 None] Deleted volume successfully. Apr 17 15:23:40 user cinder-volume[84696]: DEBUG cinder.coordination [None req-21eb6b89-a742-42f1-9d2f-4bcf12a902bc tempest-VolumesTestJSON-1329991747 None] Lock "cinder-e03a489c-cc11-4ae5-bf0d-f270a6a5ee02-delete_volume" released by "delete_volume" :: held 5.053s {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:207}} Apr 17 15:23:40 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-a3d4b2c6-26c9-497e-8932-f55143203bdd tempest-VolumesTestJSON-1329991747 None] Geting {'code': 200, 'error': None, 'data': {'results': 0, 'entries': []}} from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_f8f9aea9-6773-4928-9e8e-153c3796301b/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:23:40 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-a3d4b2c6-26c9-497e-8932-f55143203bdd tempest-VolumesTestJSON-1329991747 None] get volume v_f8f9aea9-6773-4928-9e8e-153c3796301b info {{(pid=84696) get_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:239}} Apr 17 15:23:40 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-a3d4b2c6-26c9-497e-8932-f55143203bdd tempest-VolumesTestJSON-1329991747 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_f8f9aea9-6773-4928-9e8e-153c3796301b {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:23:40 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-a3d4b2c6-26c9-497e-8932-f55143203bdd tempest-VolumesTestJSON-1329991747 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_f8f9aea9-6773-4928-9e8e-153c3796301b {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:23:40 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-a3d4b2c6-26c9-497e-8932-f55143203bdd tempest-VolumesTestJSON-1329991747 None] Geting {'code': 200, 'error': None, 'data': {'origin': None, 'relatime': None, 'acltype': None, 'vscan': None, 'full_name': 'Pool-0/v_f8f9aea9-6773-4928-9e8e-153c3796301b', 'userrefs': None, 'primarycache': 'all', 'logbias': 'latency', 'creation': '1681746403', 'sync': 'always', 'is_clone': False, 'dedup': 'off', 'sharenfs': None, 'receive_resume_token': None, 'volsize': '1073741824', 'referenced': '57344', 'sharesmb': None, 'createtxg': '187803', 'reservation': '0', 'scontext': None, 'mountpoint': None, 'casesensitivity': None, 'guid': '15640231974819944577', 'usedbyrefreservation': '0', 'dnodesize': None, 'written': '57344', 'logicalused': '28672', 'compressratio': '1.00', 'rootcontext': 'none', 'default_scsi_id': 'fab5974579d43261', 'type': 'volume', 'compression': 'lz4', 'snapdir': None, 'overlay': None, 'encryption': 'off', 'xattr': None, 'volmode': 'default', 'copies': '1', 'snapshot_limit': '18446744073709551615', 'aclinherit': None, 'defcontext': 'none', 'readonly': 'off', 'version': None, 'recordsize': None, 'filesystem_limit': None, 'mounted': None, 'mlslabel': 'none', 'secondarycache': 'all', 'refreservation': '0', 'available': '113926549504', 'san:volume_id': 'fab5974579d43261afaf9f8415e8c4820041b71f', 'encryptionroot': None, 'exec': None, 'refquota': None, 'refcompressratio': '1.00', 'quota': None, 'utf8only': None, 'keylocation': 'none', 'snapdev': 'hidden', 'snapshot_count': '18446744073709551615', 'fscontext': 'none', 'clones': None, 'canmount': None, 'keystatus': None, 'atime': None, 'usedbysnapshots': '0', 'normalization': None, 'usedbychildren': '0', 'volblocksize': '131072', 'usedbydataset': '57344', 'objsetid': '121819', 'name': 'v_f8f9aea9-6773-4928-9e8e-153c3796301b', 'defer_destroy': None, 'pbkdf2iters': '0', 'checksum': 'on', 'redundant_metadata': 'all', 'filesystem_count': None, 'devices': None, 'keyformat': 'none', 'setuid': None, 'used': '57344', 'logicalreferenced': '28672', 'context': 'none', 'zoned': None, 'nbmand': None}} from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_f8f9aea9-6773-4928-9e8e-153c3796301b {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:23:40 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-a3d4b2c6-26c9-497e-8932-f55143203bdd tempest-VolumesTestJSON-1329991747 None] delete volume:v_f8f9aea9-6773-4928-9e8e-153c3796301b recursively children:{'force_umount': True} {{(pid=84696) delete_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:335}} Apr 17 15:23:40 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-a3d4b2c6-26c9-497e-8932-f55143203bdd tempest-VolumesTestJSON-1329991747 None] Sending pool request DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_f8f9aea9-6773-4928-9e8e-153c3796301b {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:23:40 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-a3d4b2c6-26c9-497e-8932-f55143203bdd tempest-VolumesTestJSON-1329991747 None] Sending DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_f8f9aea9-6773-4928-9e8e-153c3796301b {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:23:41 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-8991a11b-9d09-425a-8267-3a06c24ef669 tempest-GroupSnapshotsTest-1926061586 None] Geting {'code': 204} from DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/s_f815ac77-8ec1-4c5c-a0a6-a00cb2779806/snapshots/v_3b30abbe-d7f1-4e56-9991-a082bc596387 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:23:41 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-8991a11b-9d09-425a-8267-3a06c24ef669 tempest-GroupSnapshotsTest-1926061586 None] snapshot v_3b30abbe-d7f1-4e56-9991-a082bc596387 deleted {{(pid=84696) delete_snapshot /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:772}} Apr 17 15:23:41 user cinder-volume[84696]: DEBUG cinder.quota [None req-8991a11b-9d09-425a-8267-3a06c24ef669 tempest-GroupSnapshotsTest-1926061586 None] Created reservations ['88af5654-a445-4c88-90c1-29d9910dbab2'] {{(pid=84696) reserve /opt/stack/cinder/cinder/quota.py:755}} Apr 17 15:23:41 user cinder-volume[84696]: WARNING cinder.quota [None req-8991a11b-9d09-425a-8267-3a06c24ef669 tempest-GroupSnapshotsTest-1926061586 None] Deprecated: Default quota for resource: volumes_tempest-GroupSnapshotsTest-volume-type-344582814 is set by the default quota flag: quota_volumes_tempest-GroupSnapshotsTest-volume-type-344582814, it is now deprecated. Please use the default quota class for default quota. Apr 17 15:23:41 user cinder-volume[84696]: WARNING cinder.quota [None req-8991a11b-9d09-425a-8267-3a06c24ef669 tempest-GroupSnapshotsTest-1926061586 None] Deprecated: Default quota for resource: gigabytes_tempest-GroupSnapshotsTest-volume-type-344582814 is set by the default quota flag: quota_gigabytes_tempest-GroupSnapshotsTest-volume-type-344582814, it is now deprecated. Please use the default quota class for default quota. Apr 17 15:23:41 user cinder-volume[84696]: DEBUG cinder.quota [None req-8991a11b-9d09-425a-8267-3a06c24ef669 tempest-GroupSnapshotsTest-1926061586 None] Created reservations ['a1a1cfcc-30f4-4f9f-8e4e-cff23da4c3a0', '63bcfcbd-3320-469e-af2f-9cdbf6df7bea', '4d51737c-b531-41c6-b3b3-dc03d7a84b7f', '7138aa26-4c99-4758-9135-9635e55611d0'] {{(pid=84696) reserve /opt/stack/cinder/cinder/quota.py:755}} Apr 17 15:23:41 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-8991a11b-9d09-425a-8267-3a06c24ef669 tempest-GroupSnapshotsTest-1926061586 None] Updating volume stats {{(pid=84696) _update_volume_stats /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:724}} Apr 17 15:23:41 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-8991a11b-9d09-425a-8267-3a06c24ef669 tempest-GroupSnapshotsTest-1926061586 None] Get pool Pool-0 fsprops {{(pid=84696) get_pool_stats /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:894}} Apr 17 15:23:41 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-8991a11b-9d09-425a-8267-3a06c24ef669 tempest-GroupSnapshotsTest-1926061586 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:23:41 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-8991a11b-9d09-425a-8267-3a06c24ef669 tempest-GroupSnapshotsTest-1926061586 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:23:42 user cinder-volume[84696]: INFO cinder.volume.manager [None req-f98767e8-b476-4db5-9f1a-3f92d731e3c6 tempest-GroupSnapshotsTest-1926061586 None] group_snapshot e790b696-22cc-4fac-8b67-57499e6de8b3: deleting Apr 17 15:23:42 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f98767e8-b476-4db5-9f1a-3f92d731e3c6 tempest-GroupSnapshotsTest-1926061586 None] group_snapshot e790b696-22cc-4fac-8b67-57499e6de8b3: deleting {{(pid=84696) delete_group_snapshot /opt/stack/cinder/cinder/volume/manager.py:4304}} Apr 17 15:23:42 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-f98767e8-b476-4db5-9f1a-3f92d731e3c6 tempest-GroupSnapshotsTest-1926061586 None] deleating snapshot s_f815ac77-8ec1-4c5c-a0a6-a00cb2779806. {{(pid=84696) delete_snapshot /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:649}} Apr 17 15:23:42 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-f98767e8-b476-4db5-9f1a-3f92d731e3c6 tempest-GroupSnapshotsTest-1926061586 None] get snapshots for volume s_f815ac77-8ec1-4c5c-a0a6-a00cb2779806 {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 15:23:42 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-f98767e8-b476-4db5-9f1a-3f92d731e3c6 tempest-GroupSnapshotsTest-1926061586 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/s_f815ac77-8ec1-4c5c-a0a6-a00cb2779806/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:23:42 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-f98767e8-b476-4db5-9f1a-3f92d731e3c6 tempest-GroupSnapshotsTest-1926061586 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/s_f815ac77-8ec1-4c5c-a0a6-a00cb2779806/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:23:42 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-8991a11b-9d09-425a-8267-3a06c24ef669 tempest-GroupSnapshotsTest-1926061586 None] Geting {'code': 200, 'error': None, 'data': {'available': '113926574080', 'status': 30, 'import_status': {'import_steps': {'san_setup': True, 'vip_setup': True, 'zfs_import': True, 'mount_system_volume': True, 'nas_setup': True}, 'import_successful': True}, 'scan': None, 'encryption': {'enabled': False}, 'deduplication_rate': '1.00', 'sysvol_upgrade': 'not_needed', 'vdevs': [{'name': 'wwn-0x6000c296b793bf526a107a9d823e947a', 'iostats': {'read': '0', 'write': '0', 'chksum': '0'}, 'disks': [{'origin': 'local', 'slot': '', 'led': 'off', 'name': 'sdn', 'iostats': {'read': '0', 'write': '0', 'chksum': '0'}, 'alias': '', 'health': 'ONLINE', 'sn': '6000c296b793bf526a107a9d823e947a', 'trim_data': {'status': 'trim_unsupported', 'progress': None, 'trim_time': None}, 'path': 'pci-0000:03:00.0-scsi-0:0:14:0', 'model': 'Virtual disk', 'id': 'wwn-0x6000c296b793bf526a107a9d823e947a', 'size': 128849018880}], 'health': 'ONLINE', 'vdev_replacings': [], 'vdev_spares': [], 'type': ''}], 'id': '14181317334440137330', 'health': 'ONLINE', 'iostats': {'read': '0', 'write': '0', 'chksum': '0'}, 'operation': 'none', 'size': '127775277056', 'autotrim': False, 'name': 'Pool-0'}} from GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:23:42 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-8991a11b-9d09-425a-8267-3a06c24ef669 tempest-GroupSnapshotsTest-1926061586 None] Total capacity: 119, Free 106. {{(pid=84696) _update_volume_stats /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:757}} Apr 17 15:23:42 user cinder-volume[84696]: DEBUG cinder.manager [None req-8991a11b-9d09-425a-8267-3a06c24ef669 tempest-GroupSnapshotsTest-1926061586 None] Notifying Schedulers of capabilities ... {{(pid=84696) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 17 15:23:42 user cinder-volume[84696]: INFO cinder.volume.manager [None req-8991a11b-9d09-425a-8267-3a06c24ef669 tempest-GroupSnapshotsTest-1926061586 None] Delete group completed successfully. Apr 17 15:23:42 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-f98767e8-b476-4db5-9f1a-3f92d731e3c6 tempest-GroupSnapshotsTest-1926061586 None] Geting {'code': 200, 'error': None, 'data': {'results': 0, 'entries': []}} from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/s_f815ac77-8ec1-4c5c-a0a6-a00cb2779806/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:23:42 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-f98767e8-b476-4db5-9f1a-3f92d731e3c6 tempest-GroupSnapshotsTest-1926061586 None] get volume s_f815ac77-8ec1-4c5c-a0a6-a00cb2779806 info {{(pid=84696) get_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:239}} Apr 17 15:23:42 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-f98767e8-b476-4db5-9f1a-3f92d731e3c6 tempest-GroupSnapshotsTest-1926061586 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/s_f815ac77-8ec1-4c5c-a0a6-a00cb2779806 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:23:42 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-f98767e8-b476-4db5-9f1a-3f92d731e3c6 tempest-GroupSnapshotsTest-1926061586 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/s_f815ac77-8ec1-4c5c-a0a6-a00cb2779806 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:23:42 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-f98767e8-b476-4db5-9f1a-3f92d731e3c6 tempest-GroupSnapshotsTest-1926061586 None] Geting {'code': 200, 'error': None, 'data': {'origin': 'Pool-0/v_74f7ff47-86dd-433a-8a61-f46278efe3fd@s_f815ac77-8ec1-4c5c-a0a6-a00cb2779806', 'relatime': None, 'acltype': None, 'vscan': None, 'full_name': 'Pool-0/s_f815ac77-8ec1-4c5c-a0a6-a00cb2779806', 'userrefs': None, 'primarycache': 'all', 'logbias': 'latency', 'creation': '1681746406', 'sync': 'always', 'is_clone': True, 'dedup': 'off', 'sharenfs': None, 'receive_resume_token': None, 'volsize': '1073741824', 'referenced': '57344', 'sharesmb': None, 'createtxg': '187810', 'reservation': '0', 'scontext': None, 'mountpoint': None, 'casesensitivity': None, 'guid': '8576326793922689971', 'usedbyrefreservation': '0', 'dnodesize': None, 'written': '0', 'logicalused': '0', 'compressratio': '1.00', 'rootcontext': 'none', 'default_scsi_id': '8373203b5d6c83f8', 'type': 'volume', 'compression': 'lz4', 'snapdir': None, 'overlay': None, 'encryption': 'off', 'xattr': None, 'volmode': 'default', 'copies': '1', 'snapshot_limit': '18446744073709551615', 'aclinherit': None, 'defcontext': 'none', 'readonly': 'on', 'version': None, 'recordsize': None, 'filesystem_limit': None, 'mounted': None, 'mlslabel': 'none', 'secondarycache': 'all', 'refreservation': '0', 'available': '113926656000', 'used': '0', 'encryptionroot': None, 'exec': None, 'refquota': None, 'refcompressratio': '1.00', 'quota': None, 'utf8only': None, 'keylocation': 'none', 'snapdev': 'hidden', 'snapshot_count': '18446744073709551615', 'fscontext': 'none', 'clones': None, 'canmount': None, 'keystatus': None, 'atime': None, 'usedbysnapshots': '0', 'normalization': None, 'usedbychildren': '0', 'volblocksize': '131072', 'usedbydataset': '0', 'objsetid': '122677', 'name': 's_f815ac77-8ec1-4c5c-a0a6-a00cb2779806', 'defer_destroy': None, 'pbkdf2iters': '0', 'checksum': 'on', 'redundant_metadata': 'all', 'filesystem_count': None, 'devices': None, 'keyformat': 'none', 'setuid': None, 'logicalreferenced': '28672', 'context': 'none', 'zoned': None, 'nbmand': None}} from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/s_f815ac77-8ec1-4c5c-a0a6-a00cb2779806 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:23:42 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-f98767e8-b476-4db5-9f1a-3f92d731e3c6 tempest-GroupSnapshotsTest-1926061586 None] delete snapshot s_f815ac77-8ec1-4c5c-a0a6-a00cb2779806 of volume v_74f7ff47-86dd-433a-8a61-f46278efe3fd {{(pid=84696) delete_snapshot /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:751}} Apr 17 15:23:42 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-f98767e8-b476-4db5-9f1a-3f92d731e3c6 tempest-GroupSnapshotsTest-1926061586 None] Sending pool request DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_74f7ff47-86dd-433a-8a61-f46278efe3fd/snapshots/s_f815ac77-8ec1-4c5c-a0a6-a00cb2779806 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:23:42 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-f98767e8-b476-4db5-9f1a-3f92d731e3c6 tempest-GroupSnapshotsTest-1926061586 None] Sending DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_74f7ff47-86dd-433a-8a61-f46278efe3fd/snapshots/s_f815ac77-8ec1-4c5c-a0a6-a00cb2779806 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:23:43 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-a3d4b2c6-26c9-497e-8932-f55143203bdd tempest-VolumesTestJSON-1329991747 None] Geting {'code': 204} from DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_f8f9aea9-6773-4928-9e8e-153c3796301b {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:23:43 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-a3d4b2c6-26c9-497e-8932-f55143203bdd tempest-VolumesTestJSON-1329991747 None] volume v_f8f9aea9-6773-4928-9e8e-153c3796301b deleted {{(pid=84696) delete_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:346}} Apr 17 15:23:43 user cinder-volume[84696]: DEBUG cinder.quota [None req-a3d4b2c6-26c9-497e-8932-f55143203bdd tempest-VolumesTestJSON-1329991747 None] Created reservations ['1dd5bd89-d6c7-400a-aee2-b71b16232b1a', 'cbb5dadc-a1dd-448d-abc4-cd4c208084f3', '852db145-7700-4b14-b30a-494bc6f25c7e', 'f5b7fa8d-0d61-4b5d-af7f-2220a76515c2'] {{(pid=84696) reserve /opt/stack/cinder/cinder/quota.py:755}} Apr 17 15:23:43 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-a3d4b2c6-26c9-497e-8932-f55143203bdd tempest-VolumesTestJSON-1329991747 None] Updating volume stats {{(pid=84696) _update_volume_stats /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:724}} Apr 17 15:23:43 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-a3d4b2c6-26c9-497e-8932-f55143203bdd tempest-VolumesTestJSON-1329991747 None] Get pool Pool-0 fsprops {{(pid=84696) get_pool_stats /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:894}} Apr 17 15:23:43 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-a3d4b2c6-26c9-497e-8932-f55143203bdd tempest-VolumesTestJSON-1329991747 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:23:43 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-a3d4b2c6-26c9-497e-8932-f55143203bdd tempest-VolumesTestJSON-1329991747 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:23:44 user cinder-volume[84696]: DEBUG cinder.coordination [None req-16e1020c-fa96-497c-86b8-f8a35e1f0dfb tempest-VolumesTestJSON-1329991747 None] Acquiring lock "cinder-c5f04e05-b556-4cb3-84c8-590aa69695b0-delete_volume" by "delete_volume" {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:192}} Apr 17 15:23:44 user cinder-volume[84696]: DEBUG cinder.coordination [None req-16e1020c-fa96-497c-86b8-f8a35e1f0dfb tempest-VolumesTestJSON-1329991747 None] Lock "cinder-c5f04e05-b556-4cb3-84c8-590aa69695b0-delete_volume" acquired by "delete_volume" :: waited 0.013s {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:196}} Apr 17 15:23:44 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-16e1020c-fa96-497c-86b8-f8a35e1f0dfb tempest-VolumesTestJSON-1329991747 None] remove_export for volume: c5f04e05-b556-4cb3-84c8-590aa69695b0. {{(pid=84696) remove_export /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:718}} Apr 17 15:23:44 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-16e1020c-fa96-497c-86b8-f8a35e1f0dfb tempest-VolumesTestJSON-1329991747 None] remove export {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:912}} Apr 17 15:23:44 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-16e1020c-fa96-497c-86b8-f8a35e1f0dfb tempest-VolumesTestJSON-1329991747 None] detach volume:Volume(_name_id=None,admin_metadata={},attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:23:26Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesTestJSON-volume-420411441',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=c5f04e05-b556-4cb3-84c8-590aa69695b0,launched_at=2023-04-17T15:23:28Z,metadata={Type='work'},migration_status=None,multiattach=False,previous_status=None,project_id='01ad54f0fc8a4200a62264f0f9c33107',provider_auth='CHAP zbAZGCit iUXpb5MZDtskqx',provider_geometry=None,provider_id=None,provider_location='172.16.0.220:3260,1 iqn.2010-10.org.openstack:c5f04e05-b556-4cb3-84c8-590aa69695b0 0',replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:23:26Z,service_uuid='e628352e-c4cf-40eb-8389-cce40663026a',shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='deleting',terminated_at=2023-04-17T15:23:44Z,updated_at=2023-04-17T15:23:44Z,use_quota=True,user_id='e8b77c4bc3b84389b237889bcd55f2af',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c) from target:iqn.2010-10.org.openstack:c5f04e05-b556-4cb3-84c8-590aa69695b0. {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:913}} Apr 17 15:23:44 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-16e1020c-fa96-497c-86b8-f8a35e1f0dfb tempest-VolumesTestJSON-1329991747 None] detach volume v_c5f04e05-b556-4cb3-84c8-590aa69695b0 from target iqn.2010-10.org.openstack:c5f04e05-b556-4cb3-84c8-590aa69695b0 {{(pid=84696) detach_target_vol /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:606}} Apr 17 15:23:44 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-16e1020c-fa96-497c-86b8-f8a35e1f0dfb tempest-VolumesTestJSON-1329991747 None] Sending pool request DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:c5f04e05-b556-4cb3-84c8-590aa69695b0/luns/v_c5f04e05-b556-4cb3-84c8-590aa69695b0 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:23:44 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-16e1020c-fa96-497c-86b8-f8a35e1f0dfb tempest-VolumesTestJSON-1329991747 None] Sending DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:c5f04e05-b556-4cb3-84c8-590aa69695b0/luns/v_c5f04e05-b556-4cb3-84c8-590aa69695b0 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:23:44 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-16e1020c-fa96-497c-86b8-f8a35e1f0dfb tempest-VolumesTestJSON-1329991747 None] Geting {'code': 404, 'error': {'class': 'opene.exceptions.ItemNotFoundError', 'message': 'Target iqn.2010-10.org.openstack:c5f04e05-b556-4cb3-84c8-590aa69695b0 not exists.', 'url': 'http://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:c5f04e05-b556-4cb3-84c8-590aa69695b0/luns/v_c5f04e05-b556-4cb3-84c8-590aa69695b0'}, 'data': None} from DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:c5f04e05-b556-4cb3-84c8-590aa69695b0/luns/v_c5f04e05-b556-4cb3-84c8-590aa69695b0 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:23:44 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-16e1020c-fa96-497c-86b8-f8a35e1f0dfb tempest-VolumesTestJSON-1329991747 None] failed to remove resource iqn.2010-10.org.openstack:c5f04e05-b556-4cb3-84c8-590aa69695b0 because of JDSS resource v_c5f04e05-b556-4cb3-84c8-590aa69695b0 DNE. {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:920}} Apr 17 15:23:44 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-16e1020c-fa96-497c-86b8-f8a35e1f0dfb tempest-VolumesTestJSON-1329991747 None] delete target: iqn.2010-10.org.openstack:c5f04e05-b556-4cb3-84c8-590aa69695b0 {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:930}} Apr 17 15:23:44 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-16e1020c-fa96-497c-86b8-f8a35e1f0dfb tempest-VolumesTestJSON-1329991747 None] delete target iqn.2010-10.org.openstack:c5f04e05-b556-4cb3-84c8-590aa69695b0 {{(pid=84696) delete_target /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:442}} Apr 17 15:23:44 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-16e1020c-fa96-497c-86b8-f8a35e1f0dfb tempest-VolumesTestJSON-1329991747 None] Sending pool request DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:c5f04e05-b556-4cb3-84c8-590aa69695b0 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:23:44 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-16e1020c-fa96-497c-86b8-f8a35e1f0dfb tempest-VolumesTestJSON-1329991747 None] Sending DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:c5f04e05-b556-4cb3-84c8-590aa69695b0 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:23:44 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-16e1020c-fa96-497c-86b8-f8a35e1f0dfb tempest-VolumesTestJSON-1329991747 None] Geting {'code': 404, 'error': {'class': 'opene.exceptions.ItemNotFoundError', 'message': 'Target iqn.2010-10.org.openstack:c5f04e05-b556-4cb3-84c8-590aa69695b0 not exists.', 'url': 'http://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:c5f04e05-b556-4cb3-84c8-590aa69695b0'}, 'data': None} from DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:c5f04e05-b556-4cb3-84c8-590aa69695b0 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:23:44 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-16e1020c-fa96-497c-86b8-f8a35e1f0dfb tempest-VolumesTestJSON-1329991747 None] failed to remove resource iqn.2010-10.org.openstack:c5f04e05-b556-4cb3-84c8-590aa69695b0 because of JDSS resource iqn.2010-10.org.openstack:c5f04e05-b556-4cb3-84c8-590aa69695b0 DNE. {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:935}} Apr 17 15:23:44 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-16e1020c-fa96-497c-86b8-f8a35e1f0dfb tempest-VolumesTestJSON-1329991747 None] deleating volume v_c5f04e05-b556-4cb3-84c8-590aa69695b0 {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 15:23:44 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-16e1020c-fa96-497c-86b8-f8a35e1f0dfb tempest-VolumesTestJSON-1329991747 None] get snapshots for volume v_c5f04e05-b556-4cb3-84c8-590aa69695b0 {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 15:23:44 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-16e1020c-fa96-497c-86b8-f8a35e1f0dfb tempest-VolumesTestJSON-1329991747 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_c5f04e05-b556-4cb3-84c8-590aa69695b0/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:23:44 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-16e1020c-fa96-497c-86b8-f8a35e1f0dfb tempest-VolumesTestJSON-1329991747 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_c5f04e05-b556-4cb3-84c8-590aa69695b0/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:23:44 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-a3d4b2c6-26c9-497e-8932-f55143203bdd tempest-VolumesTestJSON-1329991747 None] Geting {'code': 200, 'error': None, 'data': {'available': '113926631424', 'status': 30, 'import_status': {'import_steps': {'san_setup': True, 'vip_setup': True, 'zfs_import': True, 'mount_system_volume': True, 'nas_setup': True}, 'import_successful': True}, 'scan': None, 'encryption': {'enabled': False}, 'deduplication_rate': '1.00', 'sysvol_upgrade': 'not_needed', 'vdevs': [{'name': 'wwn-0x6000c296b793bf526a107a9d823e947a', 'iostats': {'read': '0', 'write': '0', 'chksum': '0'}, 'disks': [{'origin': 'local', 'slot': '', 'led': 'off', 'name': 'sdn', 'iostats': {'read': '0', 'write': '0', 'chksum': '0'}, 'alias': '', 'health': 'ONLINE', 'sn': '6000c296b793bf526a107a9d823e947a', 'trim_data': {'status': 'trim_unsupported', 'progress': None, 'trim_time': None}, 'path': 'pci-0000:03:00.0-scsi-0:0:14:0', 'model': 'Virtual disk', 'id': 'wwn-0x6000c296b793bf526a107a9d823e947a', 'size': 128849018880}], 'health': 'ONLINE', 'vdev_replacings': [], 'vdev_spares': [], 'type': ''}], 'id': '14181317334440137330', 'health': 'ONLINE', 'iostats': {'read': '0', 'write': '0', 'chksum': '0'}, 'operation': 'none', 'size': '127775277056', 'autotrim': False, 'name': 'Pool-0'}} from GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:23:44 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-a3d4b2c6-26c9-497e-8932-f55143203bdd tempest-VolumesTestJSON-1329991747 None] Total capacity: 119, Free 106. {{(pid=84696) _update_volume_stats /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:757}} Apr 17 15:23:44 user cinder-volume[84696]: DEBUG cinder.manager [None req-a3d4b2c6-26c9-497e-8932-f55143203bdd tempest-VolumesTestJSON-1329991747 None] Notifying Schedulers of capabilities ... {{(pid=84696) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 17 15:23:44 user cinder-volume[84696]: INFO cinder.volume.manager [None req-a3d4b2c6-26c9-497e-8932-f55143203bdd tempest-VolumesTestJSON-1329991747 None] Deleted volume successfully. Apr 17 15:23:44 user cinder-volume[84696]: DEBUG cinder.coordination [None req-a3d4b2c6-26c9-497e-8932-f55143203bdd tempest-VolumesTestJSON-1329991747 None] Lock "cinder-f8f9aea9-6773-4928-9e8e-153c3796301b-delete_volume" released by "delete_volume" :: held 4.585s {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:207}} Apr 17 15:23:45 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-16e1020c-fa96-497c-86b8-f8a35e1f0dfb tempest-VolumesTestJSON-1329991747 None] Geting {'code': 200, 'error': None, 'data': {'results': 0, 'entries': []}} from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_c5f04e05-b556-4cb3-84c8-590aa69695b0/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:23:45 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-16e1020c-fa96-497c-86b8-f8a35e1f0dfb tempest-VolumesTestJSON-1329991747 None] get volume v_c5f04e05-b556-4cb3-84c8-590aa69695b0 info {{(pid=84696) get_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:239}} Apr 17 15:23:45 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-16e1020c-fa96-497c-86b8-f8a35e1f0dfb tempest-VolumesTestJSON-1329991747 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_c5f04e05-b556-4cb3-84c8-590aa69695b0 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:23:45 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-16e1020c-fa96-497c-86b8-f8a35e1f0dfb tempest-VolumesTestJSON-1329991747 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_c5f04e05-b556-4cb3-84c8-590aa69695b0 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:23:45 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-f98767e8-b476-4db5-9f1a-3f92d731e3c6 tempest-GroupSnapshotsTest-1926061586 None] Geting {'code': 204} from DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_74f7ff47-86dd-433a-8a61-f46278efe3fd/snapshots/s_f815ac77-8ec1-4c5c-a0a6-a00cb2779806 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:23:45 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-f98767e8-b476-4db5-9f1a-3f92d731e3c6 tempest-GroupSnapshotsTest-1926061586 None] snapshot s_f815ac77-8ec1-4c5c-a0a6-a00cb2779806 deleted {{(pid=84696) delete_snapshot /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:772}} Apr 17 15:23:45 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-16e1020c-fa96-497c-86b8-f8a35e1f0dfb tempest-VolumesTestJSON-1329991747 None] Geting {'code': 200, 'error': None, 'data': {'origin': None, 'relatime': None, 'acltype': None, 'vscan': None, 'full_name': 'Pool-0/v_c5f04e05-b556-4cb3-84c8-590aa69695b0', 'userrefs': None, 'primarycache': 'all', 'logbias': 'latency', 'creation': '1681746400', 'sync': 'always', 'is_clone': False, 'dedup': 'off', 'sharenfs': None, 'receive_resume_token': None, 'volsize': '1073741824', 'referenced': '57344', 'sharesmb': None, 'createtxg': '187792', 'reservation': '0', 'scontext': None, 'mountpoint': None, 'casesensitivity': None, 'guid': '14758680439539949879', 'usedbyrefreservation': '0', 'dnodesize': None, 'written': '57344', 'logicalused': '28672', 'compressratio': '1.00', 'rootcontext': 'none', 'default_scsi_id': 'ea1fa42682607c9d', 'type': 'volume', 'compression': 'lz4', 'snapdir': None, 'overlay': None, 'encryption': 'off', 'xattr': None, 'volmode': 'default', 'copies': '1', 'snapshot_limit': '18446744073709551615', 'aclinherit': None, 'defcontext': 'none', 'readonly': 'off', 'version': None, 'recordsize': None, 'filesystem_limit': None, 'mounted': None, 'mlslabel': 'none', 'secondarycache': 'all', 'refreservation': '0', 'available': '113926643712', 'san:volume_id': 'ea1fa42682607c9df810d6c2785a993be604dde4', 'encryptionroot': None, 'exec': None, 'refquota': None, 'refcompressratio': '1.00', 'quota': None, 'utf8only': None, 'keylocation': 'none', 'snapdev': 'hidden', 'snapshot_count': '18446744073709551615', 'fscontext': 'none', 'clones': None, 'canmount': None, 'keystatus': None, 'atime': None, 'usedbysnapshots': '0', 'normalization': None, 'usedbychildren': '0', 'volblocksize': '131072', 'usedbydataset': '57344', 'objsetid': '122238', 'name': 'v_c5f04e05-b556-4cb3-84c8-590aa69695b0', 'defer_destroy': None, 'pbkdf2iters': '0', 'checksum': 'on', 'redundant_metadata': 'all', 'filesystem_count': None, 'devices': None, 'keyformat': 'none', 'setuid': None, 'used': '57344', 'logicalreferenced': '28672', 'context': 'none', 'zoned': None, 'nbmand': None}} from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_c5f04e05-b556-4cb3-84c8-590aa69695b0 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:23:45 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-16e1020c-fa96-497c-86b8-f8a35e1f0dfb tempest-VolumesTestJSON-1329991747 None] delete volume:v_c5f04e05-b556-4cb3-84c8-590aa69695b0 recursively children:{'force_umount': True} {{(pid=84696) delete_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:335}} Apr 17 15:23:45 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-16e1020c-fa96-497c-86b8-f8a35e1f0dfb tempest-VolumesTestJSON-1329991747 None] Sending pool request DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_c5f04e05-b556-4cb3-84c8-590aa69695b0 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:23:45 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-16e1020c-fa96-497c-86b8-f8a35e1f0dfb tempest-VolumesTestJSON-1329991747 None] Sending DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_c5f04e05-b556-4cb3-84c8-590aa69695b0 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:23:45 user cinder-volume[84696]: WARNING cinder.quota [None req-f98767e8-b476-4db5-9f1a-3f92d731e3c6 tempest-GroupSnapshotsTest-1926061586 None] Deprecated: Default quota for resource: snapshots_tempest-GroupSnapshotsTest-volume-type-344582814 is set by the default quota flag: quota_snapshots_tempest-GroupSnapshotsTest-volume-type-344582814, it is now deprecated. Please use the default quota class for default quota. Apr 17 15:23:45 user cinder-volume[84696]: DEBUG cinder.quota [None req-f98767e8-b476-4db5-9f1a-3f92d731e3c6 tempest-GroupSnapshotsTest-1926061586 None] Created reservations ['a8898182-70d1-4224-84f4-0d8d4d379db1', 'f784fe2f-76c9-47bd-881f-47bdcaae22c6', 'b3571918-6099-4af0-bd72-4188dbe3e714', '38dbd665-143b-40c6-b535-8c84ec30a1de'] {{(pid=84696) reserve /opt/stack/cinder/cinder/quota.py:755}} Apr 17 15:23:45 user cinder-volume[84696]: INFO cinder.volume.manager [None req-f98767e8-b476-4db5-9f1a-3f92d731e3c6 tempest-GroupSnapshotsTest-1926061586 None] group_snapshot e790b696-22cc-4fac-8b67-57499e6de8b3: deleted successfully Apr 17 15:23:45 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-3057b4d3-c7b5-4dbd-8911-ddf44a44c1e7 tempest-GroupSnapshotsTest-1926061586 None] remove_export for volume: 74f7ff47-86dd-433a-8a61-f46278efe3fd. {{(pid=84696) remove_export /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:718}} Apr 17 15:23:45 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-3057b4d3-c7b5-4dbd-8911-ddf44a44c1e7 tempest-GroupSnapshotsTest-1926061586 None] remove export {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:912}} Apr 17 15:23:45 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-3057b4d3-c7b5-4dbd-8911-ddf44a44c1e7 tempest-GroupSnapshotsTest-1926061586 None] detach volume:Volume(_name_id=None,admin_metadata={},attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:23:28Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-volume-1843290826',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=203550cd-a3a4-4ff5-bfe6-6ed2607e334b,host='user@joviandss-2#joviandss-2',id=74f7ff47-86dd-433a-8a61-f46278efe3fd,launched_at=2023-04-17T15:23:30Z,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='6cae12f3c2614d6fb32f21db60fec8d3',provider_auth='CHAP hoLHQqZi EG7nRVX38zaA1Z',provider_geometry=None,provider_id=None,provider_location='172.16.0.220:3260,1 iqn.2010-10.org.openstack:74f7ff47-86dd-433a-8a61-f46278efe3fd 0',replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:23:28Z,service_uuid='e628352e-c4cf-40eb-8389-cce40663026a',shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='deleting',terminated_at=None,updated_at=2023-04-17T15:23:46Z,use_quota=True,user_id='560c850c324949d7a214e7a59dd65f8d',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(a40cb290-464c-4f83-9d6b-adf225cc2e0c),volume_type_id=a40cb290-464c-4f83-9d6b-adf225cc2e0c) from target:iqn.2010-10.org.openstack:74f7ff47-86dd-433a-8a61-f46278efe3fd. {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:913}} Apr 17 15:23:45 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-3057b4d3-c7b5-4dbd-8911-ddf44a44c1e7 tempest-GroupSnapshotsTest-1926061586 None] detach volume v_74f7ff47-86dd-433a-8a61-f46278efe3fd from target iqn.2010-10.org.openstack:74f7ff47-86dd-433a-8a61-f46278efe3fd {{(pid=84696) detach_target_vol /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:606}} Apr 17 15:23:45 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-3057b4d3-c7b5-4dbd-8911-ddf44a44c1e7 tempest-GroupSnapshotsTest-1926061586 None] Sending pool request DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:74f7ff47-86dd-433a-8a61-f46278efe3fd/luns/v_74f7ff47-86dd-433a-8a61-f46278efe3fd {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:23:45 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-3057b4d3-c7b5-4dbd-8911-ddf44a44c1e7 tempest-GroupSnapshotsTest-1926061586 None] Sending DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:74f7ff47-86dd-433a-8a61-f46278efe3fd/luns/v_74f7ff47-86dd-433a-8a61-f46278efe3fd {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:23:45 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-3057b4d3-c7b5-4dbd-8911-ddf44a44c1e7 tempest-GroupSnapshotsTest-1926061586 None] Geting {'code': 404, 'error': {'class': 'opene.exceptions.ItemNotFoundError', 'message': 'Target iqn.2010-10.org.openstack:74f7ff47-86dd-433a-8a61-f46278efe3fd not exists.', 'url': 'http://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:74f7ff47-86dd-433a-8a61-f46278efe3fd/luns/v_74f7ff47-86dd-433a-8a61-f46278efe3fd'}, 'data': None} from DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:74f7ff47-86dd-433a-8a61-f46278efe3fd/luns/v_74f7ff47-86dd-433a-8a61-f46278efe3fd {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:23:45 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-3057b4d3-c7b5-4dbd-8911-ddf44a44c1e7 tempest-GroupSnapshotsTest-1926061586 None] failed to remove resource iqn.2010-10.org.openstack:74f7ff47-86dd-433a-8a61-f46278efe3fd because of JDSS resource v_74f7ff47-86dd-433a-8a61-f46278efe3fd DNE. {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:920}} Apr 17 15:23:45 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-3057b4d3-c7b5-4dbd-8911-ddf44a44c1e7 tempest-GroupSnapshotsTest-1926061586 None] delete target: iqn.2010-10.org.openstack:74f7ff47-86dd-433a-8a61-f46278efe3fd {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:930}} Apr 17 15:23:45 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-3057b4d3-c7b5-4dbd-8911-ddf44a44c1e7 tempest-GroupSnapshotsTest-1926061586 None] delete target iqn.2010-10.org.openstack:74f7ff47-86dd-433a-8a61-f46278efe3fd {{(pid=84696) delete_target /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:442}} Apr 17 15:23:45 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-3057b4d3-c7b5-4dbd-8911-ddf44a44c1e7 tempest-GroupSnapshotsTest-1926061586 None] Sending pool request DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:74f7ff47-86dd-433a-8a61-f46278efe3fd {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:23:45 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-3057b4d3-c7b5-4dbd-8911-ddf44a44c1e7 tempest-GroupSnapshotsTest-1926061586 None] Sending DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:74f7ff47-86dd-433a-8a61-f46278efe3fd {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:23:45 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-3057b4d3-c7b5-4dbd-8911-ddf44a44c1e7 tempest-GroupSnapshotsTest-1926061586 None] Geting {'code': 404, 'error': {'class': 'opene.exceptions.ItemNotFoundError', 'message': 'Target iqn.2010-10.org.openstack:74f7ff47-86dd-433a-8a61-f46278efe3fd not exists.', 'url': 'http://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:74f7ff47-86dd-433a-8a61-f46278efe3fd'}, 'data': None} from DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:74f7ff47-86dd-433a-8a61-f46278efe3fd {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:23:45 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-3057b4d3-c7b5-4dbd-8911-ddf44a44c1e7 tempest-GroupSnapshotsTest-1926061586 None] failed to remove resource iqn.2010-10.org.openstack:74f7ff47-86dd-433a-8a61-f46278efe3fd because of JDSS resource iqn.2010-10.org.openstack:74f7ff47-86dd-433a-8a61-f46278efe3fd DNE. {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:935}} Apr 17 15:23:45 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-3057b4d3-c7b5-4dbd-8911-ddf44a44c1e7 tempest-GroupSnapshotsTest-1926061586 None] deleating volume v_74f7ff47-86dd-433a-8a61-f46278efe3fd {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 15:23:45 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-3057b4d3-c7b5-4dbd-8911-ddf44a44c1e7 tempest-GroupSnapshotsTest-1926061586 None] get snapshots for volume v_74f7ff47-86dd-433a-8a61-f46278efe3fd {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 15:23:45 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-3057b4d3-c7b5-4dbd-8911-ddf44a44c1e7 tempest-GroupSnapshotsTest-1926061586 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_74f7ff47-86dd-433a-8a61-f46278efe3fd/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:23:45 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-3057b4d3-c7b5-4dbd-8911-ddf44a44c1e7 tempest-GroupSnapshotsTest-1926061586 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_74f7ff47-86dd-433a-8a61-f46278efe3fd/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:23:46 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-3057b4d3-c7b5-4dbd-8911-ddf44a44c1e7 tempest-GroupSnapshotsTest-1926061586 None] Geting {'code': 200, 'error': None, 'data': {'results': 0, 'entries': []}} from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_74f7ff47-86dd-433a-8a61-f46278efe3fd/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:23:46 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-3057b4d3-c7b5-4dbd-8911-ddf44a44c1e7 tempest-GroupSnapshotsTest-1926061586 None] get volume v_74f7ff47-86dd-433a-8a61-f46278efe3fd info {{(pid=84696) get_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:239}} Apr 17 15:23:46 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-3057b4d3-c7b5-4dbd-8911-ddf44a44c1e7 tempest-GroupSnapshotsTest-1926061586 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_74f7ff47-86dd-433a-8a61-f46278efe3fd {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:23:46 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-3057b4d3-c7b5-4dbd-8911-ddf44a44c1e7 tempest-GroupSnapshotsTest-1926061586 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_74f7ff47-86dd-433a-8a61-f46278efe3fd {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:23:46 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-3057b4d3-c7b5-4dbd-8911-ddf44a44c1e7 tempest-GroupSnapshotsTest-1926061586 None] Geting {'code': 200, 'error': None, 'data': {'origin': None, 'relatime': None, 'acltype': None, 'vscan': None, 'full_name': 'Pool-0/v_74f7ff47-86dd-433a-8a61-f46278efe3fd', 'userrefs': None, 'primarycache': 'all', 'logbias': 'latency', 'creation': '1681746402', 'sync': 'always', 'is_clone': False, 'dedup': 'off', 'sharenfs': None, 'receive_resume_token': None, 'volsize': '1073741824', 'referenced': '57344', 'sharesmb': None, 'createtxg': '187799', 'reservation': '0', 'scontext': None, 'mountpoint': None, 'casesensitivity': None, 'guid': '1604825945791133178', 'usedbyrefreservation': '0', 'dnodesize': None, 'written': '57344', 'logicalused': '28672', 'compressratio': '1.00', 'rootcontext': 'none', 'default_scsi_id': '1a021ce2147b156c', 'type': 'volume', 'compression': 'lz4', 'snapdir': None, 'overlay': None, 'encryption': 'off', 'xattr': None, 'volmode': 'default', 'copies': '1', 'snapshot_limit': '18446744073709551615', 'aclinherit': None, 'defcontext': 'none', 'readonly': 'off', 'version': None, 'recordsize': None, 'filesystem_limit': None, 'mounted': None, 'mlslabel': 'none', 'secondarycache': 'all', 'refreservation': '0', 'available': '113926643712', 'san:volume_id': '1a021ce2147b156c926c79cc62997adcd404327b', 'encryptionroot': None, 'exec': None, 'refquota': None, 'refcompressratio': '1.00', 'quota': None, 'utf8only': None, 'keylocation': 'none', 'snapdev': 'hidden', 'snapshot_count': '18446744073709551615', 'fscontext': 'none', 'clones': None, 'canmount': None, 'keystatus': None, 'atime': None, 'usedbysnapshots': '0', 'normalization': None, 'usedbychildren': '0', 'volblocksize': '131072', 'usedbydataset': '57344', 'objsetid': '123012', 'name': 'v_74f7ff47-86dd-433a-8a61-f46278efe3fd', 'defer_destroy': None, 'pbkdf2iters': '0', 'checksum': 'on', 'redundant_metadata': 'all', 'filesystem_count': None, 'devices': None, 'keyformat': 'none', 'setuid': None, 'used': '57344', 'logicalreferenced': '28672', 'context': 'none', 'zoned': None, 'nbmand': None}} from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_74f7ff47-86dd-433a-8a61-f46278efe3fd {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:23:46 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-3057b4d3-c7b5-4dbd-8911-ddf44a44c1e7 tempest-GroupSnapshotsTest-1926061586 None] delete volume:v_74f7ff47-86dd-433a-8a61-f46278efe3fd recursively children:{'force_umount': True} {{(pid=84696) delete_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:335}} Apr 17 15:23:46 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-3057b4d3-c7b5-4dbd-8911-ddf44a44c1e7 tempest-GroupSnapshotsTest-1926061586 None] Sending pool request DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_74f7ff47-86dd-433a-8a61-f46278efe3fd {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:23:46 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-3057b4d3-c7b5-4dbd-8911-ddf44a44c1e7 tempest-GroupSnapshotsTest-1926061586 None] Sending DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_74f7ff47-86dd-433a-8a61-f46278efe3fd {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:23:47 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-16e1020c-fa96-497c-86b8-f8a35e1f0dfb tempest-VolumesTestJSON-1329991747 None] Geting {'code': 204} from DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_c5f04e05-b556-4cb3-84c8-590aa69695b0 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:23:47 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-16e1020c-fa96-497c-86b8-f8a35e1f0dfb tempest-VolumesTestJSON-1329991747 None] volume v_c5f04e05-b556-4cb3-84c8-590aa69695b0 deleted {{(pid=84696) delete_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:346}} Apr 17 15:23:47 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=56,cinder:INSERT=6,cinder:UPDATE=38 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:23:47 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=63,cinder:UPDATE=27,cinder:DELETE=2,cinder:INSERT=8 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:23:47 user cinder-volume[84696]: DEBUG cinder.quota [None req-16e1020c-fa96-497c-86b8-f8a35e1f0dfb tempest-VolumesTestJSON-1329991747 None] Created reservations ['5fd125b5-abf6-4b52-8de6-ccd2f5b805e5', '173bc077-2164-489f-ba0a-d7da741b9e15', 'c8b02f18-15db-4d43-a49f-a58e41f2130b', 'a2089334-b81e-4480-9a07-ae865aab3268'] {{(pid=84696) reserve /opt/stack/cinder/cinder/quota.py:755}} Apr 17 15:23:47 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-16e1020c-fa96-497c-86b8-f8a35e1f0dfb tempest-VolumesTestJSON-1329991747 None] Updating volume stats {{(pid=84696) _update_volume_stats /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:724}} Apr 17 15:23:47 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-16e1020c-fa96-497c-86b8-f8a35e1f0dfb tempest-VolumesTestJSON-1329991747 None] Get pool Pool-0 fsprops {{(pid=84696) get_pool_stats /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:894}} Apr 17 15:23:47 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-16e1020c-fa96-497c-86b8-f8a35e1f0dfb tempest-VolumesTestJSON-1329991747 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:23:47 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-16e1020c-fa96-497c-86b8-f8a35e1f0dfb tempest-VolumesTestJSON-1329991747 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:23:48 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-16e1020c-fa96-497c-86b8-f8a35e1f0dfb tempest-VolumesTestJSON-1329991747 None] Geting {'code': 200, 'error': None, 'data': {'available': '113926725632', 'status': 30, 'import_status': {'import_steps': {'san_setup': True, 'vip_setup': True, 'zfs_import': True, 'mount_system_volume': True, 'nas_setup': True}, 'import_successful': True}, 'scan': None, 'encryption': {'enabled': False}, 'deduplication_rate': '1.00', 'sysvol_upgrade': 'not_needed', 'vdevs': [{'name': 'wwn-0x6000c296b793bf526a107a9d823e947a', 'iostats': {'read': '0', 'write': '0', 'chksum': '0'}, 'disks': [{'origin': 'local', 'slot': '', 'led': 'off', 'name': 'sdn', 'iostats': {'read': '0', 'write': '0', 'chksum': '0'}, 'alias': '', 'health': 'ONLINE', 'sn': '6000c296b793bf526a107a9d823e947a', 'trim_data': {'status': 'trim_unsupported', 'progress': None, 'trim_time': None}, 'path': 'pci-0000:03:00.0-scsi-0:0:14:0', 'model': 'Virtual disk', 'id': 'wwn-0x6000c296b793bf526a107a9d823e947a', 'size': 128849018880}], 'health': 'ONLINE', 'vdev_replacings': [], 'vdev_spares': [], 'type': ''}], 'id': '14181317334440137330', 'health': 'ONLINE', 'iostats': {'read': '0', 'write': '0', 'chksum': '0'}, 'operation': 'none', 'size': '127775277056', 'autotrim': False, 'name': 'Pool-0'}} from GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:23:48 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-16e1020c-fa96-497c-86b8-f8a35e1f0dfb tempest-VolumesTestJSON-1329991747 None] Total capacity: 119, Free 106. {{(pid=84696) _update_volume_stats /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:757}} Apr 17 15:23:48 user cinder-volume[84696]: DEBUG cinder.manager [None req-16e1020c-fa96-497c-86b8-f8a35e1f0dfb tempest-VolumesTestJSON-1329991747 None] Notifying Schedulers of capabilities ... {{(pid=84696) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 17 15:23:49 user cinder-volume[84696]: INFO cinder.volume.manager [None req-16e1020c-fa96-497c-86b8-f8a35e1f0dfb tempest-VolumesTestJSON-1329991747 None] Deleted volume successfully. Apr 17 15:23:49 user cinder-volume[84696]: DEBUG cinder.coordination [None req-16e1020c-fa96-497c-86b8-f8a35e1f0dfb tempest-VolumesTestJSON-1329991747 None] Lock "cinder-c5f04e05-b556-4cb3-84c8-590aa69695b0-delete_volume" released by "delete_volume" :: held 4.768s {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:207}} Apr 17 15:23:50 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-3057b4d3-c7b5-4dbd-8911-ddf44a44c1e7 tempest-GroupSnapshotsTest-1926061586 None] Geting {'code': 204} from DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_74f7ff47-86dd-433a-8a61-f46278efe3fd {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:23:50 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-3057b4d3-c7b5-4dbd-8911-ddf44a44c1e7 tempest-GroupSnapshotsTest-1926061586 None] volume v_74f7ff47-86dd-433a-8a61-f46278efe3fd deleted {{(pid=84696) delete_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:346}} Apr 17 15:23:50 user cinder-volume[84696]: DEBUG cinder.quota [None req-3057b4d3-c7b5-4dbd-8911-ddf44a44c1e7 tempest-GroupSnapshotsTest-1926061586 None] Created reservations ['01aedab8-7ecc-4188-8c17-9bd4780de05e'] {{(pid=84696) reserve /opt/stack/cinder/cinder/quota.py:755}} Apr 17 15:23:50 user cinder-volume[84696]: WARNING cinder.db.sqlalchemy.api [None req-3057b4d3-c7b5-4dbd-8911-ddf44a44c1e7 tempest-GroupSnapshotsTest-1926061586 None] Reservation would make usage less than 0 for the following resources, so on commit they will be limited to prevent going below 0: ['gigabytes', 'gigabytes_tempest-GroupSnapshotsTest-volume-type-344582814'] Apr 17 15:23:50 user cinder-volume[84696]: DEBUG cinder.quota [None req-3057b4d3-c7b5-4dbd-8911-ddf44a44c1e7 tempest-GroupSnapshotsTest-1926061586 None] Created reservations ['645c157d-98da-4d36-87ba-5d248cf4fdfc', '00c21870-1679-4eb4-bcf1-22662f6bbf0c', '319ad9f7-f3b7-4161-badf-22a0b267b763', '6f228482-78a5-4e44-99ae-fb06785e7d84'] {{(pid=84696) reserve /opt/stack/cinder/cinder/quota.py:755}} Apr 17 15:23:50 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-3057b4d3-c7b5-4dbd-8911-ddf44a44c1e7 tempest-GroupSnapshotsTest-1926061586 None] Updating volume stats {{(pid=84696) _update_volume_stats /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:724}} Apr 17 15:23:50 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-3057b4d3-c7b5-4dbd-8911-ddf44a44c1e7 tempest-GroupSnapshotsTest-1926061586 None] Get pool Pool-0 fsprops {{(pid=84696) get_pool_stats /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:894}} Apr 17 15:23:50 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-3057b4d3-c7b5-4dbd-8911-ddf44a44c1e7 tempest-GroupSnapshotsTest-1926061586 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:23:50 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-3057b4d3-c7b5-4dbd-8911-ddf44a44c1e7 tempest-GroupSnapshotsTest-1926061586 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:23:51 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-3057b4d3-c7b5-4dbd-8911-ddf44a44c1e7 tempest-GroupSnapshotsTest-1926061586 None] Geting {'code': 200, 'error': None, 'data': {'available': '113926807552', 'status': 30, 'import_status': {'import_steps': {'san_setup': True, 'vip_setup': True, 'zfs_import': True, 'mount_system_volume': True, 'nas_setup': True}, 'import_successful': True}, 'scan': None, 'encryption': {'enabled': False}, 'deduplication_rate': '1.00', 'sysvol_upgrade': 'not_needed', 'vdevs': [{'name': 'wwn-0x6000c296b793bf526a107a9d823e947a', 'iostats': {'read': '0', 'write': '0', 'chksum': '0'}, 'disks': [{'origin': 'local', 'slot': '', 'led': 'off', 'name': 'sdn', 'iostats': {'read': '0', 'write': '0', 'chksum': '0'}, 'alias': '', 'health': 'ONLINE', 'sn': '6000c296b793bf526a107a9d823e947a', 'trim_data': {'status': 'trim_unsupported', 'progress': None, 'trim_time': None}, 'path': 'pci-0000:03:00.0-scsi-0:0:14:0', 'model': 'Virtual disk', 'id': 'wwn-0x6000c296b793bf526a107a9d823e947a', 'size': 128849018880}], 'health': 'ONLINE', 'vdev_replacings': [], 'vdev_spares': [], 'type': ''}], 'id': '14181317334440137330', 'health': 'ONLINE', 'iostats': {'read': '0', 'write': '0', 'chksum': '0'}, 'operation': 'none', 'size': '127775277056', 'autotrim': False, 'name': 'Pool-0'}} from GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:23:51 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-3057b4d3-c7b5-4dbd-8911-ddf44a44c1e7 tempest-GroupSnapshotsTest-1926061586 None] Total capacity: 119, Free 106. {{(pid=84696) _update_volume_stats /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:757}} Apr 17 15:23:51 user cinder-volume[84696]: DEBUG cinder.manager [None req-3057b4d3-c7b5-4dbd-8911-ddf44a44c1e7 tempest-GroupSnapshotsTest-1926061586 None] Notifying Schedulers of capabilities ... {{(pid=84696) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 17 15:23:51 user cinder-volume[84696]: INFO cinder.volume.manager [None req-3057b4d3-c7b5-4dbd-8911-ddf44a44c1e7 tempest-GroupSnapshotsTest-1926061586 None] Delete group completed successfully. Apr 17 15:23:51 user cinder-volume[84696]: INFO cinder.volume.manager [None req-3c81aaab-1504-4ec3-ace1-79b934b9bbd8 tempest-GroupSnapshotsTest-1926061586 None] Group tempest-GroupSnapshotsTest-Group-300554396: creating Apr 17 15:23:51 user cinder-volume[84696]: INFO cinder.volume.manager [None req-3c81aaab-1504-4ec3-ace1-79b934b9bbd8 tempest-GroupSnapshotsTest-1926061586 None] Group tempest-GroupSnapshotsTest-Group-300554396: created successfully Apr 17 15:23:51 user cinder-volume[84696]: INFO cinder.volume.manager [None req-3c81aaab-1504-4ec3-ace1-79b934b9bbd8 tempest-GroupSnapshotsTest-1926061586 None] Create group completed successfully. Apr 17 15:23:53 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-5f9b222b-dc4a-4328-8276-c24281b88b12 tempest-GroupSnapshotsTest-1926061586 None] Volume reschedule parameters: True retry: {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 15:23:53 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-5f9b222b-dc4a-4328-8276-c24281b88b12 tempest-GroupSnapshotsTest-1926061586 None] Flow 'volume_create_manager' (3a43a5b8-a6cb-44ec-831e-82fa65f773f2) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:23:53 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-5f9b222b-dc4a-4328-8276-c24281b88b12 tempest-GroupSnapshotsTest-1926061586 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (9f50e1b9-e925-49d4-8240-b24256d48a6a) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:23:53 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-5f9b222b-dc4a-4328-8276-c24281b88b12 tempest-GroupSnapshotsTest-1926061586 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (9f50e1b9-e925-49d4-8240-b24256d48a6a) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:23:53Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-volume-231798850',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=286e53b0-285a-4d83-8cd3-cb2b5641b31f,host='user@joviandss-2#joviandss-2',id=7249f8e4-ed4b-4689-aba3-2766c924f42c,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='6cae12f3c2614d6fb32f21db60fec8d3',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:23:53Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:23:53Z,use_quota=True,user_id='560c850c324949d7a214e7a59dd65f8d',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(73c9cbe6-d3bc-4af2-810c-855e0b88bd3f),volume_type_id=73c9cbe6-d3bc-4af2-810c-855e0b88bd3f)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:23:53 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-5f9b222b-dc4a-4328-8276-c24281b88b12 tempest-GroupSnapshotsTest-1926061586 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (663b4f25-b997-4d4b-912c-84c7479e5894) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:23:53 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-5f9b222b-dc4a-4328-8276-c24281b88b12 tempest-GroupSnapshotsTest-1926061586 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (663b4f25-b997-4d4b-912c-84c7479e5894) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:23:53 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-5f9b222b-dc4a-4328-8276-c24281b88b12 tempest-GroupSnapshotsTest-1926061586 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (6883fd9d-72fd-4dc3-94d4-f8ce5bfa0b14) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:23:53 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-5f9b222b-dc4a-4328-8276-c24281b88b12 tempest-GroupSnapshotsTest-1926061586 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (6883fd9d-72fd-4dc3-94d4-f8ce5bfa0b14) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': '7249f8e4-ed4b-4689-aba3-2766c924f42c', 'volume_name': 'volume-7249f8e4-ed4b-4689-aba3-2766c924f42c', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:23:53 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-5f9b222b-dc4a-4328-8276-c24281b88b12 tempest-GroupSnapshotsTest-1926061586 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (5ed69a8c-521a-4af0-8cc8-9de261ac0e7a) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:23:53 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-5f9b222b-dc4a-4328-8276-c24281b88b12 tempest-GroupSnapshotsTest-1926061586 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (5ed69a8c-521a-4af0-8cc8-9de261ac0e7a) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:23:53 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-5f9b222b-dc4a-4328-8276-c24281b88b12 tempest-GroupSnapshotsTest-1926061586 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (91d0ec82-f9f9-46fb-acf5-983b1e9ea169) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:23:53 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-5f9b222b-dc4a-4328-8276-c24281b88b12 tempest-GroupSnapshotsTest-1926061586 None] Volume 7249f8e4-ed4b-4689-aba3-2766c924f42c: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-7249f8e4-ed4b-4689-aba3-2766c924f42c', 'volume_size': 1} Apr 17 15:23:53 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-5f9b222b-dc4a-4328-8276-c24281b88b12 tempest-GroupSnapshotsTest-1926061586 None] creating volume v_7249f8e4-ed4b-4689-aba3-2766c924f42c. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 15:23:53 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-5f9b222b-dc4a-4328-8276-c24281b88b12 tempest-GroupSnapshotsTest-1926061586 None] create volume {'name': 'v_7249f8e4-ed4b-4689-aba3-2766c924f42c', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 15:23:53 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5f9b222b-dc4a-4328-8276-c24281b88b12 tempest-GroupSnapshotsTest-1926061586 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:23:53 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5f9b222b-dc4a-4328-8276-c24281b88b12 tempest-GroupSnapshotsTest-1926061586 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:23:54 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5f9b222b-dc4a-4328-8276-c24281b88b12 tempest-GroupSnapshotsTest-1926061586 None] Geting {'code': 201, 'error': None, 'data': {'full_name': 'Pool-0/v_7249f8e4-ed4b-4689-aba3-2766c924f42c', 'is_clone': False, 'name': 'v_7249f8e4-ed4b-4689-aba3-2766c924f42c'}} from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:23:54 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-5f9b222b-dc4a-4328-8276-c24281b88b12 tempest-GroupSnapshotsTest-1926061586 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (91d0ec82-f9f9-46fb-acf5-983b1e9ea169) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'volume_name': 'volume-7249f8e4-ed4b-4689-aba3-2766c924f42c', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:23:54 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-5f9b222b-dc4a-4328-8276-c24281b88b12 tempest-GroupSnapshotsTest-1926061586 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeOnFinishTask;volume:create, create.end' (6a085dc4-572e-4e9f-8537-07c1c34b956d) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:23:54 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-5f9b222b-dc4a-4328-8276-c24281b88b12 tempest-GroupSnapshotsTest-1926061586 None] Volume volume-7249f8e4-ed4b-4689-aba3-2766c924f42c (7249f8e4-ed4b-4689-aba3-2766c924f42c): created successfully Apr 17 15:23:54 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-5f9b222b-dc4a-4328-8276-c24281b88b12 tempest-GroupSnapshotsTest-1926061586 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeOnFinishTask;volume:create, create.end' (6a085dc4-572e-4e9f-8537-07c1c34b956d) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:23:54 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-5f9b222b-dc4a-4328-8276-c24281b88b12 tempest-GroupSnapshotsTest-1926061586 None] Flow 'volume_create_manager' (3a43a5b8-a6cb-44ec-831e-82fa65f773f2) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:23:54 user cinder-volume[84696]: INFO cinder.volume.manager [None req-5f9b222b-dc4a-4328-8276-c24281b88b12 tempest-GroupSnapshotsTest-1926061586 None] Created volume successfully. Apr 17 15:23:55 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-f2715cc1-a4b1-4939-9297-fae00819ee03 tempest-VolumesCloneNegativeTest-166008807 None] Volume reschedule parameters: True retry: {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 15:23:55 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f2715cc1-a4b1-4939-9297-fae00819ee03 tempest-VolumesCloneNegativeTest-166008807 None] Flow 'volume_create_manager' (9b50d380-14dd-40be-8e53-57319414cafe) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:23:55 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f2715cc1-a4b1-4939-9297-fae00819ee03 tempest-VolumesCloneNegativeTest-166008807 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (8228d8de-9642-448a-a799-e839638f2ca5) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:23:55 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f2715cc1-a4b1-4939-9297-fae00819ee03 tempest-VolumesCloneNegativeTest-166008807 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (8228d8de-9642-448a-a799-e839638f2ca5) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:23:55Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesCloneNegativeTest-Volume-1363201589',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=ff4859e2-8777-4959-8da6-2086553c6247,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='d95a37e41b1d433d92d16ae882a22070',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:23:56Z,service_uuid=None,shared_targets=True,size=2,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:23:56Z,use_quota=True,user_id='b92392662be14505953e5b4a3903dbbb',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:23:55 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f2715cc1-a4b1-4939-9297-fae00819ee03 tempest-VolumesCloneNegativeTest-166008807 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (40462cf2-c72b-48da-a5fb-f28229aac127) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:23:55 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f2715cc1-a4b1-4939-9297-fae00819ee03 tempest-VolumesCloneNegativeTest-166008807 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (40462cf2-c72b-48da-a5fb-f28229aac127) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:23:55 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f2715cc1-a4b1-4939-9297-fae00819ee03 tempest-VolumesCloneNegativeTest-166008807 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (b70dd264-f30c-488e-ba6c-c101d998c769) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:23:55 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f2715cc1-a4b1-4939-9297-fae00819ee03 tempest-VolumesCloneNegativeTest-166008807 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (b70dd264-f30c-488e-ba6c-c101d998c769) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': 'ff4859e2-8777-4959-8da6-2086553c6247', 'volume_name': 'volume-ff4859e2-8777-4959-8da6-2086553c6247', 'volume_size': 2}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:23:55 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f2715cc1-a4b1-4939-9297-fae00819ee03 tempest-VolumesCloneNegativeTest-166008807 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (cec787ce-0b0b-45d5-8070-cef618d62031) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:23:55 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f2715cc1-a4b1-4939-9297-fae00819ee03 tempest-VolumesCloneNegativeTest-166008807 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (cec787ce-0b0b-45d5-8070-cef618d62031) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:23:55 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f2715cc1-a4b1-4939-9297-fae00819ee03 tempest-VolumesCloneNegativeTest-166008807 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (2bbb506e-24c9-4e2b-b721-8c24d89e5f29) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:23:55 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-f2715cc1-a4b1-4939-9297-fae00819ee03 tempest-VolumesCloneNegativeTest-166008807 None] Volume ff4859e2-8777-4959-8da6-2086553c6247: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-ff4859e2-8777-4959-8da6-2086553c6247', 'volume_size': 2} Apr 17 15:23:55 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-f2715cc1-a4b1-4939-9297-fae00819ee03 tempest-VolumesCloneNegativeTest-166008807 None] creating volume v_ff4859e2-8777-4959-8da6-2086553c6247. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 15:23:55 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-f2715cc1-a4b1-4939-9297-fae00819ee03 tempest-VolumesCloneNegativeTest-166008807 None] create volume {'name': 'v_ff4859e2-8777-4959-8da6-2086553c6247', 'size': '2147483648', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 15:23:55 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-f2715cc1-a4b1-4939-9297-fae00819ee03 tempest-VolumesCloneNegativeTest-166008807 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:23:55 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-f2715cc1-a4b1-4939-9297-fae00819ee03 tempest-VolumesCloneNegativeTest-166008807 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:23:56 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-47c0924f-fe01-4cbe-b793-24ff6b90fa3a tempest-GroupSnapshotsTest-1926061586 None] Volume reschedule parameters: True retry: {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 15:23:56 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-47c0924f-fe01-4cbe-b793-24ff6b90fa3a tempest-GroupSnapshotsTest-1926061586 None] Flow 'volume_create_manager' (a004c7ef-2ce5-44ca-bb24-27b528e12992) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:23:56 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-47c0924f-fe01-4cbe-b793-24ff6b90fa3a tempest-GroupSnapshotsTest-1926061586 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (51589463-e8c6-4361-8e5e-d233aaed4b4a) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:23:56 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-47c0924f-fe01-4cbe-b793-24ff6b90fa3a tempest-GroupSnapshotsTest-1926061586 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (51589463-e8c6-4361-8e5e-d233aaed4b4a) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:23:56Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-volume-591301068',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=286e53b0-285a-4d83-8cd3-cb2b5641b31f,host='user@joviandss-2#joviandss-2',id=f6603de5-71d3-468b-a673-ce1ff5cd1285,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='6cae12f3c2614d6fb32f21db60fec8d3',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:23:56Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:23:56Z,use_quota=True,user_id='560c850c324949d7a214e7a59dd65f8d',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(73c9cbe6-d3bc-4af2-810c-855e0b88bd3f),volume_type_id=73c9cbe6-d3bc-4af2-810c-855e0b88bd3f)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:23:56 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-47c0924f-fe01-4cbe-b793-24ff6b90fa3a tempest-GroupSnapshotsTest-1926061586 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (339b1ede-b929-47bf-a669-3e881f726152) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:23:56 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-47c0924f-fe01-4cbe-b793-24ff6b90fa3a tempest-GroupSnapshotsTest-1926061586 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (339b1ede-b929-47bf-a669-3e881f726152) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:23:56 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-47c0924f-fe01-4cbe-b793-24ff6b90fa3a tempest-GroupSnapshotsTest-1926061586 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (dcd9216f-9675-423b-8b33-b53abe96d1d2) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:23:56 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-47c0924f-fe01-4cbe-b793-24ff6b90fa3a tempest-GroupSnapshotsTest-1926061586 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (dcd9216f-9675-423b-8b33-b53abe96d1d2) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': 'f6603de5-71d3-468b-a673-ce1ff5cd1285', 'volume_name': 'volume-f6603de5-71d3-468b-a673-ce1ff5cd1285', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:23:56 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-47c0924f-fe01-4cbe-b793-24ff6b90fa3a tempest-GroupSnapshotsTest-1926061586 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (ef84be28-6986-4005-b445-654717833e8e) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:23:56 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-47c0924f-fe01-4cbe-b793-24ff6b90fa3a tempest-GroupSnapshotsTest-1926061586 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (ef84be28-6986-4005-b445-654717833e8e) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:23:56 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-47c0924f-fe01-4cbe-b793-24ff6b90fa3a tempest-GroupSnapshotsTest-1926061586 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (1f9cfa1b-1b1f-4e00-bc62-d3e4972829ee) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:23:56 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-47c0924f-fe01-4cbe-b793-24ff6b90fa3a tempest-GroupSnapshotsTest-1926061586 None] Volume f6603de5-71d3-468b-a673-ce1ff5cd1285: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-f6603de5-71d3-468b-a673-ce1ff5cd1285', 'volume_size': 1} Apr 17 15:23:56 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-47c0924f-fe01-4cbe-b793-24ff6b90fa3a tempest-GroupSnapshotsTest-1926061586 None] creating volume v_f6603de5-71d3-468b-a673-ce1ff5cd1285. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 15:23:56 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-47c0924f-fe01-4cbe-b793-24ff6b90fa3a tempest-GroupSnapshotsTest-1926061586 None] create volume {'name': 'v_f6603de5-71d3-468b-a673-ce1ff5cd1285', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 15:23:56 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-47c0924f-fe01-4cbe-b793-24ff6b90fa3a tempest-GroupSnapshotsTest-1926061586 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:23:56 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-47c0924f-fe01-4cbe-b793-24ff6b90fa3a tempest-GroupSnapshotsTest-1926061586 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:23:56 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-e5f771c1-c99f-4537-96d6-3b271ab54578 tempest-VolumeTypesAccessTest-1000893842 None] Volume reschedule parameters: True retry: {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 15:23:56 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e5f771c1-c99f-4537-96d6-3b271ab54578 tempest-VolumeTypesAccessTest-1000893842 None] Flow 'volume_create_manager' (e3d028b1-0e4d-4929-b11a-3c5a8668bb3c) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:23:56 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e5f771c1-c99f-4537-96d6-3b271ab54578 tempest-VolumeTypesAccessTest-1000893842 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (20d87d5c-1a00-4184-9841-2b96d9b0e787) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:23:56 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e5f771c1-c99f-4537-96d6-3b271ab54578 tempest-VolumeTypesAccessTest-1000893842 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (20d87d5c-1a00-4184-9841-2b96d9b0e787) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:23:56Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumeTypesAccessTest-Volume-1976391366',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=53ff0d0b-94f7-4575-bb23-81cac29166fe,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='f3f0a0101c2840bc9570fc0e98c065cb',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:23:57Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:23:57Z,use_quota=True,user_id='0cd3cc8229eb44748661d1ee5a78ed61',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(5d7108d1-10ea-4a52-bbef-e6ef8e5b8983),volume_type_id=5d7108d1-10ea-4a52-bbef-e6ef8e5b8983)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:23:56 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e5f771c1-c99f-4537-96d6-3b271ab54578 tempest-VolumeTypesAccessTest-1000893842 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (81114c30-8cf2-485a-82f1-74ffbe4ca65a) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:23:56 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e5f771c1-c99f-4537-96d6-3b271ab54578 tempest-VolumeTypesAccessTest-1000893842 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (81114c30-8cf2-485a-82f1-74ffbe4ca65a) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:23:56 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e5f771c1-c99f-4537-96d6-3b271ab54578 tempest-VolumeTypesAccessTest-1000893842 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (8ff92a98-99c2-4892-9281-d0675db8f22f) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:23:56 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e5f771c1-c99f-4537-96d6-3b271ab54578 tempest-VolumeTypesAccessTest-1000893842 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (8ff92a98-99c2-4892-9281-d0675db8f22f) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': '53ff0d0b-94f7-4575-bb23-81cac29166fe', 'volume_name': 'volume-53ff0d0b-94f7-4575-bb23-81cac29166fe', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:23:56 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e5f771c1-c99f-4537-96d6-3b271ab54578 tempest-VolumeTypesAccessTest-1000893842 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (24604fa2-4dee-41c3-b279-ac551865e0b2) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:23:56 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e5f771c1-c99f-4537-96d6-3b271ab54578 tempest-VolumeTypesAccessTest-1000893842 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (24604fa2-4dee-41c3-b279-ac551865e0b2) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:23:56 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e5f771c1-c99f-4537-96d6-3b271ab54578 tempest-VolumeTypesAccessTest-1000893842 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (b128cde9-61ee-4e24-ae33-2caca58215d9) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:23:56 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-e5f771c1-c99f-4537-96d6-3b271ab54578 tempest-VolumeTypesAccessTest-1000893842 None] Volume 53ff0d0b-94f7-4575-bb23-81cac29166fe: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-53ff0d0b-94f7-4575-bb23-81cac29166fe', 'volume_size': 1} Apr 17 15:23:56 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-e5f771c1-c99f-4537-96d6-3b271ab54578 tempest-VolumeTypesAccessTest-1000893842 None] creating volume v_53ff0d0b-94f7-4575-bb23-81cac29166fe. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 15:23:56 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-e5f771c1-c99f-4537-96d6-3b271ab54578 tempest-VolumeTypesAccessTest-1000893842 None] create volume {'name': 'v_53ff0d0b-94f7-4575-bb23-81cac29166fe', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 15:23:56 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-e5f771c1-c99f-4537-96d6-3b271ab54578 tempest-VolumeTypesAccessTest-1000893842 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:23:56 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-e5f771c1-c99f-4537-96d6-3b271ab54578 tempest-VolumeTypesAccessTest-1000893842 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:23:57 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-f2715cc1-a4b1-4939-9297-fae00819ee03 tempest-VolumesCloneNegativeTest-166008807 None] Geting {'code': 201, 'error': None, 'data': {'full_name': 'Pool-0/v_ff4859e2-8777-4959-8da6-2086553c6247', 'is_clone': False, 'name': 'v_ff4859e2-8777-4959-8da6-2086553c6247'}} from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:23:57 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f2715cc1-a4b1-4939-9297-fae00819ee03 tempest-VolumesCloneNegativeTest-166008807 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (2bbb506e-24c9-4e2b-b721-8c24d89e5f29) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'volume_name': 'volume-ff4859e2-8777-4959-8da6-2086553c6247', 'volume_size': 2}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:23:57 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f2715cc1-a4b1-4939-9297-fae00819ee03 tempest-VolumesCloneNegativeTest-166008807 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeOnFinishTask;volume:create, create.end' (85e33736-b522-4923-b87f-59496e96e159) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:23:57 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-f2715cc1-a4b1-4939-9297-fae00819ee03 tempest-VolumesCloneNegativeTest-166008807 None] Volume volume-ff4859e2-8777-4959-8da6-2086553c6247 (ff4859e2-8777-4959-8da6-2086553c6247): created successfully Apr 17 15:23:57 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f2715cc1-a4b1-4939-9297-fae00819ee03 tempest-VolumesCloneNegativeTest-166008807 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeOnFinishTask;volume:create, create.end' (85e33736-b522-4923-b87f-59496e96e159) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:23:57 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f2715cc1-a4b1-4939-9297-fae00819ee03 tempest-VolumesCloneNegativeTest-166008807 None] Flow 'volume_create_manager' (9b50d380-14dd-40be-8e53-57319414cafe) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:23:57 user cinder-volume[84696]: INFO cinder.volume.manager [None req-f2715cc1-a4b1-4939-9297-fae00819ee03 tempest-VolumesCloneNegativeTest-166008807 None] Created volume successfully. Apr 17 15:23:57 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-47c0924f-fe01-4cbe-b793-24ff6b90fa3a tempest-GroupSnapshotsTest-1926061586 None] Geting {'code': 201, 'error': None, 'data': {'full_name': 'Pool-0/v_f6603de5-71d3-468b-a673-ce1ff5cd1285', 'is_clone': False, 'name': 'v_f6603de5-71d3-468b-a673-ce1ff5cd1285'}} from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:23:57 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-47c0924f-fe01-4cbe-b793-24ff6b90fa3a tempest-GroupSnapshotsTest-1926061586 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (1f9cfa1b-1b1f-4e00-bc62-d3e4972829ee) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'volume_name': 'volume-f6603de5-71d3-468b-a673-ce1ff5cd1285', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:23:57 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-47c0924f-fe01-4cbe-b793-24ff6b90fa3a tempest-GroupSnapshotsTest-1926061586 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeOnFinishTask;volume:create, create.end' (b7d4ebaa-0ac3-408d-b929-f533cf848d14) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:23:57 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-47c0924f-fe01-4cbe-b793-24ff6b90fa3a tempest-GroupSnapshotsTest-1926061586 None] Volume volume-f6603de5-71d3-468b-a673-ce1ff5cd1285 (f6603de5-71d3-468b-a673-ce1ff5cd1285): created successfully Apr 17 15:23:57 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-47c0924f-fe01-4cbe-b793-24ff6b90fa3a tempest-GroupSnapshotsTest-1926061586 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeOnFinishTask;volume:create, create.end' (b7d4ebaa-0ac3-408d-b929-f533cf848d14) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:23:57 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-47c0924f-fe01-4cbe-b793-24ff6b90fa3a tempest-GroupSnapshotsTest-1926061586 None] Flow 'volume_create_manager' (a004c7ef-2ce5-44ca-bb24-27b528e12992) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:23:57 user cinder-volume[84696]: INFO cinder.volume.manager [None req-47c0924f-fe01-4cbe-b793-24ff6b90fa3a tempest-GroupSnapshotsTest-1926061586 None] Created volume successfully. Apr 17 15:23:58 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-e5f771c1-c99f-4537-96d6-3b271ab54578 tempest-VolumeTypesAccessTest-1000893842 None] Geting {'code': 201, 'error': None, 'data': {'full_name': 'Pool-0/v_53ff0d0b-94f7-4575-bb23-81cac29166fe', 'is_clone': False, 'name': 'v_53ff0d0b-94f7-4575-bb23-81cac29166fe'}} from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:23:58 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e5f771c1-c99f-4537-96d6-3b271ab54578 tempest-VolumeTypesAccessTest-1000893842 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (b128cde9-61ee-4e24-ae33-2caca58215d9) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'volume_name': 'volume-53ff0d0b-94f7-4575-bb23-81cac29166fe', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:23:58 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e5f771c1-c99f-4537-96d6-3b271ab54578 tempest-VolumeTypesAccessTest-1000893842 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeOnFinishTask;volume:create, create.end' (fd018449-c586-455d-919b-97dff35818d6) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:23:58 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-e5f771c1-c99f-4537-96d6-3b271ab54578 tempest-VolumeTypesAccessTest-1000893842 None] Volume volume-53ff0d0b-94f7-4575-bb23-81cac29166fe (53ff0d0b-94f7-4575-bb23-81cac29166fe): created successfully Apr 17 15:23:58 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e5f771c1-c99f-4537-96d6-3b271ab54578 tempest-VolumeTypesAccessTest-1000893842 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeOnFinishTask;volume:create, create.end' (fd018449-c586-455d-919b-97dff35818d6) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:23:58 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e5f771c1-c99f-4537-96d6-3b271ab54578 tempest-VolumeTypesAccessTest-1000893842 None] Flow 'volume_create_manager' (e3d028b1-0e4d-4929-b11a-3c5a8668bb3c) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:23:58 user cinder-volume[84696]: INFO cinder.volume.manager [None req-e5f771c1-c99f-4537-96d6-3b271ab54578 tempest-VolumeTypesAccessTest-1000893842 None] Created volume successfully. Apr 17 15:23:58 user cinder-volume[84696]: DEBUG cinder.coordination [None req-d9e3d6c8-17ef-48d6-8d2c-c3d4f35bd575 tempest-VolumesCloneNegativeTest-166008807 None] Acquiring lock "cinder-ff4859e2-8777-4959-8da6-2086553c6247-delete_volume" by "delete_volume" {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:192}} Apr 17 15:23:58 user cinder-volume[84696]: DEBUG cinder.coordination [None req-d9e3d6c8-17ef-48d6-8d2c-c3d4f35bd575 tempest-VolumesCloneNegativeTest-166008807 None] Lock "cinder-ff4859e2-8777-4959-8da6-2086553c6247-delete_volume" acquired by "delete_volume" :: waited 0.016s {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:196}} Apr 17 15:23:58 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-d9e3d6c8-17ef-48d6-8d2c-c3d4f35bd575 tempest-VolumesCloneNegativeTest-166008807 None] remove_export for volume: ff4859e2-8777-4959-8da6-2086553c6247. {{(pid=84696) remove_export /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:718}} Apr 17 15:23:58 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-d9e3d6c8-17ef-48d6-8d2c-c3d4f35bd575 tempest-VolumesCloneNegativeTest-166008807 None] remove export {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:912}} Apr 17 15:23:58 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-d9e3d6c8-17ef-48d6-8d2c-c3d4f35bd575 tempest-VolumesCloneNegativeTest-166008807 None] detach volume:Volume(_name_id=None,admin_metadata={},attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:23:55Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesCloneNegativeTest-Volume-1363201589',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=ff4859e2-8777-4959-8da6-2086553c6247,launched_at=2023-04-17T15:23:57Z,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='d95a37e41b1d433d92d16ae882a22070',provider_auth='CHAP nbSWzFYE 5E3U6SfxtRhrWL',provider_geometry=None,provider_id=None,provider_location='172.16.0.220:3260,1 iqn.2010-10.org.openstack:ff4859e2-8777-4959-8da6-2086553c6247 0',replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:23:56Z,service_uuid='e628352e-c4cf-40eb-8389-cce40663026a',shared_targets=True,size=2,snapshot_id=None,snapshots=,source_volid=None,status='deleting',terminated_at=2023-04-17T15:23:58Z,updated_at=2023-04-17T15:23:58Z,use_quota=True,user_id='b92392662be14505953e5b4a3903dbbb',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c) from target:iqn.2010-10.org.openstack:ff4859e2-8777-4959-8da6-2086553c6247. {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:913}} Apr 17 15:23:58 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-d9e3d6c8-17ef-48d6-8d2c-c3d4f35bd575 tempest-VolumesCloneNegativeTest-166008807 None] detach volume v_ff4859e2-8777-4959-8da6-2086553c6247 from target iqn.2010-10.org.openstack:ff4859e2-8777-4959-8da6-2086553c6247 {{(pid=84696) detach_target_vol /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:606}} Apr 17 15:23:58 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-d9e3d6c8-17ef-48d6-8d2c-c3d4f35bd575 tempest-VolumesCloneNegativeTest-166008807 None] Sending pool request DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:ff4859e2-8777-4959-8da6-2086553c6247/luns/v_ff4859e2-8777-4959-8da6-2086553c6247 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:23:58 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-d9e3d6c8-17ef-48d6-8d2c-c3d4f35bd575 tempest-VolumesCloneNegativeTest-166008807 None] Sending DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:ff4859e2-8777-4959-8da6-2086553c6247/luns/v_ff4859e2-8777-4959-8da6-2086553c6247 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:23:58 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-d9e3d6c8-17ef-48d6-8d2c-c3d4f35bd575 tempest-VolumesCloneNegativeTest-166008807 None] Geting {'code': 404, 'error': {'class': 'opene.exceptions.ItemNotFoundError', 'message': 'Target iqn.2010-10.org.openstack:ff4859e2-8777-4959-8da6-2086553c6247 not exists.', 'url': 'http://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:ff4859e2-8777-4959-8da6-2086553c6247/luns/v_ff4859e2-8777-4959-8da6-2086553c6247'}, 'data': None} from DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:ff4859e2-8777-4959-8da6-2086553c6247/luns/v_ff4859e2-8777-4959-8da6-2086553c6247 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:23:58 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-d9e3d6c8-17ef-48d6-8d2c-c3d4f35bd575 tempest-VolumesCloneNegativeTest-166008807 None] failed to remove resource iqn.2010-10.org.openstack:ff4859e2-8777-4959-8da6-2086553c6247 because of JDSS resource v_ff4859e2-8777-4959-8da6-2086553c6247 DNE. {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:920}} Apr 17 15:23:58 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-d9e3d6c8-17ef-48d6-8d2c-c3d4f35bd575 tempest-VolumesCloneNegativeTest-166008807 None] delete target: iqn.2010-10.org.openstack:ff4859e2-8777-4959-8da6-2086553c6247 {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:930}} Apr 17 15:23:58 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-d9e3d6c8-17ef-48d6-8d2c-c3d4f35bd575 tempest-VolumesCloneNegativeTest-166008807 None] delete target iqn.2010-10.org.openstack:ff4859e2-8777-4959-8da6-2086553c6247 {{(pid=84696) delete_target /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:442}} Apr 17 15:23:58 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-d9e3d6c8-17ef-48d6-8d2c-c3d4f35bd575 tempest-VolumesCloneNegativeTest-166008807 None] Sending pool request DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:ff4859e2-8777-4959-8da6-2086553c6247 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:23:58 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-d9e3d6c8-17ef-48d6-8d2c-c3d4f35bd575 tempest-VolumesCloneNegativeTest-166008807 None] Sending DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:ff4859e2-8777-4959-8da6-2086553c6247 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:23:58 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-d9e3d6c8-17ef-48d6-8d2c-c3d4f35bd575 tempest-VolumesCloneNegativeTest-166008807 None] Geting {'code': 404, 'error': {'class': 'opene.exceptions.ItemNotFoundError', 'message': 'Target iqn.2010-10.org.openstack:ff4859e2-8777-4959-8da6-2086553c6247 not exists.', 'url': 'http://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:ff4859e2-8777-4959-8da6-2086553c6247'}, 'data': None} from DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:ff4859e2-8777-4959-8da6-2086553c6247 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:23:58 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-d9e3d6c8-17ef-48d6-8d2c-c3d4f35bd575 tempest-VolumesCloneNegativeTest-166008807 None] failed to remove resource iqn.2010-10.org.openstack:ff4859e2-8777-4959-8da6-2086553c6247 because of JDSS resource iqn.2010-10.org.openstack:ff4859e2-8777-4959-8da6-2086553c6247 DNE. {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:935}} Apr 17 15:23:58 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-d9e3d6c8-17ef-48d6-8d2c-c3d4f35bd575 tempest-VolumesCloneNegativeTest-166008807 None] deleating volume v_ff4859e2-8777-4959-8da6-2086553c6247 {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 15:23:58 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-d9e3d6c8-17ef-48d6-8d2c-c3d4f35bd575 tempest-VolumesCloneNegativeTest-166008807 None] get snapshots for volume v_ff4859e2-8777-4959-8da6-2086553c6247 {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 15:23:58 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-d9e3d6c8-17ef-48d6-8d2c-c3d4f35bd575 tempest-VolumesCloneNegativeTest-166008807 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_ff4859e2-8777-4959-8da6-2086553c6247/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:23:58 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-d9e3d6c8-17ef-48d6-8d2c-c3d4f35bd575 tempest-VolumesCloneNegativeTest-166008807 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_ff4859e2-8777-4959-8da6-2086553c6247/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:23:58 user cinder-volume[84696]: INFO cinder.volume.manager [None req-a785f6df-db7e-4331-9a65-83e5d301a450 tempest-GroupSnapshotsTest-1926061586 None] Update group completed successfully. Apr 17 15:23:58 user cinder-volume[84696]: DEBUG cinder.coordination [None req-245cebd8-8396-4696-af5e-f63f63c4f353 tempest-VolumeTypesAccessTest-1000893842 None] Acquiring lock "cinder-53ff0d0b-94f7-4575-bb23-81cac29166fe-delete_volume" by "delete_volume" {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:192}} Apr 17 15:23:58 user cinder-volume[84696]: DEBUG cinder.coordination [None req-245cebd8-8396-4696-af5e-f63f63c4f353 tempest-VolumeTypesAccessTest-1000893842 None] Lock "cinder-53ff0d0b-94f7-4575-bb23-81cac29166fe-delete_volume" acquired by "delete_volume" :: waited 0.025s {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:196}} Apr 17 15:23:58 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=65,cinder:INSERT=3,cinder:UPDATE=30,cinder:DELETE=2 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:23:59 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-245cebd8-8396-4696-af5e-f63f63c4f353 tempest-VolumeTypesAccessTest-1000893842 None] remove_export for volume: 53ff0d0b-94f7-4575-bb23-81cac29166fe. {{(pid=84696) remove_export /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:718}} Apr 17 15:23:59 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-245cebd8-8396-4696-af5e-f63f63c4f353 tempest-VolumeTypesAccessTest-1000893842 None] remove export {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:912}} Apr 17 15:23:59 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-245cebd8-8396-4696-af5e-f63f63c4f353 tempest-VolumeTypesAccessTest-1000893842 None] detach volume:Volume(_name_id=None,admin_metadata={},attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:23:56Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumeTypesAccessTest-Volume-1976391366',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=53ff0d0b-94f7-4575-bb23-81cac29166fe,launched_at=2023-04-17T15:23:58Z,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='f3f0a0101c2840bc9570fc0e98c065cb',provider_auth='CHAP ylxWQnLi ujcaWEsLUoV6Yr',provider_geometry=None,provider_id=None,provider_location='172.16.0.220:3260,1 iqn.2010-10.org.openstack:53ff0d0b-94f7-4575-bb23-81cac29166fe 0',replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:23:57Z,service_uuid='e628352e-c4cf-40eb-8389-cce40663026a',shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='deleting',terminated_at=2023-04-17T15:23:59Z,updated_at=2023-04-17T15:23:59Z,use_quota=True,user_id='0cd3cc8229eb44748661d1ee5a78ed61',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(5d7108d1-10ea-4a52-bbef-e6ef8e5b8983),volume_type_id=5d7108d1-10ea-4a52-bbef-e6ef8e5b8983) from target:iqn.2010-10.org.openstack:53ff0d0b-94f7-4575-bb23-81cac29166fe. {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:913}} Apr 17 15:23:59 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-245cebd8-8396-4696-af5e-f63f63c4f353 tempest-VolumeTypesAccessTest-1000893842 None] detach volume v_53ff0d0b-94f7-4575-bb23-81cac29166fe from target iqn.2010-10.org.openstack:53ff0d0b-94f7-4575-bb23-81cac29166fe {{(pid=84696) detach_target_vol /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:606}} Apr 17 15:23:59 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-245cebd8-8396-4696-af5e-f63f63c4f353 tempest-VolumeTypesAccessTest-1000893842 None] Sending pool request DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:53ff0d0b-94f7-4575-bb23-81cac29166fe/luns/v_53ff0d0b-94f7-4575-bb23-81cac29166fe {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:23:59 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-245cebd8-8396-4696-af5e-f63f63c4f353 tempest-VolumeTypesAccessTest-1000893842 None] Sending DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:53ff0d0b-94f7-4575-bb23-81cac29166fe/luns/v_53ff0d0b-94f7-4575-bb23-81cac29166fe {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:23:59 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-245cebd8-8396-4696-af5e-f63f63c4f353 tempest-VolumeTypesAccessTest-1000893842 None] Geting {'code': 404, 'error': {'class': 'opene.exceptions.ItemNotFoundError', 'message': 'Target iqn.2010-10.org.openstack:53ff0d0b-94f7-4575-bb23-81cac29166fe not exists.', 'url': 'http://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:53ff0d0b-94f7-4575-bb23-81cac29166fe/luns/v_53ff0d0b-94f7-4575-bb23-81cac29166fe'}, 'data': None} from DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:53ff0d0b-94f7-4575-bb23-81cac29166fe/luns/v_53ff0d0b-94f7-4575-bb23-81cac29166fe {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:23:59 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-245cebd8-8396-4696-af5e-f63f63c4f353 tempest-VolumeTypesAccessTest-1000893842 None] failed to remove resource iqn.2010-10.org.openstack:53ff0d0b-94f7-4575-bb23-81cac29166fe because of JDSS resource v_53ff0d0b-94f7-4575-bb23-81cac29166fe DNE. {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:920}} Apr 17 15:23:59 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-245cebd8-8396-4696-af5e-f63f63c4f353 tempest-VolumeTypesAccessTest-1000893842 None] delete target: iqn.2010-10.org.openstack:53ff0d0b-94f7-4575-bb23-81cac29166fe {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:930}} Apr 17 15:23:59 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-245cebd8-8396-4696-af5e-f63f63c4f353 tempest-VolumeTypesAccessTest-1000893842 None] delete target iqn.2010-10.org.openstack:53ff0d0b-94f7-4575-bb23-81cac29166fe {{(pid=84696) delete_target /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:442}} Apr 17 15:23:59 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-245cebd8-8396-4696-af5e-f63f63c4f353 tempest-VolumeTypesAccessTest-1000893842 None] Sending pool request DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:53ff0d0b-94f7-4575-bb23-81cac29166fe {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:23:59 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-245cebd8-8396-4696-af5e-f63f63c4f353 tempest-VolumeTypesAccessTest-1000893842 None] Sending DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:53ff0d0b-94f7-4575-bb23-81cac29166fe {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:23:59 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-245cebd8-8396-4696-af5e-f63f63c4f353 tempest-VolumeTypesAccessTest-1000893842 None] Geting {'code': 404, 'error': {'class': 'opene.exceptions.ItemNotFoundError', 'message': 'Target iqn.2010-10.org.openstack:53ff0d0b-94f7-4575-bb23-81cac29166fe not exists.', 'url': 'http://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:53ff0d0b-94f7-4575-bb23-81cac29166fe'}, 'data': None} from DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:53ff0d0b-94f7-4575-bb23-81cac29166fe {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:23:59 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-245cebd8-8396-4696-af5e-f63f63c4f353 tempest-VolumeTypesAccessTest-1000893842 None] failed to remove resource iqn.2010-10.org.openstack:53ff0d0b-94f7-4575-bb23-81cac29166fe because of JDSS resource iqn.2010-10.org.openstack:53ff0d0b-94f7-4575-bb23-81cac29166fe DNE. {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:935}} Apr 17 15:23:59 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-245cebd8-8396-4696-af5e-f63f63c4f353 tempest-VolumeTypesAccessTest-1000893842 None] deleating volume v_53ff0d0b-94f7-4575-bb23-81cac29166fe {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 15:23:59 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-245cebd8-8396-4696-af5e-f63f63c4f353 tempest-VolumeTypesAccessTest-1000893842 None] get snapshots for volume v_53ff0d0b-94f7-4575-bb23-81cac29166fe {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 15:23:59 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-245cebd8-8396-4696-af5e-f63f63c4f353 tempest-VolumeTypesAccessTest-1000893842 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_53ff0d0b-94f7-4575-bb23-81cac29166fe/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:23:59 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-245cebd8-8396-4696-af5e-f63f63c4f353 tempest-VolumeTypesAccessTest-1000893842 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_53ff0d0b-94f7-4575-bb23-81cac29166fe/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:23:59 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-d9e3d6c8-17ef-48d6-8d2c-c3d4f35bd575 tempest-VolumesCloneNegativeTest-166008807 None] Geting {'code': 200, 'error': None, 'data': {'results': 0, 'entries': []}} from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_ff4859e2-8777-4959-8da6-2086553c6247/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:23:59 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-d9e3d6c8-17ef-48d6-8d2c-c3d4f35bd575 tempest-VolumesCloneNegativeTest-166008807 None] get volume v_ff4859e2-8777-4959-8da6-2086553c6247 info {{(pid=84696) get_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:239}} Apr 17 15:23:59 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-d9e3d6c8-17ef-48d6-8d2c-c3d4f35bd575 tempest-VolumesCloneNegativeTest-166008807 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_ff4859e2-8777-4959-8da6-2086553c6247 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:23:59 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-d9e3d6c8-17ef-48d6-8d2c-c3d4f35bd575 tempest-VolumesCloneNegativeTest-166008807 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_ff4859e2-8777-4959-8da6-2086553c6247 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:23:59 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-d9e3d6c8-17ef-48d6-8d2c-c3d4f35bd575 tempest-VolumesCloneNegativeTest-166008807 None] Geting {'code': 200, 'error': None, 'data': {'origin': None, 'relatime': None, 'acltype': None, 'vscan': None, 'full_name': 'Pool-0/v_ff4859e2-8777-4959-8da6-2086553c6247', 'userrefs': None, 'primarycache': 'all', 'logbias': 'latency', 'creation': '1681746430', 'sync': 'always', 'is_clone': False, 'dedup': 'off', 'sharenfs': None, 'receive_resume_token': None, 'volsize': '2147483648', 'referenced': '57344', 'sharesmb': None, 'createtxg': '187840', 'reservation': '0', 'scontext': None, 'mountpoint': None, 'casesensitivity': None, 'guid': '439540807923734808', 'usedbyrefreservation': '0', 'dnodesize': None, 'written': '57344', 'logicalused': '28672', 'compressratio': '1.00', 'rootcontext': 'none', 'default_scsi_id': '01f05b246e3e4fe7', 'type': 'volume', 'compression': 'lz4', 'snapdir': None, 'overlay': None, 'encryption': 'off', 'xattr': None, 'volmode': 'default', 'copies': '1', 'snapshot_limit': '18446744073709551615', 'aclinherit': None, 'defcontext': 'none', 'readonly': 'off', 'version': None, 'recordsize': None, 'filesystem_limit': None, 'mounted': None, 'mlslabel': 'none', 'secondarycache': 'all', 'refreservation': '0', 'available': '113926479872', 'san:volume_id': '01f05b246e3e4fe74b97506fc53a3a91a5ac2ff4', 'encryptionroot': None, 'exec': None, 'refquota': None, 'refcompressratio': '1.00', 'quota': None, 'utf8only': None, 'keylocation': 'none', 'snapdev': 'hidden', 'snapshot_count': '18446744073709551615', 'fscontext': 'none', 'clones': None, 'canmount': None, 'keystatus': None, 'atime': None, 'usedbysnapshots': '0', 'normalization': None, 'usedbychildren': '0', 'volblocksize': '131072', 'usedbydataset': '57344', 'objsetid': '121842', 'name': 'v_ff4859e2-8777-4959-8da6-2086553c6247', 'defer_destroy': None, 'pbkdf2iters': '0', 'checksum': 'on', 'redundant_metadata': 'all', 'filesystem_count': None, 'devices': None, 'keyformat': 'none', 'setuid': None, 'used': '57344', 'logicalreferenced': '28672', 'context': 'none', 'zoned': None, 'nbmand': None}} from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_ff4859e2-8777-4959-8da6-2086553c6247 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:23:59 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-d9e3d6c8-17ef-48d6-8d2c-c3d4f35bd575 tempest-VolumesCloneNegativeTest-166008807 None] delete volume:v_ff4859e2-8777-4959-8da6-2086553c6247 recursively children:{'force_umount': True} {{(pid=84696) delete_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:335}} Apr 17 15:23:59 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-d9e3d6c8-17ef-48d6-8d2c-c3d4f35bd575 tempest-VolumesCloneNegativeTest-166008807 None] Sending pool request DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_ff4859e2-8777-4959-8da6-2086553c6247 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:23:59 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-d9e3d6c8-17ef-48d6-8d2c-c3d4f35bd575 tempest-VolumesCloneNegativeTest-166008807 None] Sending DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_ff4859e2-8777-4959-8da6-2086553c6247 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:23:59 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-245cebd8-8396-4696-af5e-f63f63c4f353 tempest-VolumeTypesAccessTest-1000893842 None] Geting {'code': 200, 'error': None, 'data': {'results': 0, 'entries': []}} from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_53ff0d0b-94f7-4575-bb23-81cac29166fe/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:23:59 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-245cebd8-8396-4696-af5e-f63f63c4f353 tempest-VolumeTypesAccessTest-1000893842 None] get volume v_53ff0d0b-94f7-4575-bb23-81cac29166fe info {{(pid=84696) get_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:239}} Apr 17 15:23:59 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-245cebd8-8396-4696-af5e-f63f63c4f353 tempest-VolumeTypesAccessTest-1000893842 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_53ff0d0b-94f7-4575-bb23-81cac29166fe {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:23:59 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-245cebd8-8396-4696-af5e-f63f63c4f353 tempest-VolumeTypesAccessTest-1000893842 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_53ff0d0b-94f7-4575-bb23-81cac29166fe {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:23:59 user cinder-volume[84696]: INFO cinder.volume.manager [None req-7bf0e538-bcd7-4647-a6e1-5c8b8bf3d35a tempest-GroupSnapshotsTest-1926061586 None] Update group completed successfully. Apr 17 15:23:59 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-245cebd8-8396-4696-af5e-f63f63c4f353 tempest-VolumeTypesAccessTest-1000893842 None] Geting {'code': 200, 'error': None, 'data': {'origin': None, 'relatime': None, 'acltype': None, 'vscan': None, 'full_name': 'Pool-0/v_53ff0d0b-94f7-4575-bb23-81cac29166fe', 'userrefs': None, 'primarycache': 'all', 'logbias': 'latency', 'creation': '1681746430', 'sync': 'always', 'is_clone': False, 'dedup': 'off', 'sharenfs': None, 'receive_resume_token': None, 'volsize': '1073741824', 'referenced': '57344', 'sharesmb': None, 'createtxg': '187847', 'reservation': '0', 'scontext': None, 'mountpoint': None, 'casesensitivity': None, 'guid': '11927380066330330845', 'usedbyrefreservation': '0', 'dnodesize': None, 'written': '57344', 'logicalused': '28672', 'compressratio': '1.00', 'rootcontext': 'none', 'default_scsi_id': '83e63c0d57f0dc07', 'type': 'volume', 'compression': 'lz4', 'snapdir': None, 'overlay': None, 'encryption': 'off', 'xattr': None, 'volmode': 'default', 'copies': '1', 'snapshot_limit': '18446744073709551615', 'aclinherit': None, 'defcontext': 'none', 'readonly': 'off', 'version': None, 'recordsize': None, 'filesystem_limit': None, 'mounted': None, 'mlslabel': 'none', 'secondarycache': 'all', 'refreservation': '0', 'available': '113926479872', 'san:volume_id': '83e63c0d57f0dc0764c544046452961c69801be3', 'encryptionroot': None, 'exec': None, 'refquota': None, 'refcompressratio': '1.00', 'quota': None, 'utf8only': None, 'keylocation': 'none', 'snapdev': 'hidden', 'snapshot_count': '18446744073709551615', 'fscontext': 'none', 'clones': None, 'canmount': None, 'keystatus': None, 'atime': None, 'usedbysnapshots': '0', 'normalization': None, 'usedbychildren': '0', 'volblocksize': '131072', 'usedbydataset': '57344', 'objsetid': '123139', 'name': 'v_53ff0d0b-94f7-4575-bb23-81cac29166fe', 'defer_destroy': None, 'pbkdf2iters': '0', 'checksum': 'on', 'redundant_metadata': 'all', 'filesystem_count': None, 'devices': None, 'keyformat': 'none', 'setuid': None, 'used': '57344', 'logicalreferenced': '28672', 'context': 'none', 'zoned': None, 'nbmand': None}} from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_53ff0d0b-94f7-4575-bb23-81cac29166fe {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:23:59 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-245cebd8-8396-4696-af5e-f63f63c4f353 tempest-VolumeTypesAccessTest-1000893842 None] delete volume:v_53ff0d0b-94f7-4575-bb23-81cac29166fe recursively children:{'force_umount': True} {{(pid=84696) delete_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:335}} Apr 17 15:23:59 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-245cebd8-8396-4696-af5e-f63f63c4f353 tempest-VolumeTypesAccessTest-1000893842 None] Sending pool request DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_53ff0d0b-94f7-4575-bb23-81cac29166fe {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:23:59 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-245cebd8-8396-4696-af5e-f63f63c4f353 tempest-VolumeTypesAccessTest-1000893842 None] Sending DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_53ff0d0b-94f7-4575-bb23-81cac29166fe {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:24:00 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] Volume reschedule parameters: True retry: {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 15:24:00 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] Flow 'volume_create_manager' (8483a0a5-3732-4994-aae0-0c0dcfbdbc19) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:24:00 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (3e5d77b1-96b5-42c4-9076-cff2b54dc235) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:24:00 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (3e5d77b1-96b5-42c4-9076-cff2b54dc235) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:24:01Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TaggedBootDevicesTest_v242-volume-1367581919',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=c74b7878-4936-4015-8b63-142923947310,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='8f0c848b490b47629051e961c682c5a5',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:24:01Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:24:01Z,use_quota=True,user_id='3e4b069991954c57993051fbe8665d35',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:24:00 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (b3cac014-64c2-4fce-8050-bf78cce22498) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:24:00 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (b3cac014-64c2-4fce-8050-bf78cce22498) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:24:00 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (823d4566-16aa-490c-a3d9-b994132eddb8) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:24:01 user cinder-volume[84696]: INFO cinder.volume.manager [None req-c939015d-8164-40d6-af6c-458d57b23607 tempest-GroupSnapshotsTest-1926061586 None] Update group completed successfully. Apr 17 15:24:01 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (823d4566-16aa-490c-a3d9-b994132eddb8) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'image', 'volume_id': 'c74b7878-4936-4015-8b63-142923947310', 'volume_name': 'volume-c74b7878-4936-4015-8b63-142923947310', 'volume_size': 1, 'image_id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'image_location': (None, None), 'image_meta': {'name': 'cirros-0.5.2-x86_64-disk', 'disk_format': 'qcow2', 'container_format': 'bare', 'visibility': 'public', 'size': 16300544, 'virtual_size': 117440512, 'status': 'active', 'checksum': 'b874c39491a2377b8490f5f1e89761a4', 'protected': False, 'min_ram': 0, 'min_disk': 0, 'owner': 'a43c211b4ed542538799349489b1d884', 'os_hidden': False, 'os_hash_algo': 'sha512', 'os_hash_value': '6b813aa46bb90b4da216a4d19376593fa3f4fc7e617f03a92b7fe11e9a3981cbe8f0959dbebe36225e5f53dc4492341a4863cac4ed1ee0909f3fc78ef9c3e869', 'id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'created_at': datetime.datetime(2023, 4, 17, 15, 19, 53, tzinfo=datetime.timezone.utc), 'updated_at': datetime.datetime(2023, 4, 17, 15, 19, 55, tzinfo=datetime.timezone.utc), 'tags': [], 'file': '/v2/images/b3f0dec1-7758-44a8-82e0-76ada8837428/file', 'properties': {'hw_rng_model': 'virtio', 'owner_specified.openstack.md5': '', 'owner_specified.openstack.object': 'images/cirros-0.5.2-x86_64-disk', 'owner_specified.openstack.sha256': ''}}, 'image_service': }' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:24:01 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (1150abf6-9909-4d62-a3b0-d40dd3755d2e) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:24:01 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (1150abf6-9909-4d62-a3b0-d40dd3755d2e) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:24:01 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (a0bec28b-ba7f-4d26-8d24-66f65b71c718) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:24:01 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] Volume c74b7878-4936-4015-8b63-142923947310: being created as image with specification: {'status': 'creating', 'volume_name': 'volume-c74b7878-4936-4015-8b63-142923947310', 'volume_size': 1, 'image_id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'image_location': (None, None), 'image_meta': {'name': 'cirros-0.5.2-x86_64-disk', 'disk_format': 'qcow2', 'container_format': 'bare', 'visibility': 'public', 'size': 16300544, 'virtual_size': 117440512, 'status': 'active', 'checksum': 'b874c39491a2377b8490f5f1e89761a4', 'protected': False, 'min_ram': 0, 'min_disk': 0, 'owner': 'a43c211b4ed542538799349489b1d884', 'os_hidden': False, 'os_hash_algo': 'sha512', 'os_hash_value': '6b813aa46bb90b4da216a4d19376593fa3f4fc7e617f03a92b7fe11e9a3981cbe8f0959dbebe36225e5f53dc4492341a4863cac4ed1ee0909f3fc78ef9c3e869', 'id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'created_at': datetime.datetime(2023, 4, 17, 15, 19, 53, tzinfo=datetime.timezone.utc), 'updated_at': datetime.datetime(2023, 4, 17, 15, 19, 55, tzinfo=datetime.timezone.utc), 'tags': [], 'file': '/v2/images/b3f0dec1-7758-44a8-82e0-76ada8837428/file', 'properties': {'hw_rng_model': 'virtio', 'owner_specified.openstack.md5': '', 'owner_specified.openstack.object': 'images/cirros-0.5.2-x86_64-disk', 'owner_specified.openstack.sha256': ''}}, 'image_service': } Apr 17 15:24:01 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] Cloning c74b7878-4936-4015-8b63-142923947310 from image b3f0dec1-7758-44a8-82e0-76ada8837428 at location (None, None). {{(pid=84696) _create_from_image /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1071}} Apr 17 15:24:01 user cinder-volume[84696]: DEBUG cinder.coordination [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] Acquiring lock "cinder-b3f0dec1-7758-44a8-82e0-76ada8837428" by "_prepare_image_cache_entry" {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:192}} Apr 17 15:24:01 user cinder-volume[84696]: DEBUG cinder.coordination [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] Lock "cinder-b3f0dec1-7758-44a8-82e0-76ada8837428" acquired by "_prepare_image_cache_entry" :: waited 0.032s {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:196}} Apr 17 15:24:01 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:INSERT=6,cinder:SELECT=62,cinder:UPDATE=29,cinder:DELETE=3 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:24:01 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] Preparing cache entry for image = b3f0dec1-7758-44a8-82e0-76ada8837428 on host user@joviandss-2#joviandss-2. {{(pid=84696) _prepare_image_cache_entry /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:891}} Apr 17 15:24:01 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] Attempting to retrieve cache entry for image = b3f0dec1-7758-44a8-82e0-76ada8837428 on host user@joviandss-2#joviandss-2. {{(pid=84696) _create_from_image_cache /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:827}} Apr 17 15:24:02 user cinder-volume[84696]: DEBUG cinder.image.image_utils [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] Image fetch details: dest /opt/stack/data/cinder/conversion/image_fetch_b3f0dec1-7758-44a8-82e0-76ada8837428_zt3n6g1xuser@joviandss-2, size 15.55 MB, duration 1.00 sec {{(pid=84696) fetch /opt/stack/cinder/cinder/image/image_utils.py:628}} Apr 17 15:24:02 user cinder-volume[84696]: INFO cinder.image.image_utils [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] Image download 15.55 MB at 15.55 MB/s Apr 17 15:24:02 user cinder-volume[84696]: DEBUG oslo_concurrency.processutils [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] Running cmd (subprocess): /usr/bin/python3.10 -m oslo_concurrency.prlimit --as=1073741824 --cpu=60 -- sudo cinder-rootwrap /etc/cinder/rootwrap.conf env LC_ALL=C qemu-img info --output=json /opt/stack/data/cinder/conversion/image_fetch_b3f0dec1-7758-44a8-82e0-76ada8837428_zt3n6g1xuser@joviandss-2 {{(pid=84696) execute /usr/local/lib/python3.10/dist-packages/oslo_concurrency/processutils.py:384}} Apr 17 15:24:02 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-d9e3d6c8-17ef-48d6-8d2c-c3d4f35bd575 tempest-VolumesCloneNegativeTest-166008807 None] Geting {'code': 204} from DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_ff4859e2-8777-4959-8da6-2086553c6247 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:24:02 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-d9e3d6c8-17ef-48d6-8d2c-c3d4f35bd575 tempest-VolumesCloneNegativeTest-166008807 None] volume v_ff4859e2-8777-4959-8da6-2086553c6247 deleted {{(pid=84696) delete_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:346}} Apr 17 15:24:02 user cinder-volume[84696]: DEBUG cinder.quota [None req-d9e3d6c8-17ef-48d6-8d2c-c3d4f35bd575 tempest-VolumesCloneNegativeTest-166008807 None] Created reservations ['ad7110a9-6916-4fc4-aed6-6fd2fde0fc57', 'da001849-e7b5-482f-9c32-6bd17608c6e4', 'd11d309b-e5f9-4b04-8c26-8c21e12c7087', '857d3652-cae6-4043-94b7-74cf2bff2752'] {{(pid=84696) reserve /opt/stack/cinder/cinder/quota.py:755}} Apr 17 15:24:02 user sudo[91544]: stack : PWD=/ ; USER=root ; COMMAND=/usr/local/bin/cinder-rootwrap /etc/cinder/rootwrap.conf env LC_ALL=C qemu-img info --output=json /opt/stack/data/cinder/conversion/image_fetch_b3f0dec1-7758-44a8-82e0-76ada8837428_zt3n6g1xuser@joviandss-2 Apr 17 15:24:02 user sudo[91544]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=1001) Apr 17 15:24:02 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-d9e3d6c8-17ef-48d6-8d2c-c3d4f35bd575 tempest-VolumesCloneNegativeTest-166008807 None] Updating volume stats {{(pid=84696) _update_volume_stats /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:724}} Apr 17 15:24:02 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-d9e3d6c8-17ef-48d6-8d2c-c3d4f35bd575 tempest-VolumesCloneNegativeTest-166008807 None] Get pool Pool-0 fsprops {{(pid=84696) get_pool_stats /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:894}} Apr 17 15:24:02 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-d9e3d6c8-17ef-48d6-8d2c-c3d4f35bd575 tempest-VolumesCloneNegativeTest-166008807 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:24:02 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-d9e3d6c8-17ef-48d6-8d2c-c3d4f35bd575 tempest-VolumesCloneNegativeTest-166008807 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:24:02 user cinder-volume[84696]: INFO cinder.volume.manager [None req-b8efccb4-4b0c-4d68-af8f-494b46fa40f4 tempest-GroupSnapshotsTest-1926061586 None] Update group completed successfully. Apr 17 15:24:03 user sudo[91544]: pam_unix(sudo:session): session closed for user root Apr 17 15:24:03 user cinder-volume[84696]: DEBUG oslo_concurrency.processutils [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] CMD "/usr/bin/python3.10 -m oslo_concurrency.prlimit --as=1073741824 --cpu=60 -- sudo cinder-rootwrap /etc/cinder/rootwrap.conf env LC_ALL=C qemu-img info --output=json /opt/stack/data/cinder/conversion/image_fetch_b3f0dec1-7758-44a8-82e0-76ada8837428_zt3n6g1xuser@joviandss-2" returned: 0 in 1.204s {{(pid=84696) execute /usr/local/lib/python3.10/dist-packages/oslo_concurrency/processutils.py:422}} Apr 17 15:24:03 user cinder-volume[84696]: DEBUG cinder.image.image_utils [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] Temporary image b3f0dec1-7758-44a8-82e0-76ada8837428 is fetched for user 3e4b069991954c57993051fbe8665d35. {{(pid=84696) fetch /opt/stack/cinder/cinder/image/image_utils.py:1279}} Apr 17 15:24:03 user cinder-volume[84696]: DEBUG oslo_concurrency.processutils [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] Running cmd (subprocess): /usr/bin/python3.10 -m oslo_concurrency.prlimit --as=1073741824 --cpu=60 -- sudo cinder-rootwrap /etc/cinder/rootwrap.conf env LC_ALL=C qemu-img info --output=json /opt/stack/data/cinder/conversion/image_fetch_b3f0dec1-7758-44a8-82e0-76ada8837428_zt3n6g1xuser@joviandss-2 {{(pid=84696) execute /usr/local/lib/python3.10/dist-packages/oslo_concurrency/processutils.py:384}} Apr 17 15:24:03 user sudo[91606]: stack : PWD=/ ; USER=root ; COMMAND=/usr/local/bin/cinder-rootwrap /etc/cinder/rootwrap.conf env LC_ALL=C qemu-img info --output=json /opt/stack/data/cinder/conversion/image_fetch_b3f0dec1-7758-44a8-82e0-76ada8837428_zt3n6g1xuser@joviandss-2 Apr 17 15:24:03 user sudo[91606]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=1001) Apr 17 15:24:03 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-d9e3d6c8-17ef-48d6-8d2c-c3d4f35bd575 tempest-VolumesCloneNegativeTest-166008807 None] Geting {'code': 200, 'error': None, 'data': {'available': '113926643712', 'status': 30, 'import_status': {'import_steps': {'san_setup': True, 'vip_setup': True, 'zfs_import': True, 'mount_system_volume': True, 'nas_setup': True}, 'import_successful': True}, 'scan': None, 'encryption': {'enabled': False}, 'deduplication_rate': '1.00', 'sysvol_upgrade': 'not_needed', 'vdevs': [{'name': 'wwn-0x6000c296b793bf526a107a9d823e947a', 'iostats': {'read': '0', 'write': '0', 'chksum': '0'}, 'disks': [{'origin': 'local', 'slot': '', 'led': 'off', 'name': 'sdn', 'iostats': {'read': '0', 'write': '0', 'chksum': '0'}, 'alias': '', 'health': 'ONLINE', 'sn': '6000c296b793bf526a107a9d823e947a', 'trim_data': {'status': 'trim_unsupported', 'progress': None, 'trim_time': None}, 'path': 'pci-0000:03:00.0-scsi-0:0:14:0', 'model': 'Virtual disk', 'id': 'wwn-0x6000c296b793bf526a107a9d823e947a', 'size': 128849018880}], 'health': 'ONLINE', 'vdev_replacings': [], 'vdev_spares': [], 'type': ''}], 'id': '14181317334440137330', 'health': 'ONLINE', 'iostats': {'read': '0', 'write': '0', 'chksum': '0'}, 'operation': 'none', 'size': '127775277056', 'autotrim': False, 'name': 'Pool-0'}} from GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:24:03 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-d9e3d6c8-17ef-48d6-8d2c-c3d4f35bd575 tempest-VolumesCloneNegativeTest-166008807 None] Total capacity: 119, Free 106. {{(pid=84696) _update_volume_stats /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:757}} Apr 17 15:24:03 user cinder-volume[84696]: DEBUG cinder.manager [None req-d9e3d6c8-17ef-48d6-8d2c-c3d4f35bd575 tempest-VolumesCloneNegativeTest-166008807 None] Notifying Schedulers of capabilities ... {{(pid=84696) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 17 15:24:03 user cinder-volume[84696]: INFO cinder.volume.manager [None req-d9e3d6c8-17ef-48d6-8d2c-c3d4f35bd575 tempest-VolumesCloneNegativeTest-166008807 None] Deleted volume successfully. Apr 17 15:24:03 user cinder-volume[84696]: DEBUG cinder.coordination [None req-d9e3d6c8-17ef-48d6-8d2c-c3d4f35bd575 tempest-VolumesCloneNegativeTest-166008807 None] Lock "cinder-ff4859e2-8777-4959-8da6-2086553c6247-delete_volume" released by "delete_volume" :: held 5.576s {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:207}} Apr 17 15:24:03 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-245cebd8-8396-4696-af5e-f63f63c4f353 tempest-VolumeTypesAccessTest-1000893842 None] Geting {'code': 204} from DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_53ff0d0b-94f7-4575-bb23-81cac29166fe {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:24:03 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-245cebd8-8396-4696-af5e-f63f63c4f353 tempest-VolumeTypesAccessTest-1000893842 None] volume v_53ff0d0b-94f7-4575-bb23-81cac29166fe deleted {{(pid=84696) delete_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:346}} Apr 17 15:24:03 user cinder-volume[84696]: WARNING cinder.quota [None req-245cebd8-8396-4696-af5e-f63f63c4f353 tempest-VolumeTypesAccessTest-1000893842 None] Deprecated: Default quota for resource: volumes_tempest-VolumeTypesAccessTest-volume-type-1377280117 is set by the default quota flag: quota_volumes_tempest-VolumeTypesAccessTest-volume-type-1377280117, it is now deprecated. Please use the default quota class for default quota. Apr 17 15:24:03 user cinder-volume[84696]: WARNING cinder.quota [None req-245cebd8-8396-4696-af5e-f63f63c4f353 tempest-VolumeTypesAccessTest-1000893842 None] Deprecated: Default quota for resource: gigabytes_tempest-VolumeTypesAccessTest-volume-type-1377280117 is set by the default quota flag: quota_gigabytes_tempest-VolumeTypesAccessTest-volume-type-1377280117, it is now deprecated. Please use the default quota class for default quota. Apr 17 15:24:03 user cinder-volume[84696]: DEBUG cinder.quota [None req-245cebd8-8396-4696-af5e-f63f63c4f353 tempest-VolumeTypesAccessTest-1000893842 None] Created reservations ['2ff15367-ebbe-4407-a3fe-36ef83b22de8', '3a80a24e-b360-49f6-908f-98814b5aba21', '65777458-1b1e-4063-a77f-ffc0cd1c6d5b', 'a740b262-1d74-42d8-b27d-0fbb814c3460'] {{(pid=84696) reserve /opt/stack/cinder/cinder/quota.py:755}} Apr 17 15:24:03 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-245cebd8-8396-4696-af5e-f63f63c4f353 tempest-VolumeTypesAccessTest-1000893842 None] Updating volume stats {{(pid=84696) _update_volume_stats /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:724}} Apr 17 15:24:03 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-245cebd8-8396-4696-af5e-f63f63c4f353 tempest-VolumeTypesAccessTest-1000893842 None] Get pool Pool-0 fsprops {{(pid=84696) get_pool_stats /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:894}} Apr 17 15:24:03 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-245cebd8-8396-4696-af5e-f63f63c4f353 tempest-VolumeTypesAccessTest-1000893842 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:24:03 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-245cebd8-8396-4696-af5e-f63f63c4f353 tempest-VolumeTypesAccessTest-1000893842 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:24:04 user cinder-volume[84696]: INFO cinder.volume.manager [None req-11377079-0af3-44c0-b136-161935214a6b tempest-GroupSnapshotsTest-1926061586 None] GroupSnapshot aa6864c4-c754-46e2-a4a4-18248ad70055: creating. Apr 17 15:24:04 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-11377079-0af3-44c0-b136-161935214a6b tempest-GroupSnapshotsTest-1926061586 None] Group snapshot aa6864c4-c754-46e2-a4a4-18248ad70055: creating. {{(pid=84696) create_group_snapshot /opt/stack/cinder/cinder/volume/manager.py:4129}} Apr 17 15:24:04 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-11377079-0af3-44c0-b136-161935214a6b tempest-GroupSnapshotsTest-1926061586 None] create snapshot 8a4c296c-977d-4a9d-a08a-e3b2a1b9a8d4 for volume 7249f8e4-ed4b-4689-aba3-2766c924f42c {{(pid=84696) create_snapshot /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:625}} Apr 17 15:24:04 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-11377079-0af3-44c0-b136-161935214a6b tempest-GroupSnapshotsTest-1926061586 None] cloning v_7249f8e4-ed4b-4689-aba3-2766c924f42c to s_8a4c296c-977d-4a9d-a08a-e3b2a1b9a8d4 {{(pid=84696) _clone_object /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:456}} Apr 17 15:24:04 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-11377079-0af3-44c0-b136-161935214a6b tempest-GroupSnapshotsTest-1926061586 None] create snapshot s_8a4c296c-977d-4a9d-a08a-e3b2a1b9a8d4 {{(pid=84696) create_snapshot /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:635}} Apr 17 15:24:04 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-11377079-0af3-44c0-b136-161935214a6b tempest-GroupSnapshotsTest-1926061586 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_7249f8e4-ed4b-4689-aba3-2766c924f42c/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:24:04 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-11377079-0af3-44c0-b136-161935214a6b tempest-GroupSnapshotsTest-1926061586 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_7249f8e4-ed4b-4689-aba3-2766c924f42c/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:24:04 user sudo[91606]: pam_unix(sudo:session): session closed for user root Apr 17 15:24:04 user cinder-volume[84696]: DEBUG oslo_concurrency.processutils [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] CMD "/usr/bin/python3.10 -m oslo_concurrency.prlimit --as=1073741824 --cpu=60 -- sudo cinder-rootwrap /etc/cinder/rootwrap.conf env LC_ALL=C qemu-img info --output=json /opt/stack/data/cinder/conversion/image_fetch_b3f0dec1-7758-44a8-82e0-76ada8837428_zt3n6g1xuser@joviandss-2" returned: 0 in 1.092s {{(pid=84696) execute /usr/local/lib/python3.10/dist-packages/oslo_concurrency/processutils.py:422}} Apr 17 15:24:04 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] creating volume v_c74b7878-4936-4015-8b63-142923947310. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 15:24:04 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] create volume {'name': 'v_c74b7878-4936-4015-8b63-142923947310', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 15:24:04 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:24:04 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:24:04 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-245cebd8-8396-4696-af5e-f63f63c4f353 tempest-VolumeTypesAccessTest-1000893842 None] Geting {'code': 200, 'error': None, 'data': {'available': '113926643712', 'status': 30, 'import_status': {'import_steps': {'san_setup': True, 'vip_setup': True, 'zfs_import': True, 'mount_system_volume': True, 'nas_setup': True}, 'import_successful': True}, 'scan': None, 'encryption': {'enabled': False}, 'deduplication_rate': '1.00', 'sysvol_upgrade': 'not_needed', 'vdevs': [{'name': 'wwn-0x6000c296b793bf526a107a9d823e947a', 'iostats': {'read': '0', 'write': '0', 'chksum': '0'}, 'disks': [{'origin': 'local', 'slot': '', 'led': 'off', 'name': 'sdn', 'iostats': {'read': '0', 'write': '0', 'chksum': '0'}, 'alias': '', 'health': 'ONLINE', 'sn': '6000c296b793bf526a107a9d823e947a', 'trim_data': {'status': 'trim_unsupported', 'progress': None, 'trim_time': None}, 'path': 'pci-0000:03:00.0-scsi-0:0:14:0', 'model': 'Virtual disk', 'id': 'wwn-0x6000c296b793bf526a107a9d823e947a', 'size': 128849018880}], 'health': 'ONLINE', 'vdev_replacings': [], 'vdev_spares': [], 'type': ''}], 'id': '14181317334440137330', 'health': 'ONLINE', 'iostats': {'read': '0', 'write': '0', 'chksum': '0'}, 'operation': 'none', 'size': '127775277056', 'autotrim': False, 'name': 'Pool-0'}} from GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:24:04 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-245cebd8-8396-4696-af5e-f63f63c4f353 tempest-VolumeTypesAccessTest-1000893842 None] Total capacity: 119, Free 106. {{(pid=84696) _update_volume_stats /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:757}} Apr 17 15:24:04 user cinder-volume[84696]: DEBUG cinder.manager [None req-245cebd8-8396-4696-af5e-f63f63c4f353 tempest-VolumeTypesAccessTest-1000893842 None] Notifying Schedulers of capabilities ... {{(pid=84696) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 17 15:24:04 user cinder-volume[84696]: INFO cinder.volume.manager [None req-245cebd8-8396-4696-af5e-f63f63c4f353 tempest-VolumeTypesAccessTest-1000893842 None] Deleted volume successfully. Apr 17 15:24:04 user cinder-volume[84696]: DEBUG cinder.coordination [None req-245cebd8-8396-4696-af5e-f63f63c4f353 tempest-VolumeTypesAccessTest-1000893842 None] Lock "cinder-53ff0d0b-94f7-4575-bb23-81cac29166fe-delete_volume" released by "delete_volume" :: held 5.994s {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:207}} Apr 17 15:24:06 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-11377079-0af3-44c0-b136-161935214a6b tempest-GroupSnapshotsTest-1926061586 None] Geting {'code': 200, 'error': None, 'data': {'name': 's_8a4c296c-977d-4a9d-a08a-e3b2a1b9a8d4'}} from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_7249f8e4-ed4b-4689-aba3-2766c924f42c/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:24:06 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-11377079-0af3-44c0-b136-161935214a6b tempest-GroupSnapshotsTest-1926061586 None] create volume s_8a4c296c-977d-4a9d-a08a-e3b2a1b9a8d4 from snapshot s_8a4c296c-977d-4a9d-a08a-e3b2a1b9a8d4 {{(pid=84696) create_volume_from_snapshot /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:674}} Apr 17 15:24:06 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-11377079-0af3-44c0-b136-161935214a6b tempest-GroupSnapshotsTest-1926061586 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_7249f8e4-ed4b-4689-aba3-2766c924f42c/clone {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:24:06 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-11377079-0af3-44c0-b136-161935214a6b tempest-GroupSnapshotsTest-1926061586 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_7249f8e4-ed4b-4689-aba3-2766c924f42c/clone {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:24:07 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] Geting {'code': 201, 'error': None, 'data': {'full_name': 'Pool-0/v_c74b7878-4936-4015-8b63-142923947310', 'is_clone': False, 'name': 'v_c74b7878-4936-4015-8b63-142923947310'}} from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:24:07 user cinder-volume[84696]: DEBUG cinder.volume.volume_utils [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] Attempting download of b3f0dec1-7758-44a8-82e0-76ada8837428 ((None, None)) to volume c74b7878-4936-4015-8b63-142923947310. {{(pid=84696) copy_image_to_volume /opt/stack/cinder/cinder/volume/volume_utils.py:1190}} Apr 17 15:24:07 user cinder-volume[84696]: DEBUG cinder.volume.driver [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] copy_image_to_volume volume-c74b7878-4936-4015-8b63-142923947310. {{(pid=84696) _copy_image_data_to_volume /opt/stack/cinder/cinder/volume/driver.py:890}} Apr 17 15:24:07 user cinder-volume[84696]: DEBUG os_brick.utils [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] ==> get_connector_properties: call "{'root_helper': 'sudo cinder-rootwrap /etc/cinder/rootwrap.conf', 'my_ip': '10.0.0.210', 'multipath': False, 'enforce_multipath': False, 'host': None, 'execute': None}" {{(pid=84696) trace_logging_wrapper /usr/local/lib/python3.10/dist-packages/os_brick/utils.py:175}} Apr 17 15:24:07 user cinder-volume[84696]: INFO oslo.privsep.daemon [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] Running privsep helper: ['sudo', 'cinder-rootwrap', '/etc/cinder/rootwrap.conf', 'privsep-helper', '--config-file', '/etc/cinder/cinder.conf', '--privsep_context', 'os_brick.privileged.default', '--privsep_sock_path', '/tmp/tmp5x2nn_wc/privsep.sock'] Apr 17 15:24:07 user sudo[91735]: stack : PWD=/ ; USER=root ; COMMAND=/usr/local/bin/cinder-rootwrap /etc/cinder/rootwrap.conf privsep-helper --config-file /etc/cinder/cinder.conf --privsep_context os_brick.privileged.default --privsep_sock_path /tmp/tmp5x2nn_wc/privsep.sock Apr 17 15:24:07 user sudo[91735]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=1001) Apr 17 15:24:07 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-01e53132-32f6-458d-8c41-044e160c456f tempest-TestVolumeSwap-1714109745 None] Volume reschedule parameters: True retry: {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 15:24:07 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-01e53132-32f6-458d-8c41-044e160c456f tempest-TestVolumeSwap-1714109745 None] Flow 'volume_create_manager' (b15a69e0-a98b-4be0-955b-067db7c8d0db) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:24:07 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-01e53132-32f6-458d-8c41-044e160c456f tempest-TestVolumeSwap-1714109745 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (affa8c7f-e6f5-4667-b87f-659c503ea115) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:24:07 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-01e53132-32f6-458d-8c41-044e160c456f tempest-TestVolumeSwap-1714109745 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (affa8c7f-e6f5-4667-b87f-659c503ea115) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:24:08Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TestVolumeSwap-volume-1757059761',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=66000c93-eb12-4c23-8a69-71e1aa6b521f,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='00b574796d1942938ed2f348803fcf87',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:24:08Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:24:08Z,use_quota=True,user_id='4c2bfcb4119e483ab1a41e3c598ee1b0',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:24:07 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-01e53132-32f6-458d-8c41-044e160c456f tempest-TestVolumeSwap-1714109745 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (236b7b42-deba-4a00-8733-30b72d4dd005) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:24:07 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-01e53132-32f6-458d-8c41-044e160c456f tempest-TestVolumeSwap-1714109745 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (236b7b42-deba-4a00-8733-30b72d4dd005) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:24:07 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-01e53132-32f6-458d-8c41-044e160c456f tempest-TestVolumeSwap-1714109745 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (88fcc94b-d546-49b2-8aeb-5026fc1b6547) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:24:07 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-01e53132-32f6-458d-8c41-044e160c456f tempest-TestVolumeSwap-1714109745 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (88fcc94b-d546-49b2-8aeb-5026fc1b6547) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': '66000c93-eb12-4c23-8a69-71e1aa6b521f', 'volume_name': 'volume-66000c93-eb12-4c23-8a69-71e1aa6b521f', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:24:07 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-01e53132-32f6-458d-8c41-044e160c456f tempest-TestVolumeSwap-1714109745 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (d4da2b43-306c-469e-ac40-6fc450369d8e) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:24:07 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-01e53132-32f6-458d-8c41-044e160c456f tempest-TestVolumeSwap-1714109745 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (d4da2b43-306c-469e-ac40-6fc450369d8e) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:24:07 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-01e53132-32f6-458d-8c41-044e160c456f tempest-TestVolumeSwap-1714109745 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (ecef328b-7b17-4e85-84c1-6c5a22e0f900) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:24:07 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-01e53132-32f6-458d-8c41-044e160c456f tempest-TestVolumeSwap-1714109745 None] Volume 66000c93-eb12-4c23-8a69-71e1aa6b521f: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-66000c93-eb12-4c23-8a69-71e1aa6b521f', 'volume_size': 1} Apr 17 15:24:07 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-01e53132-32f6-458d-8c41-044e160c456f tempest-TestVolumeSwap-1714109745 None] creating volume v_66000c93-eb12-4c23-8a69-71e1aa6b521f. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 15:24:07 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-01e53132-32f6-458d-8c41-044e160c456f tempest-TestVolumeSwap-1714109745 None] create volume {'name': 'v_66000c93-eb12-4c23-8a69-71e1aa6b521f', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 15:24:07 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-01e53132-32f6-458d-8c41-044e160c456f tempest-TestVolumeSwap-1714109745 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:24:07 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-01e53132-32f6-458d-8c41-044e160c456f tempest-TestVolumeSwap-1714109745 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:24:08 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-4c11ecf3-dc04-4262-bc9d-a7dd71483cc5 tempest-VolumesGetTest-1961901732 None] Volume reschedule parameters: True retry: {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 15:24:08 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-4c11ecf3-dc04-4262-bc9d-a7dd71483cc5 tempest-VolumesGetTest-1961901732 None] Flow 'volume_create_manager' (33577b12-4a89-4ad3-bf26-5089bd045c19) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:24:08 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-4c11ecf3-dc04-4262-bc9d-a7dd71483cc5 tempest-VolumesGetTest-1961901732 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (98132b0a-aa63-482a-8ecc-e4e53f76205f) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:24:08 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-4c11ecf3-dc04-4262-bc9d-a7dd71483cc5 tempest-VolumesGetTest-1961901732 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (98132b0a-aa63-482a-8ecc-e4e53f76205f) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:24:08Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesGetTest-Volume-1860415110',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=f088992b-786a-4641-8bcc-5cb7c819d313,launched_at=None,metadata={Type='Test'},migration_status=None,multiattach=False,previous_status=None,project_id='a65d14b0a2894a6f90506d86feaefd15',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:24:08Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:24:08Z,use_quota=True,user_id='6ffa3051570a4721a48a0171b3cc4d16',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:24:08 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-4c11ecf3-dc04-4262-bc9d-a7dd71483cc5 tempest-VolumesGetTest-1961901732 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (c2562c5c-fdbd-40ff-a81c-79c1d718b519) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:24:08 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-4c11ecf3-dc04-4262-bc9d-a7dd71483cc5 tempest-VolumesGetTest-1961901732 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (c2562c5c-fdbd-40ff-a81c-79c1d718b519) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:24:08 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-4c11ecf3-dc04-4262-bc9d-a7dd71483cc5 tempest-VolumesGetTest-1961901732 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (3cd17d7e-af58-4c0c-a591-cae0aeec51d2) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:24:08 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-4c11ecf3-dc04-4262-bc9d-a7dd71483cc5 tempest-VolumesGetTest-1961901732 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (3cd17d7e-af58-4c0c-a591-cae0aeec51d2) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': 'f088992b-786a-4641-8bcc-5cb7c819d313', 'volume_name': 'volume-f088992b-786a-4641-8bcc-5cb7c819d313', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:24:08 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-4c11ecf3-dc04-4262-bc9d-a7dd71483cc5 tempest-VolumesGetTest-1961901732 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (235653f7-7663-4f34-bccd-1e801bd03465) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:24:08 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-4c11ecf3-dc04-4262-bc9d-a7dd71483cc5 tempest-VolumesGetTest-1961901732 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (235653f7-7663-4f34-bccd-1e801bd03465) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:24:08 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-4c11ecf3-dc04-4262-bc9d-a7dd71483cc5 tempest-VolumesGetTest-1961901732 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (f0024055-5925-4fc8-acfe-e72db7193404) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:24:08 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-4c11ecf3-dc04-4262-bc9d-a7dd71483cc5 tempest-VolumesGetTest-1961901732 None] Volume f088992b-786a-4641-8bcc-5cb7c819d313: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-f088992b-786a-4641-8bcc-5cb7c819d313', 'volume_size': 1} Apr 17 15:24:08 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-4c11ecf3-dc04-4262-bc9d-a7dd71483cc5 tempest-VolumesGetTest-1961901732 None] creating volume v_f088992b-786a-4641-8bcc-5cb7c819d313. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 15:24:08 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-4c11ecf3-dc04-4262-bc9d-a7dd71483cc5 tempest-VolumesGetTest-1961901732 None] create volume {'name': 'v_f088992b-786a-4641-8bcc-5cb7c819d313', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 15:24:08 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-4c11ecf3-dc04-4262-bc9d-a7dd71483cc5 tempest-VolumesGetTest-1961901732 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:24:08 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-4c11ecf3-dc04-4262-bc9d-a7dd71483cc5 tempest-VolumesGetTest-1961901732 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:24:08 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-11377079-0af3-44c0-b136-161935214a6b tempest-GroupSnapshotsTest-1926061586 None] Geting {'code': 200, 'error': None, 'data': {'origin': 'Pool-0/v_7249f8e4-ed4b-4689-aba3-2766c924f42c@s_8a4c296c-977d-4a9d-a08a-e3b2a1b9a8d4', 'is_clone': True, 'full_name': 'Pool-0/s_8a4c296c-977d-4a9d-a08a-e3b2a1b9a8d4', 'name': 's_8a4c296c-977d-4a9d-a08a-e3b2a1b9a8d4'}} from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_7249f8e4-ed4b-4689-aba3-2766c924f42c/clone {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:24:08 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-11377079-0af3-44c0-b136-161935214a6b tempest-GroupSnapshotsTest-1926061586 None] Sending pool request PUT to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/s_8a4c296c-977d-4a9d-a08a-e3b2a1b9a8d4/properties {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:24:08 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-11377079-0af3-44c0-b136-161935214a6b tempest-GroupSnapshotsTest-1926061586 None] Sending PUT to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/s_8a4c296c-977d-4a9d-a08a-e3b2a1b9a8d4/properties {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:24:09 user sudo[91735]: pam_unix(sudo:session): session closed for user root Apr 17 15:24:09 user cinder-volume[84696]: INFO oslo.privsep.daemon [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] Spawned new privsep daemon via rootwrap Apr 17 15:24:09 user cinder-volume[84696]: DEBUG oslo.privsep.daemon [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] Accepted privsep connection to /tmp/tmp5x2nn_wc/privsep.sock {{(pid=84696) __init__ /usr/local/lib/python3.10/dist-packages/oslo_privsep/daemon.py:362}} Apr 17 15:24:09 user cinder-volume[84696]: INFO oslo.privsep.daemon [-] privsep daemon starting Apr 17 15:24:09 user cinder-volume[84696]: INFO oslo.privsep.daemon [-] privsep process running with uid/gid: 0/0 Apr 17 15:24:09 user cinder-volume[84696]: INFO oslo.privsep.daemon [-] privsep process running with capabilities (eff/prm/inh): CAP_SYS_ADMIN/CAP_SYS_ADMIN/none Apr 17 15:24:09 user cinder-volume[84696]: INFO oslo.privsep.daemon [-] privsep daemon running as pid 91817 Apr 17 15:24:09 user cinder-volume[84696]: DEBUG oslo.privsep.daemon [-] privsep: reply[54c741ff-5265-4e6e-bbb4-642313af0f8a]: (2,) {{(pid=91817) _call_back /usr/local/lib/python3.10/dist-packages/oslo_privsep/daemon.py:501}} Apr 17 15:24:10 user cinder-volume[84696]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /etc/iscsi/initiatorname.iscsi {{(pid=91817) execute /usr/local/lib/python3.10/dist-packages/oslo_concurrency/processutils.py:384}} Apr 17 15:24:10 user cinder-volume[84696]: DEBUG oslo_concurrency.processutils [-] CMD "cat /etc/iscsi/initiatorname.iscsi" returned: 0 in 0.015s {{(pid=91817) execute /usr/local/lib/python3.10/dist-packages/oslo_concurrency/processutils.py:422}} Apr 17 15:24:10 user cinder-volume[84696]: DEBUG oslo.privsep.daemon [-] privsep: reply[39bc5df1-1903-4f80-97dd-fbaa5700b1f3]: (4, ('InitiatorName=iqn.2016-04.com.open-iscsi:b55ce9acc74\n', '')) {{(pid=91817) _call_back /usr/local/lib/python3.10/dist-packages/oslo_privsep/daemon.py:501}} Apr 17 15:24:10 user cinder-volume[84696]: WARNING os_brick.initiator.connectors.nvmeof [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] Could not find nvme_core/parameters/multipath: FileNotFoundError: [Errno 2] No such file or directory: '/sys/module/nvme_core/parameters/multipath' Apr 17 15:24:10 user cinder-volume[84696]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): findmnt / -n -o SOURCE {{(pid=91817) execute /usr/local/lib/python3.10/dist-packages/oslo_concurrency/processutils.py:384}} Apr 17 15:24:10 user cinder-volume[84696]: DEBUG oslo_concurrency.processutils [-] CMD "findmnt / -n -o SOURCE" returned: 0 in 0.024s {{(pid=91817) execute /usr/local/lib/python3.10/dist-packages/oslo_concurrency/processutils.py:422}} Apr 17 15:24:10 user cinder-volume[84696]: DEBUG oslo.privsep.daemon [-] privsep: reply[391a7795-50c5-4e7b-9d94-cb07ebb4005a]: (4, ('/dev/mapper/vg0-lv--0\n', '')) {{(pid=91817) _call_back /usr/local/lib/python3.10/dist-packages/oslo_privsep/daemon.py:501}} Apr 17 15:24:10 user cinder-volume[84696]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): blkid /dev/mapper/vg0-lv--0 -s UUID -o value {{(pid=91817) execute /usr/local/lib/python3.10/dist-packages/oslo_concurrency/processutils.py:384}} Apr 17 15:24:10 user cinder-volume[84696]: DEBUG oslo_concurrency.processutils [-] CMD "blkid /dev/mapper/vg0-lv--0 -s UUID -o value" returned: 0 in 0.024s {{(pid=91817) execute /usr/local/lib/python3.10/dist-packages/oslo_concurrency/processutils.py:422}} Apr 17 15:24:10 user cinder-volume[84696]: DEBUG oslo.privsep.daemon [-] privsep: reply[6318df43-dbfa-44ba-8fc8-4463a75325b5]: (4, ('e95b3b51-542d-42ca-ac40-f83360608668\n', '')) {{(pid=91817) _call_back /usr/local/lib/python3.10/dist-packages/oslo_privsep/daemon.py:501}} Apr 17 15:24:10 user cinder-volume[84696]: DEBUG oslo.privsep.daemon [-] privsep: reply[0f3e7b88-7e25-4478-9724-44035d63f703]: (4, 'e20c3142-5af9-7467-ecd8-70b2e4a210d6') {{(pid=91817) _call_back /usr/local/lib/python3.10/dist-packages/oslo_privsep/daemon.py:501}} Apr 17 15:24:10 user cinder-volume[84696]: DEBUG oslo_concurrency.processutils [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] Running cmd (subprocess): nvme version {{(pid=84696) execute /usr/local/lib/python3.10/dist-packages/oslo_concurrency/processutils.py:384}} Apr 17 15:24:10 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-4c11ecf3-dc04-4262-bc9d-a7dd71483cc5 tempest-VolumesGetTest-1961901732 None] Geting {'code': 201, 'error': None, 'data': {'full_name': 'Pool-0/v_f088992b-786a-4641-8bcc-5cb7c819d313', 'is_clone': False, 'name': 'v_f088992b-786a-4641-8bcc-5cb7c819d313'}} from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:24:10 user cinder-volume[84696]: DEBUG oslo_concurrency.processutils [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] 'nvme version' failed. Not Retrying. {{(pid=84696) execute /usr/local/lib/python3.10/dist-packages/oslo_concurrency/processutils.py:473}} Apr 17 15:24:10 user cinder-volume[84696]: DEBUG os_brick.initiator.connectors.nvmeof [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] nvme not present on system {{(pid=84696) nvme_present /usr/local/lib/python3.10/dist-packages/os_brick/initiator/connectors/nvmeof.py:757}} Apr 17 15:24:10 user cinder-volume[84696]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): nvme show-hostnqn {{(pid=91817) execute /usr/local/lib/python3.10/dist-packages/oslo_concurrency/processutils.py:384}} Apr 17 15:24:10 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-4c11ecf3-dc04-4262-bc9d-a7dd71483cc5 tempest-VolumesGetTest-1961901732 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (f0024055-5925-4fc8-acfe-e72db7193404) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'volume_name': 'volume-f088992b-786a-4641-8bcc-5cb7c819d313', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:24:10 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-4c11ecf3-dc04-4262-bc9d-a7dd71483cc5 tempest-VolumesGetTest-1961901732 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeOnFinishTask;volume:create, create.end' (fb41c7ab-1115-417b-9277-4928878e3833) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:24:10 user cinder-volume[84696]: DEBUG oslo_concurrency.processutils [-] 'nvme show-hostnqn' failed. Not Retrying. {{(pid=91817) execute /usr/local/lib/python3.10/dist-packages/oslo_concurrency/processutils.py:473}} Apr 17 15:24:10 user cinder-volume[84696]: WARNING os_brick.privileged.nvmeof [-] Could not generate host nqn: [Errno 2] No such file or directory: 'nvme' Apr 17 15:24:10 user cinder-volume[84696]: DEBUG oslo.privsep.daemon [-] privsep: reply[fd534493-78e3-4b41-bbd2-aee7efbe7f35]: (4, '') {{(pid=91817) _call_back /usr/local/lib/python3.10/dist-packages/oslo_privsep/daemon.py:501}} Apr 17 15:24:10 user cinder-volume[84696]: DEBUG os_brick.initiator.connectors.lightos [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] LIGHTOS: [Errno 111] ECONNREFUSED {{(pid=84696) find_dsc /usr/local/lib/python3.10/dist-packages/os_brick/initiator/connectors/lightos.py:98}} Apr 17 15:24:10 user cinder-volume[84696]: DEBUG os_brick.initiator.connectors.lightos [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] LIGHTOS: did not find dsc, continuing anyway. {{(pid=84696) get_connector_properties /usr/local/lib/python3.10/dist-packages/os_brick/initiator/connectors/lightos.py:76}} Apr 17 15:24:10 user cinder-volume[84696]: DEBUG os_brick.initiator.connectors.lightos [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] LIGHTOS: no hostnqn found. {{(pid=84696) get_connector_properties /usr/local/lib/python3.10/dist-packages/os_brick/initiator/connectors/lightos.py:84}} Apr 17 15:24:10 user cinder-volume[84696]: DEBUG os_brick.utils [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] <== get_connector_properties: return (2758ms) {'platform': 'x86_64', 'os_type': 'linux', 'ip': '10.0.0.210', 'host': 'user', 'multipath': False, 'initiator': 'iqn.2016-04.com.open-iscsi:b55ce9acc74', 'do_local_attach': False, 'uuid': 'e95b3b51-542d-42ca-ac40-f83360608668', 'system uuid': 'e20c3142-5af9-7467-ecd8-70b2e4a210d6', 'nvme_native_multipath': False} {{(pid=84696) trace_logging_wrapper /usr/local/lib/python3.10/dist-packages/os_brick/utils.py:202}} Apr 17 15:24:10 user cinder-volume[84696]: DEBUG cinder.volume.driver [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] Volume c74b7878-4936-4015-8b63-142923947310: creating export {{(pid=84696) _attach_volume /opt/stack/cinder/cinder/volume/driver.py:1051}} Apr 17 15:24:10 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] create export for volume: c74b7878-4936-4015-8b63-142923947310. {{(pid=84696) create_export /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:699}} Apr 17 15:24:10 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] ensure volume c74b7878-4936-4015-8b63-142923947310 assigned to a proper target {{(pid=84696) _ensure_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:866}} Apr 17 15:24:10 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] check if targe iqn.2010-10.org.openstack:c74b7878-4936-4015-8b63-142923947310 exists {{(pid=84696) is_target /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:376}} Apr 17 15:24:10 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:c74b7878-4936-4015-8b63-142923947310 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:24:10 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:c74b7878-4936-4015-8b63-142923947310 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:24:10 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-4c11ecf3-dc04-4262-bc9d-a7dd71483cc5 tempest-VolumesGetTest-1961901732 None] Volume volume-f088992b-786a-4641-8bcc-5cb7c819d313 (f088992b-786a-4641-8bcc-5cb7c819d313): created successfully Apr 17 15:24:10 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-4c11ecf3-dc04-4262-bc9d-a7dd71483cc5 tempest-VolumesGetTest-1961901732 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeOnFinishTask;volume:create, create.end' (fb41c7ab-1115-417b-9277-4928878e3833) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:24:10 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-4c11ecf3-dc04-4262-bc9d-a7dd71483cc5 tempest-VolumesGetTest-1961901732 None] Flow 'volume_create_manager' (33577b12-4a89-4ad3-bf26-5089bd045c19) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:24:10 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] Geting {'code': 404, 'error': {'class': 'opene.exceptions.ItemNotFoundError', 'message': 'Target iqn.2010-10.org.openstack:c74b7878-4936-4015-8b63-142923947310 not exists.', 'url': 'http://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:c74b7878-4936-4015-8b63-142923947310'}, 'data': None} from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:c74b7878-4936-4015-8b63-142923947310 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:24:10 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] create target and attach volume c74b7878-4936-4015-8b63-142923947310 to it {{(pid=84696) _create_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:837}} Apr 17 15:24:10 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] create target iqn.2010-10.org.openstack:c74b7878-4936-4015-8b63-142923947310 {{(pid=84696) create_target /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:410}} Apr 17 15:24:10 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:24:10 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:24:10 user cinder-volume[84696]: INFO cinder.volume.manager [None req-4c11ecf3-dc04-4262-bc9d-a7dd71483cc5 tempest-VolumesGetTest-1961901732 None] Created volume successfully. Apr 17 15:24:10 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-01e53132-32f6-458d-8c41-044e160c456f tempest-TestVolumeSwap-1714109745 None] Geting {'code': 201, 'error': None, 'data': {'full_name': 'Pool-0/v_66000c93-eb12-4c23-8a69-71e1aa6b521f', 'is_clone': False, 'name': 'v_66000c93-eb12-4c23-8a69-71e1aa6b521f'}} from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:24:10 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-01e53132-32f6-458d-8c41-044e160c456f tempest-TestVolumeSwap-1714109745 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (ecef328b-7b17-4e85-84c1-6c5a22e0f900) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'volume_name': 'volume-66000c93-eb12-4c23-8a69-71e1aa6b521f', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:24:10 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-01e53132-32f6-458d-8c41-044e160c456f tempest-TestVolumeSwap-1714109745 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeOnFinishTask;volume:create, create.end' (0ebbaaa0-fea9-4c81-99bb-f4ea9b458ad8) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:24:10 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-01e53132-32f6-458d-8c41-044e160c456f tempest-TestVolumeSwap-1714109745 None] Volume volume-66000c93-eb12-4c23-8a69-71e1aa6b521f (66000c93-eb12-4c23-8a69-71e1aa6b521f): created successfully Apr 17 15:24:10 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-01e53132-32f6-458d-8c41-044e160c456f tempest-TestVolumeSwap-1714109745 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeOnFinishTask;volume:create, create.end' (0ebbaaa0-fea9-4c81-99bb-f4ea9b458ad8) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:24:10 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-01e53132-32f6-458d-8c41-044e160c456f tempest-TestVolumeSwap-1714109745 None] Flow 'volume_create_manager' (b15a69e0-a98b-4be0-955b-067db7c8d0db) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:24:10 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-11377079-0af3-44c0-b136-161935214a6b tempest-GroupSnapshotsTest-1926061586 None] Geting {'code': 201, 'error': None, 'data': None} from PUT to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/s_8a4c296c-977d-4a9d-a08a-e3b2a1b9a8d4/properties {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:24:10 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-11377079-0af3-44c0-b136-161935214a6b tempest-GroupSnapshotsTest-1926061586 None] volume s_8a4c296c-977d-4a9d-a08a-e3b2a1b9a8d4 properties updated {{(pid=84696) modify_property_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:301}} Apr 17 15:24:10 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-11377079-0af3-44c0-b136-161935214a6b tempest-GroupSnapshotsTest-1926061586 None] create snapshot a4fd2571-7db4-463a-acbe-9e0377b5e9d4 for volume f6603de5-71d3-468b-a673-ce1ff5cd1285 {{(pid=84696) create_snapshot /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:625}} Apr 17 15:24:10 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-11377079-0af3-44c0-b136-161935214a6b tempest-GroupSnapshotsTest-1926061586 None] cloning v_f6603de5-71d3-468b-a673-ce1ff5cd1285 to s_a4fd2571-7db4-463a-acbe-9e0377b5e9d4 {{(pid=84696) _clone_object /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:456}} Apr 17 15:24:10 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-11377079-0af3-44c0-b136-161935214a6b tempest-GroupSnapshotsTest-1926061586 None] create snapshot s_a4fd2571-7db4-463a-acbe-9e0377b5e9d4 {{(pid=84696) create_snapshot /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:635}} Apr 17 15:24:10 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-11377079-0af3-44c0-b136-161935214a6b tempest-GroupSnapshotsTest-1926061586 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_f6603de5-71d3-468b-a673-ce1ff5cd1285/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:24:10 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-11377079-0af3-44c0-b136-161935214a6b tempest-GroupSnapshotsTest-1926061586 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_f6603de5-71d3-468b-a673-ce1ff5cd1285/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:24:10 user cinder-volume[84696]: INFO cinder.volume.manager [None req-01e53132-32f6-458d-8c41-044e160c456f tempest-TestVolumeSwap-1714109745 None] Created volume successfully. Apr 17 15:24:11 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-7f100682-2aaa-4d0e-9177-e13846976f68 tempest-TestVolumeSwap-1714109745 None] Volume reschedule parameters: True retry: {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 15:24:11 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-7f100682-2aaa-4d0e-9177-e13846976f68 tempest-TestVolumeSwap-1714109745 None] Flow 'volume_create_manager' (a53f65f9-93c0-44bb-8870-33c3449b0aba) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:24:11 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-7f100682-2aaa-4d0e-9177-e13846976f68 tempest-TestVolumeSwap-1714109745 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (a3e44c0c-d13c-4cba-95ce-c5f473644559) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:24:11 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-7f100682-2aaa-4d0e-9177-e13846976f68 tempest-TestVolumeSwap-1714109745 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (a3e44c0c-d13c-4cba-95ce-c5f473644559) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:24:11Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TestVolumeSwap-volume-2088850205',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=485ea70c-618d-4373-a78f-a460914fc095,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='00b574796d1942938ed2f348803fcf87',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:24:12Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:24:12Z,use_quota=True,user_id='4c2bfcb4119e483ab1a41e3c598ee1b0',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:24:11 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-7f100682-2aaa-4d0e-9177-e13846976f68 tempest-TestVolumeSwap-1714109745 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (99704662-8d4c-4f4e-ad76-4cd980a02144) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:24:11 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-7f100682-2aaa-4d0e-9177-e13846976f68 tempest-TestVolumeSwap-1714109745 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (99704662-8d4c-4f4e-ad76-4cd980a02144) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:24:11 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-7f100682-2aaa-4d0e-9177-e13846976f68 tempest-TestVolumeSwap-1714109745 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (6cf809ee-9961-4feb-ad3b-1a728dcd3654) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:24:11 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-7f100682-2aaa-4d0e-9177-e13846976f68 tempest-TestVolumeSwap-1714109745 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (6cf809ee-9961-4feb-ad3b-1a728dcd3654) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': '485ea70c-618d-4373-a78f-a460914fc095', 'volume_name': 'volume-485ea70c-618d-4373-a78f-a460914fc095', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:24:11 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-7f100682-2aaa-4d0e-9177-e13846976f68 tempest-TestVolumeSwap-1714109745 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (7ffba057-bee3-4b3a-b361-395c78e50e6d) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:24:11 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-7f100682-2aaa-4d0e-9177-e13846976f68 tempest-TestVolumeSwap-1714109745 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (7ffba057-bee3-4b3a-b361-395c78e50e6d) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:24:11 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-7f100682-2aaa-4d0e-9177-e13846976f68 tempest-TestVolumeSwap-1714109745 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (b1f403c1-212e-4425-9445-108a2d1b3572) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:24:11 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-7f100682-2aaa-4d0e-9177-e13846976f68 tempest-TestVolumeSwap-1714109745 None] Volume 485ea70c-618d-4373-a78f-a460914fc095: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-485ea70c-618d-4373-a78f-a460914fc095', 'volume_size': 1} Apr 17 15:24:11 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-7f100682-2aaa-4d0e-9177-e13846976f68 tempest-TestVolumeSwap-1714109745 None] creating volume v_485ea70c-618d-4373-a78f-a460914fc095. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 15:24:11 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-7f100682-2aaa-4d0e-9177-e13846976f68 tempest-TestVolumeSwap-1714109745 None] create volume {'name': 'v_485ea70c-618d-4373-a78f-a460914fc095', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 15:24:11 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-7f100682-2aaa-4d0e-9177-e13846976f68 tempest-TestVolumeSwap-1714109745 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:24:11 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-7f100682-2aaa-4d0e-9177-e13846976f68 tempest-TestVolumeSwap-1714109745 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:24:11 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-70b57e1d-a47a-45c2-837a-39eda91c984d tempest-VolumesGetTest-1961901732 None] Volume reschedule parameters: True retry: {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 15:24:11 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-70b57e1d-a47a-45c2-837a-39eda91c984d tempest-VolumesGetTest-1961901732 None] Flow 'volume_create_manager' (c44afd12-e650-4865-a216-c3d7f43f4947) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:24:11 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-70b57e1d-a47a-45c2-837a-39eda91c984d tempest-VolumesGetTest-1961901732 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (07699ca6-8a08-4291-9fa3-b6c2a8c210e5) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:24:11 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-70b57e1d-a47a-45c2-837a-39eda91c984d tempest-VolumesGetTest-1961901732 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (07699ca6-8a08-4291-9fa3-b6c2a8c210e5) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:24:11Z,deleted=False,deleted_at=None,display_description='tempest-@#$%^* description-1109961896',display_name=None,ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=f93cf40a-0d85-4277-91a8-d85037f79e7a,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='a65d14b0a2894a6f90506d86feaefd15',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:24:12Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:24:12Z,use_quota=True,user_id='6ffa3051570a4721a48a0171b3cc4d16',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:24:11 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-70b57e1d-a47a-45c2-837a-39eda91c984d tempest-VolumesGetTest-1961901732 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (1cb50826-46a3-416c-8e37-725c78bae393) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:24:11 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-70b57e1d-a47a-45c2-837a-39eda91c984d tempest-VolumesGetTest-1961901732 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (1cb50826-46a3-416c-8e37-725c78bae393) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:24:12 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-70b57e1d-a47a-45c2-837a-39eda91c984d tempest-VolumesGetTest-1961901732 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (23e2bced-7bfe-4d8e-bbf6-17ee2b67b18f) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:24:12 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-70b57e1d-a47a-45c2-837a-39eda91c984d tempest-VolumesGetTest-1961901732 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (23e2bced-7bfe-4d8e-bbf6-17ee2b67b18f) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': 'f93cf40a-0d85-4277-91a8-d85037f79e7a', 'volume_name': 'volume-f93cf40a-0d85-4277-91a8-d85037f79e7a', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:24:12 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-70b57e1d-a47a-45c2-837a-39eda91c984d tempest-VolumesGetTest-1961901732 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (289305c8-d8f7-4bf9-8dcc-43c248c9f0c1) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:24:12 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-70b57e1d-a47a-45c2-837a-39eda91c984d tempest-VolumesGetTest-1961901732 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (289305c8-d8f7-4bf9-8dcc-43c248c9f0c1) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:24:12 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-70b57e1d-a47a-45c2-837a-39eda91c984d tempest-VolumesGetTest-1961901732 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (7bd02b63-4750-4f4c-9a0f-6b85dfabc003) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:24:12 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-70b57e1d-a47a-45c2-837a-39eda91c984d tempest-VolumesGetTest-1961901732 None] Volume f93cf40a-0d85-4277-91a8-d85037f79e7a: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-f93cf40a-0d85-4277-91a8-d85037f79e7a', 'volume_size': 1} Apr 17 15:24:12 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-70b57e1d-a47a-45c2-837a-39eda91c984d tempest-VolumesGetTest-1961901732 None] creating volume v_f93cf40a-0d85-4277-91a8-d85037f79e7a. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 15:24:12 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-70b57e1d-a47a-45c2-837a-39eda91c984d tempest-VolumesGetTest-1961901732 None] create volume {'name': 'v_f93cf40a-0d85-4277-91a8-d85037f79e7a', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 15:24:12 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-70b57e1d-a47a-45c2-837a-39eda91c984d tempest-VolumesGetTest-1961901732 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:24:12 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-70b57e1d-a47a-45c2-837a-39eda91c984d tempest-VolumesGetTest-1961901732 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:24:12 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-11377079-0af3-44c0-b136-161935214a6b tempest-GroupSnapshotsTest-1926061586 None] Geting {'code': 200, 'error': None, 'data': {'name': 's_a4fd2571-7db4-463a-acbe-9e0377b5e9d4'}} from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_f6603de5-71d3-468b-a673-ce1ff5cd1285/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:24:12 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-11377079-0af3-44c0-b136-161935214a6b tempest-GroupSnapshotsTest-1926061586 None] create volume s_a4fd2571-7db4-463a-acbe-9e0377b5e9d4 from snapshot s_a4fd2571-7db4-463a-acbe-9e0377b5e9d4 {{(pid=84696) create_volume_from_snapshot /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:674}} Apr 17 15:24:12 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-11377079-0af3-44c0-b136-161935214a6b tempest-GroupSnapshotsTest-1926061586 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_f6603de5-71d3-468b-a673-ce1ff5cd1285/clone {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:24:12 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-11377079-0af3-44c0-b136-161935214a6b tempest-GroupSnapshotsTest-1926061586 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_f6603de5-71d3-468b-a673-ce1ff5cd1285/clone {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:24:13 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-70b57e1d-a47a-45c2-837a-39eda91c984d tempest-VolumesGetTest-1961901732 None] Geting {'code': 201, 'error': None, 'data': {'full_name': 'Pool-0/v_f93cf40a-0d85-4277-91a8-d85037f79e7a', 'is_clone': False, 'name': 'v_f93cf40a-0d85-4277-91a8-d85037f79e7a'}} from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:24:13 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-7f100682-2aaa-4d0e-9177-e13846976f68 tempest-TestVolumeSwap-1714109745 None] Geting {'code': 201, 'error': None, 'data': {'full_name': 'Pool-0/v_485ea70c-618d-4373-a78f-a460914fc095', 'is_clone': False, 'name': 'v_485ea70c-618d-4373-a78f-a460914fc095'}} from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:24:13 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-70b57e1d-a47a-45c2-837a-39eda91c984d tempest-VolumesGetTest-1961901732 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (7bd02b63-4750-4f4c-9a0f-6b85dfabc003) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'volume_name': 'volume-f93cf40a-0d85-4277-91a8-d85037f79e7a', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:24:13 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-70b57e1d-a47a-45c2-837a-39eda91c984d tempest-VolumesGetTest-1961901732 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeOnFinishTask;volume:create, create.end' (71864409-74e7-4ada-91d8-63ecc5981756) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:24:13 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-7f100682-2aaa-4d0e-9177-e13846976f68 tempest-TestVolumeSwap-1714109745 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (b1f403c1-212e-4425-9445-108a2d1b3572) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'volume_name': 'volume-485ea70c-618d-4373-a78f-a460914fc095', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:24:13 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-7f100682-2aaa-4d0e-9177-e13846976f68 tempest-TestVolumeSwap-1714109745 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeOnFinishTask;volume:create, create.end' (36a09fb9-cc1a-4caf-95d8-9d9421386b9b) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:24:13 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-7f100682-2aaa-4d0e-9177-e13846976f68 tempest-TestVolumeSwap-1714109745 None] Volume volume-485ea70c-618d-4373-a78f-a460914fc095 (485ea70c-618d-4373-a78f-a460914fc095): created successfully Apr 17 15:24:13 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-7f100682-2aaa-4d0e-9177-e13846976f68 tempest-TestVolumeSwap-1714109745 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeOnFinishTask;volume:create, create.end' (36a09fb9-cc1a-4caf-95d8-9d9421386b9b) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:24:13 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-7f100682-2aaa-4d0e-9177-e13846976f68 tempest-TestVolumeSwap-1714109745 None] Flow 'volume_create_manager' (a53f65f9-93c0-44bb-8870-33c3449b0aba) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:24:13 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-70b57e1d-a47a-45c2-837a-39eda91c984d tempest-VolumesGetTest-1961901732 None] Volume volume-f93cf40a-0d85-4277-91a8-d85037f79e7a (f93cf40a-0d85-4277-91a8-d85037f79e7a): created successfully Apr 17 15:24:13 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-70b57e1d-a47a-45c2-837a-39eda91c984d tempest-VolumesGetTest-1961901732 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeOnFinishTask;volume:create, create.end' (71864409-74e7-4ada-91d8-63ecc5981756) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:24:13 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-70b57e1d-a47a-45c2-837a-39eda91c984d tempest-VolumesGetTest-1961901732 None] Flow 'volume_create_manager' (c44afd12-e650-4865-a216-c3d7f43f4947) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:24:14 user cinder-volume[84696]: INFO cinder.volume.manager [None req-7f100682-2aaa-4d0e-9177-e13846976f68 tempest-TestVolumeSwap-1714109745 None] Created volume successfully. Apr 17 15:24:14 user cinder-volume[84696]: INFO cinder.volume.manager [None req-70b57e1d-a47a-45c2-837a-39eda91c984d tempest-VolumesGetTest-1961901732 None] Created volume successfully. Apr 17 15:24:14 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] Geting {'code': 201, 'error': None, 'data': {'active': True, 'allow_ip': [], 'conflicted': False, 'deny_ip': [], 'incoming_users_active': True, 'name': 'iqn.2010-10.org.openstack:c74b7878-4936-4015-8b63-142923947310', 'outgoing_user': None}} from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:24:14 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] atach volume v_c74b7878-4936-4015-8b63-142923947310 to target iqn.2010-10.org.openstack:c74b7878-4936-4015-8b63-142923947310 {{(pid=84696) attach_target_vol /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:576}} Apr 17 15:24:14 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:c74b7878-4936-4015-8b63-142923947310/luns {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:24:14 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:c74b7878-4936-4015-8b63-142923947310/luns {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:24:14 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-ac753b94-fda1-4a0a-a55a-df6da267ee4d tempest-TaggedAttachmentsTest-1309189231 None] Volume reschedule parameters: True retry: {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 15:24:14 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-ac753b94-fda1-4a0a-a55a-df6da267ee4d tempest-TaggedAttachmentsTest-1309189231 None] Flow 'volume_create_manager' (d206ca81-2797-480d-8065-3ba7c491bb37) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:24:14 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-ac753b94-fda1-4a0a-a55a-df6da267ee4d tempest-TaggedAttachmentsTest-1309189231 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (e11910b0-a42b-42e0-8aca-0c97ea5e734c) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:24:14 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-ac753b94-fda1-4a0a-a55a-df6da267ee4d tempest-TaggedAttachmentsTest-1309189231 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (e11910b0-a42b-42e0-8aca-0c97ea5e734c) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:24:14Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TaggedAttachmentsTest-volume-2098521363',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=3c05f4f0-17a8-4f4c-9987-1fcb502ffd3a,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='9dfa73323c3c4c689f4012505d8b4833',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:24:15Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:24:15Z,use_quota=True,user_id='f081ecb75ec14053b68d74cacf61c802',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:24:14 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-ac753b94-fda1-4a0a-a55a-df6da267ee4d tempest-TaggedAttachmentsTest-1309189231 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (70ed882c-b62f-4c6d-893a-0df3267f84ea) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:24:14 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-ac753b94-fda1-4a0a-a55a-df6da267ee4d tempest-TaggedAttachmentsTest-1309189231 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (70ed882c-b62f-4c6d-893a-0df3267f84ea) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:24:14 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-ac753b94-fda1-4a0a-a55a-df6da267ee4d tempest-TaggedAttachmentsTest-1309189231 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (ff5a8ae3-585b-4079-acea-ad235d9d7415) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:24:14 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-ac753b94-fda1-4a0a-a55a-df6da267ee4d tempest-TaggedAttachmentsTest-1309189231 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (ff5a8ae3-585b-4079-acea-ad235d9d7415) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': '3c05f4f0-17a8-4f4c-9987-1fcb502ffd3a', 'volume_name': 'volume-3c05f4f0-17a8-4f4c-9987-1fcb502ffd3a', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:24:14 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-ac753b94-fda1-4a0a-a55a-df6da267ee4d tempest-TaggedAttachmentsTest-1309189231 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (8dbc4aae-349c-4d18-b0e2-390a7c640b22) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:24:14 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-ac753b94-fda1-4a0a-a55a-df6da267ee4d tempest-TaggedAttachmentsTest-1309189231 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (8dbc4aae-349c-4d18-b0e2-390a7c640b22) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:24:14 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-ac753b94-fda1-4a0a-a55a-df6da267ee4d tempest-TaggedAttachmentsTest-1309189231 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (eca7fba9-a3f1-46ea-9e2e-a24eb8a9b2a8) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:24:14 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-ac753b94-fda1-4a0a-a55a-df6da267ee4d tempest-TaggedAttachmentsTest-1309189231 None] Volume 3c05f4f0-17a8-4f4c-9987-1fcb502ffd3a: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-3c05f4f0-17a8-4f4c-9987-1fcb502ffd3a', 'volume_size': 1} Apr 17 15:24:14 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-ac753b94-fda1-4a0a-a55a-df6da267ee4d tempest-TaggedAttachmentsTest-1309189231 None] creating volume v_3c05f4f0-17a8-4f4c-9987-1fcb502ffd3a. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 15:24:14 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-ac753b94-fda1-4a0a-a55a-df6da267ee4d tempest-TaggedAttachmentsTest-1309189231 None] create volume {'name': 'v_3c05f4f0-17a8-4f4c-9987-1fcb502ffd3a', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 15:24:14 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-ac753b94-fda1-4a0a-a55a-df6da267ee4d tempest-TaggedAttachmentsTest-1309189231 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:24:14 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-ac753b94-fda1-4a0a-a55a-df6da267ee4d tempest-TaggedAttachmentsTest-1309189231 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:24:15 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-11377079-0af3-44c0-b136-161935214a6b tempest-GroupSnapshotsTest-1926061586 None] Geting {'code': 200, 'error': None, 'data': {'origin': 'Pool-0/v_f6603de5-71d3-468b-a673-ce1ff5cd1285@s_a4fd2571-7db4-463a-acbe-9e0377b5e9d4', 'is_clone': True, 'full_name': 'Pool-0/s_a4fd2571-7db4-463a-acbe-9e0377b5e9d4', 'name': 's_a4fd2571-7db4-463a-acbe-9e0377b5e9d4'}} from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_f6603de5-71d3-468b-a673-ce1ff5cd1285/clone {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:24:15 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-11377079-0af3-44c0-b136-161935214a6b tempest-GroupSnapshotsTest-1926061586 None] Sending pool request PUT to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/s_a4fd2571-7db4-463a-acbe-9e0377b5e9d4/properties {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:24:15 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-11377079-0af3-44c0-b136-161935214a6b tempest-GroupSnapshotsTest-1926061586 None] Sending PUT to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/s_a4fd2571-7db4-463a-acbe-9e0377b5e9d4/properties {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:24:15 user cinder-volume[84696]: DEBUG cinder.coordination [None req-b5809a81-b0b1-43e3-93ac-27311b059c17 tempest-VolumesGetTest-1961901732 None] Acquiring lock "cinder-f93cf40a-0d85-4277-91a8-d85037f79e7a-delete_volume" by "delete_volume" {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:192}} Apr 17 15:24:15 user cinder-volume[84696]: DEBUG cinder.coordination [None req-b5809a81-b0b1-43e3-93ac-27311b059c17 tempest-VolumesGetTest-1961901732 None] Lock "cinder-f93cf40a-0d85-4277-91a8-d85037f79e7a-delete_volume" acquired by "delete_volume" :: waited 0.014s {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:196}} Apr 17 15:24:15 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=64,cinder:UPDATE=28,cinder:INSERT=4,cinder:DELETE=4 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:24:15 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-b5809a81-b0b1-43e3-93ac-27311b059c17 tempest-VolumesGetTest-1961901732 None] remove_export for volume: f93cf40a-0d85-4277-91a8-d85037f79e7a. {{(pid=84696) remove_export /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:718}} Apr 17 15:24:15 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-b5809a81-b0b1-43e3-93ac-27311b059c17 tempest-VolumesGetTest-1961901732 None] remove export {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:912}} Apr 17 15:24:15 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-b5809a81-b0b1-43e3-93ac-27311b059c17 tempest-VolumesGetTest-1961901732 None] detach volume:Volume(_name_id=None,admin_metadata={},attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:24:11Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesGetTest-Volume-1860415110',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=f93cf40a-0d85-4277-91a8-d85037f79e7a,launched_at=2023-04-17T15:24:14Z,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='a65d14b0a2894a6f90506d86feaefd15',provider_auth='CHAP TcWOVmaB jOshNubB0GDrQm',provider_geometry=None,provider_id=None,provider_location='172.16.0.220:3260,1 iqn.2010-10.org.openstack:f93cf40a-0d85-4277-91a8-d85037f79e7a 0',replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:24:12Z,service_uuid='e628352e-c4cf-40eb-8389-cce40663026a',shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='deleting',terminated_at=2023-04-17T15:24:15Z,updated_at=2023-04-17T15:24:15Z,use_quota=True,user_id='6ffa3051570a4721a48a0171b3cc4d16',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c) from target:iqn.2010-10.org.openstack:f93cf40a-0d85-4277-91a8-d85037f79e7a. {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:913}} Apr 17 15:24:15 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-b5809a81-b0b1-43e3-93ac-27311b059c17 tempest-VolumesGetTest-1961901732 None] detach volume v_f93cf40a-0d85-4277-91a8-d85037f79e7a from target iqn.2010-10.org.openstack:f93cf40a-0d85-4277-91a8-d85037f79e7a {{(pid=84696) detach_target_vol /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:606}} Apr 17 15:24:15 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-b5809a81-b0b1-43e3-93ac-27311b059c17 tempest-VolumesGetTest-1961901732 None] Sending pool request DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:f93cf40a-0d85-4277-91a8-d85037f79e7a/luns/v_f93cf40a-0d85-4277-91a8-d85037f79e7a {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:24:15 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-b5809a81-b0b1-43e3-93ac-27311b059c17 tempest-VolumesGetTest-1961901732 None] Sending DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:f93cf40a-0d85-4277-91a8-d85037f79e7a/luns/v_f93cf40a-0d85-4277-91a8-d85037f79e7a {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:24:15 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-b5809a81-b0b1-43e3-93ac-27311b059c17 tempest-VolumesGetTest-1961901732 None] Geting {'code': 404, 'error': {'class': 'opene.exceptions.ItemNotFoundError', 'message': 'Target iqn.2010-10.org.openstack:f93cf40a-0d85-4277-91a8-d85037f79e7a not exists.', 'url': 'http://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:f93cf40a-0d85-4277-91a8-d85037f79e7a/luns/v_f93cf40a-0d85-4277-91a8-d85037f79e7a'}, 'data': None} from DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:f93cf40a-0d85-4277-91a8-d85037f79e7a/luns/v_f93cf40a-0d85-4277-91a8-d85037f79e7a {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:24:15 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-b5809a81-b0b1-43e3-93ac-27311b059c17 tempest-VolumesGetTest-1961901732 None] failed to remove resource iqn.2010-10.org.openstack:f93cf40a-0d85-4277-91a8-d85037f79e7a because of JDSS resource v_f93cf40a-0d85-4277-91a8-d85037f79e7a DNE. {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:920}} Apr 17 15:24:15 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-b5809a81-b0b1-43e3-93ac-27311b059c17 tempest-VolumesGetTest-1961901732 None] delete target: iqn.2010-10.org.openstack:f93cf40a-0d85-4277-91a8-d85037f79e7a {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:930}} Apr 17 15:24:15 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-b5809a81-b0b1-43e3-93ac-27311b059c17 tempest-VolumesGetTest-1961901732 None] delete target iqn.2010-10.org.openstack:f93cf40a-0d85-4277-91a8-d85037f79e7a {{(pid=84696) delete_target /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:442}} Apr 17 15:24:15 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-b5809a81-b0b1-43e3-93ac-27311b059c17 tempest-VolumesGetTest-1961901732 None] Sending pool request DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:f93cf40a-0d85-4277-91a8-d85037f79e7a {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:24:15 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-b5809a81-b0b1-43e3-93ac-27311b059c17 tempest-VolumesGetTest-1961901732 None] Sending DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:f93cf40a-0d85-4277-91a8-d85037f79e7a {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:24:15 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-b5809a81-b0b1-43e3-93ac-27311b059c17 tempest-VolumesGetTest-1961901732 None] Geting {'code': 404, 'error': {'class': 'opene.exceptions.ItemNotFoundError', 'message': 'Target iqn.2010-10.org.openstack:f93cf40a-0d85-4277-91a8-d85037f79e7a not exists.', 'url': 'http://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:f93cf40a-0d85-4277-91a8-d85037f79e7a'}, 'data': None} from DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:f93cf40a-0d85-4277-91a8-d85037f79e7a {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:24:15 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-b5809a81-b0b1-43e3-93ac-27311b059c17 tempest-VolumesGetTest-1961901732 None] failed to remove resource iqn.2010-10.org.openstack:f93cf40a-0d85-4277-91a8-d85037f79e7a because of JDSS resource iqn.2010-10.org.openstack:f93cf40a-0d85-4277-91a8-d85037f79e7a DNE. {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:935}} Apr 17 15:24:15 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-b5809a81-b0b1-43e3-93ac-27311b059c17 tempest-VolumesGetTest-1961901732 None] deleating volume v_f93cf40a-0d85-4277-91a8-d85037f79e7a {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 15:24:15 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-b5809a81-b0b1-43e3-93ac-27311b059c17 tempest-VolumesGetTest-1961901732 None] get snapshots for volume v_f93cf40a-0d85-4277-91a8-d85037f79e7a {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 15:24:15 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-b5809a81-b0b1-43e3-93ac-27311b059c17 tempest-VolumesGetTest-1961901732 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_f93cf40a-0d85-4277-91a8-d85037f79e7a/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:24:15 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-b5809a81-b0b1-43e3-93ac-27311b059c17 tempest-VolumesGetTest-1961901732 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_f93cf40a-0d85-4277-91a8-d85037f79e7a/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:24:15 user cinder-volume[84696]: DEBUG oslo_service.periodic_task [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Running periodic task VolumeManager.publish_service_capabilities {{(pid=84696) run_periodic_tasks /usr/local/lib/python3.10/dist-packages/oslo_service/periodic_task.py:210}} Apr 17 15:24:15 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Updating volume stats {{(pid=84696) _update_volume_stats /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:724}} Apr 17 15:24:15 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Get pool Pool-0 fsprops {{(pid=84696) get_pool_stats /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:894}} Apr 17 15:24:15 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:24:15 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:24:16 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-e933b70a-f56c-4690-8a4f-79fe3eb067fd tempest-VolumesTransfersTest-276179586 None] Volume reschedule parameters: True retry: {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 15:24:16 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e933b70a-f56c-4690-8a4f-79fe3eb067fd tempest-VolumesTransfersTest-276179586 None] Flow 'volume_create_manager' (19a6193b-3c6c-4d5c-b9ac-15272d70be40) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:24:16 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e933b70a-f56c-4690-8a4f-79fe3eb067fd tempest-VolumesTransfersTest-276179586 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (69e5afee-4f47-4fb0-93d3-03a0dd466e66) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:24:16 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=64,cinder:INSERT=5,cinder:UPDATE=29,cinder:DELETE=2 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:24:16 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e933b70a-f56c-4690-8a4f-79fe3eb067fd tempest-VolumesTransfersTest-276179586 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (69e5afee-4f47-4fb0-93d3-03a0dd466e66) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:24:16Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesTransfersTest-Volume-604455400',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=786b4c79-f126-4791-8350-4fed7646bd3b,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='8b0ce9e109f84b6988b4adf66910910c',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:24:16Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:24:16Z,use_quota=True,user_id='bee3a4dd32ab4d0ba197cf1e28638290',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:24:16 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e933b70a-f56c-4690-8a4f-79fe3eb067fd tempest-VolumesTransfersTest-276179586 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (ef227b3d-cd3d-4e55-baeb-69b7dbb6c870) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:24:16 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e933b70a-f56c-4690-8a4f-79fe3eb067fd tempest-VolumesTransfersTest-276179586 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (ef227b3d-cd3d-4e55-baeb-69b7dbb6c870) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:24:16 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e933b70a-f56c-4690-8a4f-79fe3eb067fd tempest-VolumesTransfersTest-276179586 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (6206704b-e190-4ca3-b9dd-435d76253120) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:24:16 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e933b70a-f56c-4690-8a4f-79fe3eb067fd tempest-VolumesTransfersTest-276179586 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (6206704b-e190-4ca3-b9dd-435d76253120) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': '786b4c79-f126-4791-8350-4fed7646bd3b', 'volume_name': 'volume-786b4c79-f126-4791-8350-4fed7646bd3b', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:24:16 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e933b70a-f56c-4690-8a4f-79fe3eb067fd tempest-VolumesTransfersTest-276179586 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (910ae939-0ce6-4f5a-9ca6-8243160f13df) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:24:16 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e933b70a-f56c-4690-8a4f-79fe3eb067fd tempest-VolumesTransfersTest-276179586 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (910ae939-0ce6-4f5a-9ca6-8243160f13df) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:24:16 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e933b70a-f56c-4690-8a4f-79fe3eb067fd tempest-VolumesTransfersTest-276179586 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (7c014c73-21f1-4867-8527-d3d623277904) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:24:16 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-e933b70a-f56c-4690-8a4f-79fe3eb067fd tempest-VolumesTransfersTest-276179586 None] Volume 786b4c79-f126-4791-8350-4fed7646bd3b: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-786b4c79-f126-4791-8350-4fed7646bd3b', 'volume_size': 1} Apr 17 15:24:16 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-e933b70a-f56c-4690-8a4f-79fe3eb067fd tempest-VolumesTransfersTest-276179586 None] creating volume v_786b4c79-f126-4791-8350-4fed7646bd3b. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 15:24:16 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-e933b70a-f56c-4690-8a4f-79fe3eb067fd tempest-VolumesTransfersTest-276179586 None] create volume {'name': 'v_786b4c79-f126-4791-8350-4fed7646bd3b', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 15:24:16 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-e933b70a-f56c-4690-8a4f-79fe3eb067fd tempest-VolumesTransfersTest-276179586 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:24:16 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-e933b70a-f56c-4690-8a4f-79fe3eb067fd tempest-VolumesTransfersTest-276179586 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:24:16 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-b5809a81-b0b1-43e3-93ac-27311b059c17 tempest-VolumesGetTest-1961901732 None] Geting {'code': 200, 'error': None, 'data': {'results': 0, 'entries': []}} from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_f93cf40a-0d85-4277-91a8-d85037f79e7a/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:24:16 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-b5809a81-b0b1-43e3-93ac-27311b059c17 tempest-VolumesGetTest-1961901732 None] get volume v_f93cf40a-0d85-4277-91a8-d85037f79e7a info {{(pid=84696) get_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:239}} Apr 17 15:24:16 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-b5809a81-b0b1-43e3-93ac-27311b059c17 tempest-VolumesGetTest-1961901732 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_f93cf40a-0d85-4277-91a8-d85037f79e7a {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:24:16 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-b5809a81-b0b1-43e3-93ac-27311b059c17 tempest-VolumesGetTest-1961901732 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_f93cf40a-0d85-4277-91a8-d85037f79e7a {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:24:16 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-b5809a81-b0b1-43e3-93ac-27311b059c17 tempest-VolumesGetTest-1961901732 None] Geting {'code': 200, 'error': None, 'data': {'origin': None, 'relatime': None, 'acltype': None, 'vscan': None, 'full_name': 'Pool-0/v_f93cf40a-0d85-4277-91a8-d85037f79e7a', 'userrefs': None, 'primarycache': 'all', 'logbias': 'latency', 'creation': '1681746446', 'sync': 'always', 'is_clone': False, 'dedup': 'off', 'sharenfs': None, 'receive_resume_token': None, 'volsize': '1073741824', 'referenced': '57344', 'sharesmb': None, 'createtxg': '187874', 'reservation': '0', 'scontext': None, 'mountpoint': None, 'casesensitivity': None, 'guid': '14160501708223460183', 'usedbyrefreservation': '0', 'dnodesize': None, 'written': '57344', 'logicalused': '28672', 'compressratio': '1.00', 'rootcontext': 'none', 'default_scsi_id': '59fdaa6b7135fb29', 'type': 'volume', 'compression': 'lz4', 'snapdir': None, 'overlay': None, 'encryption': 'off', 'xattr': None, 'volmode': 'default', 'copies': '1', 'snapshot_limit': '18446744073709551615', 'aclinherit': None, 'defcontext': 'none', 'readonly': 'off', 'version': None, 'recordsize': None, 'filesystem_limit': None, 'mounted': None, 'mlslabel': 'none', 'secondarycache': 'all', 'refreservation': '0', 'available': '113926152192', 'san:volume_id': '59fdaa6b7135fb299a1e9c977fd8b0fcf3fb6892', 'encryptionroot': None, 'exec': None, 'refquota': None, 'refcompressratio': '1.00', 'quota': None, 'utf8only': None, 'keylocation': 'none', 'snapdev': 'hidden', 'snapshot_count': '18446744073709551615', 'fscontext': 'none', 'clones': None, 'canmount': None, 'keystatus': None, 'atime': None, 'usedbysnapshots': '0', 'normalization': None, 'usedbychildren': '0', 'volblocksize': '131072', 'usedbydataset': '57344', 'objsetid': '123165', 'name': 'v_f93cf40a-0d85-4277-91a8-d85037f79e7a', 'defer_destroy': None, 'pbkdf2iters': '0', 'checksum': 'on', 'redundant_metadata': 'all', 'filesystem_count': None, 'devices': None, 'keyformat': 'none', 'setuid': None, 'used': '57344', 'logicalreferenced': '28672', 'context': 'none', 'zoned': None, 'nbmand': None}} from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_f93cf40a-0d85-4277-91a8-d85037f79e7a {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:24:16 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-b5809a81-b0b1-43e3-93ac-27311b059c17 tempest-VolumesGetTest-1961901732 None] delete volume:v_f93cf40a-0d85-4277-91a8-d85037f79e7a recursively children:{'force_umount': True} {{(pid=84696) delete_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:335}} Apr 17 15:24:16 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-b5809a81-b0b1-43e3-93ac-27311b059c17 tempest-VolumesGetTest-1961901732 None] Sending pool request DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_f93cf40a-0d85-4277-91a8-d85037f79e7a {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:24:16 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-b5809a81-b0b1-43e3-93ac-27311b059c17 tempest-VolumesGetTest-1961901732 None] Sending DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_f93cf40a-0d85-4277-91a8-d85037f79e7a {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:24:16 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] Geting {'code': 201, 'error': None, 'data': {'block_size': 512, 'device_handler': 'vdisk_blockio', 'eui': '6132396565663337', 'lun': 0, 'mode': 'wt', 'name': 'v_c74b7878-4936-4015-8b63-142923947310', 'prod_id': 'Storage', 'scsi_id': 'a29eef3759db89b1'}} from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:c74b7878-4936-4015-8b63-142923947310/luns {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:24:16 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] add credentails to target iqn.2010-10.org.openstack:c74b7878-4936-4015-8b63-142923947310 {{(pid=84696) create_target_user /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:474}} Apr 17 15:24:16 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:c74b7878-4936-4015-8b63-142923947310/incoming-users {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:24:16 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:c74b7878-4936-4015-8b63-142923947310/incoming-users {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:24:16 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] Geting {'code': 201, 'error': None, 'data': {'name': 'PUvXoCNO'}} from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:c74b7878-4936-4015-8b63-142923947310/incoming-users {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:24:16 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] initialize_connection for c74b7878-4936-4015-8b63-142923947310 10.0.0.210. {{(pid=84696) initialize_connection /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:1018}} Apr 17 15:24:16 user cinder-volume[84696]: DEBUG os_brick.initiator.connector [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] Factory for iscsi on None {{(pid=84696) factory /usr/local/lib/python3.10/dist-packages/os_brick/initiator/connector.py:281}} Apr 17 15:24:16 user cinder-volume[84696]: DEBUG os_brick.initiator.connectors.iscsi [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] ==> connect_volume: call "{'self': , 'connection_properties': {'target_iqn': 'iqn.2010-10.org.openstack:c74b7878-4936-4015-8b63-142923947310', 'target_portal': '172.16.0.220:3260', 'target_discovered': False, 'auth_method': 'CHAP', 'auth_username': 'PUvXoCNO', 'auth_password': '***', 'target_lun': False}}" {{(pid=84696) trace_logging_wrapper /usr/local/lib/python3.10/dist-packages/os_brick/utils.py:175}} Apr 17 15:24:16 user cinder-volume[84696]: DEBUG os_brick.initiator.connectors.base [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] Acquiring lock "connect_volume" by "os_brick.initiator.connectors.iscsi.ISCSIConnector.connect_volume" {{(pid=84696) inner /usr/local/lib/python3.10/dist-packages/os_brick/initiator/connectors/base.py:68}} Apr 17 15:24:16 user cinder-volume[84696]: DEBUG os_brick.initiator.connectors.base [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] Lock "connect_volume" acquired by "os_brick.initiator.connectors.iscsi.ISCSIConnector.connect_volume" :: waited 0.003s {{(pid=84696) inner /usr/local/lib/python3.10/dist-packages/os_brick/initiator/connectors/base.py:73}} Apr 17 15:24:16 user cinder-volume[84696]: DEBUG os_brick.initiator.connectors.base [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] Acquiring lock "connect_to_iscsi_portal-172.16.0.220:3260-iqn.2010-10.org.openstack:c74b7878-4936-4015-8b63-142923947310" by "os_brick.initiator.connectors.iscsi.ISCSIConnector._connect_to_iscsi_portal_unsafe" {{(pid=84696) inner /usr/local/lib/python3.10/dist-packages/os_brick/initiator/connectors/base.py:68}} Apr 17 15:24:16 user cinder-volume[84696]: DEBUG os_brick.initiator.connectors.base [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] Lock "connect_to_iscsi_portal-172.16.0.220:3260-iqn.2010-10.org.openstack:c74b7878-4936-4015-8b63-142923947310" acquired by "os_brick.initiator.connectors.iscsi.ISCSIConnector._connect_to_iscsi_portal_unsafe" :: waited 0.001s {{(pid=84696) inner /usr/local/lib/python3.10/dist-packages/os_brick/initiator/connectors/base.py:73}} Apr 17 15:24:16 user cinder-volume[84696]: INFO os_brick.initiator.connectors.iscsi [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] Trying to connect to iSCSI portal 172.16.0.220:3260 Apr 17 15:24:16 user cinder-volume[84696]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m node -T iqn.2010-10.org.openstack:c74b7878-4936-4015-8b63-142923947310 -p 172.16.0.220:3260 {{(pid=91817) execute /usr/local/lib/python3.10/dist-packages/oslo_concurrency/processutils.py:384}} Apr 17 15:24:16 user cinder-volume[84696]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m node -T iqn.2010-10.org.openstack:c74b7878-4936-4015-8b63-142923947310 -p 172.16.0.220:3260" returned: 21 in 0.026s {{(pid=91817) execute /usr/local/lib/python3.10/dist-packages/oslo_concurrency/processutils.py:422}} Apr 17 15:24:16 user cinder-volume[84696]: DEBUG oslo.privsep.daemon [-] privsep: reply[d527d9b1-c8ca-43e4-8c84-613b9a500e7b]: (4, ('', 'iscsiadm: No records found\n')) {{(pid=91817) _call_back /usr/local/lib/python3.10/dist-packages/oslo_privsep/daemon.py:501}} Apr 17 15:24:16 user cinder-volume[84696]: DEBUG os_brick.initiator.connectors.iscsi [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] iscsiadm (): stdout= stderr=iscsiadm: No records found Apr 17 15:24:16 user cinder-volume[84696]: {{(pid=84696) _run_iscsiadm /usr/local/lib/python3.10/dist-packages/os_brick/initiator/connectors/iscsi.py:1026}} Apr 17 15:24:16 user cinder-volume[84696]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m node -T iqn.2010-10.org.openstack:c74b7878-4936-4015-8b63-142923947310 -p 172.16.0.220:3260 --interface default --op new {{(pid=91817) execute /usr/local/lib/python3.10/dist-packages/oslo_concurrency/processutils.py:384}} Apr 17 15:24:16 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-ac753b94-fda1-4a0a-a55a-df6da267ee4d tempest-TaggedAttachmentsTest-1309189231 None] Geting {'code': 201, 'error': None, 'data': {'full_name': 'Pool-0/v_3c05f4f0-17a8-4f4c-9987-1fcb502ffd3a', 'is_clone': False, 'name': 'v_3c05f4f0-17a8-4f4c-9987-1fcb502ffd3a'}} from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:24:16 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-ac753b94-fda1-4a0a-a55a-df6da267ee4d tempest-TaggedAttachmentsTest-1309189231 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (eca7fba9-a3f1-46ea-9e2e-a24eb8a9b2a8) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'volume_name': 'volume-3c05f4f0-17a8-4f4c-9987-1fcb502ffd3a', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:24:16 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-ac753b94-fda1-4a0a-a55a-df6da267ee4d tempest-TaggedAttachmentsTest-1309189231 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeOnFinishTask;volume:create, create.end' (951f8dde-8497-4471-86cb-4f1c6d06ce54) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:24:16 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-ac753b94-fda1-4a0a-a55a-df6da267ee4d tempest-TaggedAttachmentsTest-1309189231 None] Volume volume-3c05f4f0-17a8-4f4c-9987-1fcb502ffd3a (3c05f4f0-17a8-4f4c-9987-1fcb502ffd3a): created successfully Apr 17 15:24:16 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-ac753b94-fda1-4a0a-a55a-df6da267ee4d tempest-TaggedAttachmentsTest-1309189231 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeOnFinishTask;volume:create, create.end' (951f8dde-8497-4471-86cb-4f1c6d06ce54) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:24:16 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-ac753b94-fda1-4a0a-a55a-df6da267ee4d tempest-TaggedAttachmentsTest-1309189231 None] Flow 'volume_create_manager' (d206ca81-2797-480d-8065-3ba7c491bb37) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:24:16 user cinder-volume[84696]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m node -T iqn.2010-10.org.openstack:c74b7878-4936-4015-8b63-142923947310 -p 172.16.0.220:3260 --interface default --op new" returned: 0 in 0.038s {{(pid=91817) execute /usr/local/lib/python3.10/dist-packages/oslo_concurrency/processutils.py:422}} Apr 17 15:24:16 user cinder-volume[84696]: DEBUG oslo.privsep.daemon [-] privsep: reply[091e08dc-2129-4767-8485-a06b319b8d4e]: (4, ('New iSCSI node [tcp:[hw=,ip=,net_if=,iscsi_if=default] 172.16.0.220,3260,-1 iqn.2010-10.org.openstack:c74b7878-4936-4015-8b63-142923947310] added\n', '')) {{(pid=91817) _call_back /usr/local/lib/python3.10/dist-packages/oslo_privsep/daemon.py:501}} Apr 17 15:24:16 user cinder-volume[84696]: DEBUG os_brick.initiator.connectors.iscsi [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] iscsiadm ('--interface', 'default', '--op', 'new'): stdout=New iSCSI node [tcp:[hw=,ip=,net_if=,iscsi_if=default] 172.16.0.220,3260,-1 iqn.2010-10.org.openstack:c74b7878-4936-4015-8b63-142923947310] added Apr 17 15:24:16 user cinder-volume[84696]: stderr= {{(pid=84696) _run_iscsiadm /usr/local/lib/python3.10/dist-packages/os_brick/initiator/connectors/iscsi.py:1026}} Apr 17 15:24:16 user cinder-volume[84696]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m node -T iqn.2010-10.org.openstack:c74b7878-4936-4015-8b63-142923947310 -p 172.16.0.220:3260 --op update -n node.session.scan -v manual {{(pid=91817) execute /usr/local/lib/python3.10/dist-packages/oslo_concurrency/processutils.py:384}} Apr 17 15:24:16 user cinder-volume[84696]: INFO cinder.volume.manager [None req-ac753b94-fda1-4a0a-a55a-df6da267ee4d tempest-TaggedAttachmentsTest-1309189231 None] Created volume successfully. Apr 17 15:24:16 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-11377079-0af3-44c0-b136-161935214a6b tempest-GroupSnapshotsTest-1926061586 None] Geting {'code': 201, 'error': None, 'data': None} from PUT to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/s_a4fd2571-7db4-463a-acbe-9e0377b5e9d4/properties {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:24:16 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-11377079-0af3-44c0-b136-161935214a6b tempest-GroupSnapshotsTest-1926061586 None] volume s_a4fd2571-7db4-463a-acbe-9e0377b5e9d4 properties updated {{(pid=84696) modify_property_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:301}} Apr 17 15:24:16 user cinder-volume[84696]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m node -T iqn.2010-10.org.openstack:c74b7878-4936-4015-8b63-142923947310 -p 172.16.0.220:3260 --op update -n node.session.scan -v manual" returned: 0 in 0.025s {{(pid=91817) execute /usr/local/lib/python3.10/dist-packages/oslo_concurrency/processutils.py:422}} Apr 17 15:24:16 user cinder-volume[84696]: DEBUG oslo.privsep.daemon [-] privsep: reply[444851a8-247c-4590-9c71-13c126e1a018]: (4, ('', '')) {{(pid=91817) _call_back /usr/local/lib/python3.10/dist-packages/oslo_privsep/daemon.py:501}} Apr 17 15:24:16 user cinder-volume[84696]: DEBUG os_brick.initiator.connectors.iscsi [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] iscsiadm ('--op', 'update', '-n', 'node.session.scan', '-v', 'manual'): stdout= stderr= {{(pid=84696) _run_iscsiadm /usr/local/lib/python3.10/dist-packages/os_brick/initiator/connectors/iscsi.py:1026}} Apr 17 15:24:16 user cinder-volume[84696]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m node -T iqn.2010-10.org.openstack:c74b7878-4936-4015-8b63-142923947310 -p 172.16.0.220:3260 --op update -n node.session.auth.authmethod -v CHAP {{(pid=91817) execute /usr/local/lib/python3.10/dist-packages/oslo_concurrency/processutils.py:384}} Apr 17 15:24:16 user cinder-volume[84696]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m node -T iqn.2010-10.org.openstack:c74b7878-4936-4015-8b63-142923947310 -p 172.16.0.220:3260 --op update -n node.session.auth.authmethod -v CHAP" returned: 0 in 0.024s {{(pid=91817) execute /usr/local/lib/python3.10/dist-packages/oslo_concurrency/processutils.py:422}} Apr 17 15:24:16 user cinder-volume[84696]: DEBUG oslo.privsep.daemon [-] privsep: reply[1d3a24dc-3143-4237-a7cc-05fc24d3675c]: (4, ('', '')) {{(pid=91817) _call_back /usr/local/lib/python3.10/dist-packages/oslo_privsep/daemon.py:501}} Apr 17 15:24:16 user cinder-volume[84696]: DEBUG os_brick.initiator.connectors.iscsi [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] iscsiadm ('--op', 'update', '-n', 'node.session.auth.authmethod', '-v', 'CHAP'): stdout= stderr= {{(pid=84696) _run_iscsiadm /usr/local/lib/python3.10/dist-packages/os_brick/initiator/connectors/iscsi.py:1026}} Apr 17 15:24:16 user cinder-volume[84696]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m node -T iqn.2010-10.org.openstack:c74b7878-4936-4015-8b63-142923947310 -p 172.16.0.220:3260 --op update -n node.session.auth.username -v PUvXoCNO {{(pid=91817) execute /usr/local/lib/python3.10/dist-packages/oslo_concurrency/processutils.py:384}} Apr 17 15:24:16 user cinder-volume[84696]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m node -T iqn.2010-10.org.openstack:c74b7878-4936-4015-8b63-142923947310 -p 172.16.0.220:3260 --op update -n node.session.auth.username -v PUvXoCNO" returned: 0 in 0.028s {{(pid=91817) execute /usr/local/lib/python3.10/dist-packages/oslo_concurrency/processutils.py:422}} Apr 17 15:24:16 user cinder-volume[84696]: DEBUG oslo.privsep.daemon [-] privsep: reply[569e9bc6-e2cb-4cf6-9175-6619fb56f5d9]: (4, ('', '')) {{(pid=91817) _call_back /usr/local/lib/python3.10/dist-packages/oslo_privsep/daemon.py:501}} Apr 17 15:24:16 user cinder-volume[84696]: DEBUG os_brick.initiator.connectors.iscsi [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] iscsiadm ('--op', 'update', '-n', 'node.session.auth.username', '-v', 'PUvXoCNO'): stdout= stderr= {{(pid=84696) _run_iscsiadm /usr/local/lib/python3.10/dist-packages/os_brick/initiator/connectors/iscsi.py:1026}} Apr 17 15:24:16 user cinder-volume[84696]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m node -T iqn.2010-10.org.openstack:c74b7878-4936-4015-8b63-142923947310 -p 172.16.0.220:3260 --op update -n node.session.auth.password -v *** {{(pid=91817) execute /usr/local/lib/python3.10/dist-packages/oslo_concurrency/processutils.py:384}} Apr 17 15:24:16 user cinder-volume[84696]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m node -T iqn.2010-10.org.openstack:c74b7878-4936-4015-8b63-142923947310 -p 172.16.0.220:3260 --op update -n node.session.auth.password -v ***" returned: 0 in 0.024s {{(pid=91817) execute /usr/local/lib/python3.10/dist-packages/oslo_concurrency/processutils.py:422}} Apr 17 15:24:16 user cinder-volume[84696]: DEBUG oslo.privsep.daemon [-] privsep: reply[4d2433ac-6065-4e48-a32d-428ae335ef4e]: (4, ('', '')) {{(pid=91817) _call_back /usr/local/lib/python3.10/dist-packages/oslo_privsep/daemon.py:501}} Apr 17 15:24:16 user cinder-volume[84696]: DEBUG os_brick.initiator.connectors.iscsi [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] iscsiadm ('--op', 'update', '-n', 'node.session.auth.password', '-v', '***'): stdout= stderr= {{(pid=84696) _run_iscsiadm /usr/local/lib/python3.10/dist-packages/os_brick/initiator/connectors/iscsi.py:1026}} Apr 17 15:24:16 user cinder-volume[84696]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m session {{(pid=91817) execute /usr/local/lib/python3.10/dist-packages/oslo_concurrency/processutils.py:384}} Apr 17 15:24:16 user cinder-volume[84696]: INFO cinder.volume.manager [None req-11377079-0af3-44c0-b136-161935214a6b tempest-GroupSnapshotsTest-1926061586 None] group_snapshot aa6864c4-c754-46e2-a4a4-18248ad70055: created successfully Apr 17 15:24:16 user cinder-volume[84696]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m session" returned: 21 in 0.030s {{(pid=91817) execute /usr/local/lib/python3.10/dist-packages/oslo_concurrency/processutils.py:422}} Apr 17 15:24:16 user cinder-volume[84696]: DEBUG oslo.privsep.daemon [-] privsep: reply[256cb582-73cf-415d-be85-a19ea83024e4]: (4, ('', 'iscsiadm: No active sessions.\n')) {{(pid=91817) _call_back /usr/local/lib/python3.10/dist-packages/oslo_privsep/daemon.py:501}} Apr 17 15:24:16 user cinder-volume[84696]: DEBUG os_brick.initiator.connectors.iscsi [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] iscsiadm ('-m', 'session'): stdout= stderr=iscsiadm: No active sessions. Apr 17 15:24:16 user cinder-volume[84696]: {{(pid=84696) _run_iscsiadm_bare /usr/local/lib/python3.10/dist-packages/os_brick/initiator/connectors/iscsi.py:1182}} Apr 17 15:24:16 user cinder-volume[84696]: DEBUG os_brick.initiator.connectors.iscsi [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] iscsi session list stdout= stderr=iscsiadm: No active sessions. Apr 17 15:24:16 user cinder-volume[84696]: {{(pid=84696) _run_iscsi_session /usr/local/lib/python3.10/dist-packages/os_brick/initiator/connectors/iscsi.py:1171}} Apr 17 15:24:16 user cinder-volume[84696]: WARNING os_brick.initiator.connectors.iscsi [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] iscsiadm stderr output when getting sessions: iscsiadm: No active sessions. Apr 17 15:24:16 user cinder-volume[84696]: Apr 17 15:24:17 user cinder-volume[84696]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m node -T iqn.2010-10.org.openstack:c74b7878-4936-4015-8b63-142923947310 -p 172.16.0.220:3260 --login {{(pid=91817) execute /usr/local/lib/python3.10/dist-packages/oslo_concurrency/processutils.py:384}} Apr 17 15:24:17 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Geting {'code': 200, 'error': None, 'data': {'available': '113926152192', 'status': 30, 'import_status': {'import_steps': {'san_setup': True, 'vip_setup': True, 'zfs_import': True, 'mount_system_volume': True, 'nas_setup': True}, 'import_successful': True}, 'scan': None, 'encryption': {'enabled': False}, 'deduplication_rate': '1.00', 'sysvol_upgrade': 'not_needed', 'vdevs': [{'name': 'wwn-0x6000c296b793bf526a107a9d823e947a', 'iostats': {'read': '0', 'write': '0', 'chksum': '0'}, 'disks': [{'origin': 'local', 'slot': '', 'led': 'off', 'name': 'sdn', 'iostats': {'read': '0', 'write': '0', 'chksum': '0'}, 'alias': '', 'health': 'ONLINE', 'sn': '6000c296b793bf526a107a9d823e947a', 'trim_data': {'status': 'trim_unsupported', 'progress': None, 'trim_time': None}, 'path': 'pci-0000:03:00.0-scsi-0:0:14:0', 'model': 'Virtual disk', 'id': 'wwn-0x6000c296b793bf526a107a9d823e947a', 'size': 128849018880}], 'health': 'ONLINE', 'vdev_replacings': [], 'vdev_spares': [], 'type': ''}], 'id': '14181317334440137330', 'health': 'ONLINE', 'iostats': {'read': '0', 'write': '0', 'chksum': '0'}, 'operation': 'none', 'size': '127775277056', 'autotrim': False, 'name': 'Pool-0'}} from GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:24:17 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Total capacity: 119, Free 106. {{(pid=84696) _update_volume_stats /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:757}} Apr 17 15:24:17 user cinder-volume[84696]: DEBUG cinder.manager [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Notifying Schedulers of capabilities ... {{(pid=84696) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 17 15:24:17 user cinder-volume[84696]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m node -T iqn.2010-10.org.openstack:c74b7878-4936-4015-8b63-142923947310 -p 172.16.0.220:3260 --login" returned: 0 in 0.173s {{(pid=91817) execute /usr/local/lib/python3.10/dist-packages/oslo_concurrency/processutils.py:422}} Apr 17 15:24:17 user cinder-volume[84696]: DEBUG oslo.privsep.daemon [-] privsep: reply[f1e5dbe4-b9ab-4dfb-8a3e-1ff361f499f5]: (4, ('Logging in to [iface: default, target: iqn.2010-10.org.openstack:c74b7878-4936-4015-8b63-142923947310, portal: 172.16.0.220,3260]\nLogin to [iface: default, target: iqn.2010-10.org.openstack:c74b7878-4936-4015-8b63-142923947310, portal: 172.16.0.220,3260] successful.\n', '')) {{(pid=91817) _call_back /usr/local/lib/python3.10/dist-packages/oslo_privsep/daemon.py:501}} Apr 17 15:24:17 user cinder-volume[84696]: DEBUG os_brick.initiator.connectors.iscsi [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] iscsiadm ('--login',): stdout=Logging in to [iface: default, target: iqn.2010-10.org.openstack:c74b7878-4936-4015-8b63-142923947310, portal: 172.16.0.220,3260] Apr 17 15:24:17 user cinder-volume[84696]: Login to [iface: default, target: iqn.2010-10.org.openstack:c74b7878-4936-4015-8b63-142923947310, portal: 172.16.0.220,3260] successful. Apr 17 15:24:17 user cinder-volume[84696]: stderr= {{(pid=84696) _run_iscsiadm /usr/local/lib/python3.10/dist-packages/os_brick/initiator/connectors/iscsi.py:1026}} Apr 17 15:24:17 user cinder-volume[84696]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m node -T iqn.2010-10.org.openstack:c74b7878-4936-4015-8b63-142923947310 -p 172.16.0.220:3260 --op update -n node.startup -v automatic {{(pid=91817) execute /usr/local/lib/python3.10/dist-packages/oslo_concurrency/processutils.py:384}} Apr 17 15:24:17 user cinder-volume[84696]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m node -T iqn.2010-10.org.openstack:c74b7878-4936-4015-8b63-142923947310 -p 172.16.0.220:3260 --op update -n node.startup -v automatic" returned: 0 in 0.027s {{(pid=91817) execute /usr/local/lib/python3.10/dist-packages/oslo_concurrency/processutils.py:422}} Apr 17 15:24:17 user cinder-volume[84696]: DEBUG oslo.privsep.daemon [-] privsep: reply[4040a256-a948-4ab8-9ba2-267ef7610561]: (4, ('', '')) {{(pid=91817) _call_back /usr/local/lib/python3.10/dist-packages/oslo_privsep/daemon.py:501}} Apr 17 15:24:17 user cinder-volume[84696]: DEBUG os_brick.initiator.connectors.iscsi [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] iscsiadm ('--op', 'update', '-n', 'node.startup', '-v', 'automatic'): stdout= stderr= {{(pid=84696) _run_iscsiadm /usr/local/lib/python3.10/dist-packages/os_brick/initiator/connectors/iscsi.py:1026}} Apr 17 15:24:17 user cinder-volume[84696]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m session {{(pid=91817) execute /usr/local/lib/python3.10/dist-packages/oslo_concurrency/processutils.py:384}} Apr 17 15:24:17 user cinder-volume[84696]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m session" returned: 0 in 0.031s {{(pid=91817) execute /usr/local/lib/python3.10/dist-packages/oslo_concurrency/processutils.py:422}} Apr 17 15:24:17 user cinder-volume[84696]: DEBUG oslo.privsep.daemon [-] privsep: reply[3e294c24-01a0-4a21-98b9-4f2efb94e13e]: (4, ('tcp: [1] 172.16.0.220:3260,1 iqn.2010-10.org.openstack:c74b7878-4936-4015-8b63-142923947310 (non-flash)\n', '')) {{(pid=91817) _call_back /usr/local/lib/python3.10/dist-packages/oslo_privsep/daemon.py:501}} Apr 17 15:24:17 user cinder-volume[84696]: DEBUG os_brick.initiator.connectors.iscsi [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] iscsiadm ('-m', 'session'): stdout=tcp: [1] 172.16.0.220:3260,1 iqn.2010-10.org.openstack:c74b7878-4936-4015-8b63-142923947310 (non-flash) Apr 17 15:24:17 user cinder-volume[84696]: stderr= {{(pid=84696) _run_iscsiadm_bare /usr/local/lib/python3.10/dist-packages/os_brick/initiator/connectors/iscsi.py:1182}} Apr 17 15:24:17 user cinder-volume[84696]: DEBUG os_brick.initiator.connectors.iscsi [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] iscsi session list stdout=tcp: [1] 172.16.0.220:3260,1 iqn.2010-10.org.openstack:c74b7878-4936-4015-8b63-142923947310 (non-flash) Apr 17 15:24:17 user cinder-volume[84696]: stderr= {{(pid=84696) _run_iscsi_session /usr/local/lib/python3.10/dist-packages/os_brick/initiator/connectors/iscsi.py:1171}} Apr 17 15:24:17 user cinder-volume[84696]: DEBUG os_brick.initiator.connectors.base [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] Lock "connect_to_iscsi_portal-172.16.0.220:3260-iqn.2010-10.org.openstack:c74b7878-4936-4015-8b63-142923947310" "released" by "os_brick.initiator.connectors.iscsi.ISCSIConnector._connect_to_iscsi_portal_unsafe" :: held 0.530s {{(pid=84696) inner /usr/local/lib/python3.10/dist-packages/os_brick/initiator/connectors/base.py:87}} Apr 17 15:24:17 user cinder-volume[84696]: DEBUG os_brick.initiator.connectors.iscsi [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] Connected to 172.16.0.220:3260 {{(pid=84696) _connect_vol /usr/local/lib/python3.10/dist-packages/os_brick/initiator/connectors/iscsi.py:633}} Apr 17 15:24:17 user cinder-volume[84696]: DEBUG os_brick.initiator.linuxscsi [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] HCTL ('33', '-', '-', 0) found on session 1 with lun 0 {{(pid=84696) get_hctl /usr/local/lib/python3.10/dist-packages/os_brick/initiator/linuxscsi.py:694}} Apr 17 15:24:17 user cinder-volume[84696]: DEBUG os_brick.initiator.linuxscsi [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] Scanning host 33 c: -, t: -, l: 0) {{(pid=84696) scan_iscsi /usr/local/lib/python3.10/dist-packages/os_brick/initiator/linuxscsi.py:719}} Apr 17 15:24:17 user cinder-volume[84696]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): tee -a /sys/class/scsi_host/host33/scan {{(pid=91817) execute /usr/local/lib/python3.10/dist-packages/oslo_concurrency/processutils.py:384}} Apr 17 15:24:17 user cinder-volume[84696]: DEBUG oslo_concurrency.processutils [-] CMD "tee -a /sys/class/scsi_host/host33/scan" returned: 0 in 0.028s {{(pid=91817) execute /usr/local/lib/python3.10/dist-packages/oslo_concurrency/processutils.py:422}} Apr 17 15:24:17 user cinder-volume[84696]: DEBUG oslo.privsep.daemon [-] privsep: reply[41d979df-649c-4d61-806b-475fcee627bb]: (4, ('- - 0', '')) {{(pid=91817) _call_back /usr/local/lib/python3.10/dist-packages/oslo_privsep/daemon.py:501}} Apr 17 15:24:17 user cinder-volume[84696]: DEBUG os_brick.initiator.linuxscsi [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] Searching for a device in session 1 and hctl ['33', '*', '*', 0] yield: sdb {{(pid=84696) device_name_by_hctl /usr/local/lib/python3.10/dist-packages/os_brick/initiator/linuxscsi.py:713}} Apr 17 15:24:17 user cinder-volume[84696]: DEBUG os_brick.initiator.connectors.iscsi [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] Connected to sdb using {'target_iqn': 'iqn.2010-10.org.openstack:c74b7878-4936-4015-8b63-142923947310', 'target_portal': '172.16.0.220:3260', 'target_discovered': False, 'auth_method': 'CHAP', 'auth_username': 'PUvXoCNO', 'auth_password': '***', 'target_lun': False} {{(pid=84696) _connect_vol /usr/local/lib/python3.10/dist-packages/os_brick/initiator/connectors/iscsi.py:661}} Apr 17 15:24:17 user cinder-volume[84696]: INFO cinder.volume.manager [None req-93e81254-daef-413d-949e-6eef52a40357 tempest-GroupSnapshotsTest-1926061586 None] group_snapshot aa6864c4-c754-46e2-a4a4-18248ad70055: deleting Apr 17 15:24:17 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-93e81254-daef-413d-949e-6eef52a40357 tempest-GroupSnapshotsTest-1926061586 None] group_snapshot aa6864c4-c754-46e2-a4a4-18248ad70055: deleting {{(pid=84696) delete_group_snapshot /opt/stack/cinder/cinder/volume/manager.py:4304}} Apr 17 15:24:17 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-93e81254-daef-413d-949e-6eef52a40357 tempest-GroupSnapshotsTest-1926061586 None] deleating snapshot s_8a4c296c-977d-4a9d-a08a-e3b2a1b9a8d4. {{(pid=84696) delete_snapshot /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:649}} Apr 17 15:24:17 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-93e81254-daef-413d-949e-6eef52a40357 tempest-GroupSnapshotsTest-1926061586 None] get snapshots for volume s_8a4c296c-977d-4a9d-a08a-e3b2a1b9a8d4 {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 15:24:17 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-93e81254-daef-413d-949e-6eef52a40357 tempest-GroupSnapshotsTest-1926061586 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/s_8a4c296c-977d-4a9d-a08a-e3b2a1b9a8d4/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:24:17 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-93e81254-daef-413d-949e-6eef52a40357 tempest-GroupSnapshotsTest-1926061586 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/s_8a4c296c-977d-4a9d-a08a-e3b2a1b9a8d4/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:24:17 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-e933b70a-f56c-4690-8a4f-79fe3eb067fd tempest-VolumesTransfersTest-276179586 None] Geting {'code': 201, 'error': None, 'data': {'full_name': 'Pool-0/v_786b4c79-f126-4791-8350-4fed7646bd3b', 'is_clone': False, 'name': 'v_786b4c79-f126-4791-8350-4fed7646bd3b'}} from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:24:17 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e933b70a-f56c-4690-8a4f-79fe3eb067fd tempest-VolumesTransfersTest-276179586 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (7c014c73-21f1-4867-8527-d3d623277904) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'volume_name': 'volume-786b4c79-f126-4791-8350-4fed7646bd3b', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:24:17 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e933b70a-f56c-4690-8a4f-79fe3eb067fd tempest-VolumesTransfersTest-276179586 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeOnFinishTask;volume:create, create.end' (585d94c8-f4b1-4adf-8330-c0b2e6e99ad9) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:24:17 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-e933b70a-f56c-4690-8a4f-79fe3eb067fd tempest-VolumesTransfersTest-276179586 None] Volume volume-786b4c79-f126-4791-8350-4fed7646bd3b (786b4c79-f126-4791-8350-4fed7646bd3b): created successfully Apr 17 15:24:17 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e933b70a-f56c-4690-8a4f-79fe3eb067fd tempest-VolumesTransfersTest-276179586 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeOnFinishTask;volume:create, create.end' (585d94c8-f4b1-4adf-8330-c0b2e6e99ad9) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:24:17 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e933b70a-f56c-4690-8a4f-79fe3eb067fd tempest-VolumesTransfersTest-276179586 None] Flow 'volume_create_manager' (19a6193b-3c6c-4d5c-b9ac-15272d70be40) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:24:18 user cinder-volume[84696]: INFO cinder.volume.manager [None req-e933b70a-f56c-4690-8a4f-79fe3eb067fd tempest-VolumesTransfersTest-276179586 None] Created volume successfully. Apr 17 15:24:18 user cinder-volume[84696]: DEBUG os_brick.initiator.connectors.base [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] Lock "connect_volume" "released" by "os_brick.initiator.connectors.iscsi.ISCSIConnector.connect_volume" :: held 1.598s {{(pid=84696) inner /usr/local/lib/python3.10/dist-packages/os_brick/initiator/connectors/base.py:87}} Apr 17 15:24:18 user cinder-volume[84696]: DEBUG os_brick.initiator.connectors.iscsi [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] <== connect_volume: return (1601ms) {'type': 'block', 'scsi_wwn': '26132396565663337', 'path': '/dev/sdb'} {{(pid=84696) trace_logging_wrapper /usr/local/lib/python3.10/dist-packages/os_brick/utils.py:202}} Apr 17 15:24:18 user cinder-volume[84696]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): dd if=/dev/sdb of=/dev/null count=1 {{(pid=91817) execute /usr/local/lib/python3.10/dist-packages/oslo_concurrency/processutils.py:384}} Apr 17 15:24:18 user cinder-volume[84696]: DEBUG oslo_concurrency.processutils [-] CMD "dd if=/dev/sdb of=/dev/null count=1" returned: 0 in 0.035s {{(pid=91817) execute /usr/local/lib/python3.10/dist-packages/oslo_concurrency/processutils.py:422}} Apr 17 15:24:18 user cinder-volume[84696]: DEBUG oslo.privsep.daemon [-] privsep: reply[02c06ea5-1d82-4b90-a0e4-4660f7d4e326]: (4, ('', '1+0 records in\n1+0 records out\n512 bytes copied, 0.0113629 s, 45.1 kB/s\n')) {{(pid=91817) _call_back /usr/local/lib/python3.10/dist-packages/oslo_privsep/daemon.py:501}} Apr 17 15:24:18 user cinder-volume[84696]: DEBUG oslo_concurrency.processutils [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] Running cmd (subprocess): /usr/bin/python3.10 -m oslo_concurrency.prlimit --as=1073741824 --cpu=60 -- sudo cinder-rootwrap /etc/cinder/rootwrap.conf env LC_ALL=C qemu-img info --output=json /opt/stack/data/cinder/conversion/image_download_b3f0dec1-7758-44a8-82e0-76ada8837428_a5cfdi99 {{(pid=84696) execute /usr/local/lib/python3.10/dist-packages/oslo_concurrency/processutils.py:384}} Apr 17 15:24:18 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-93e81254-daef-413d-949e-6eef52a40357 tempest-GroupSnapshotsTest-1926061586 None] Geting {'code': 200, 'error': None, 'data': {'results': 0, 'entries': []}} from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/s_8a4c296c-977d-4a9d-a08a-e3b2a1b9a8d4/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:24:18 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-93e81254-daef-413d-949e-6eef52a40357 tempest-GroupSnapshotsTest-1926061586 None] get volume s_8a4c296c-977d-4a9d-a08a-e3b2a1b9a8d4 info {{(pid=84696) get_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:239}} Apr 17 15:24:18 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-93e81254-daef-413d-949e-6eef52a40357 tempest-GroupSnapshotsTest-1926061586 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/s_8a4c296c-977d-4a9d-a08a-e3b2a1b9a8d4 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:24:18 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-93e81254-daef-413d-949e-6eef52a40357 tempest-GroupSnapshotsTest-1926061586 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/s_8a4c296c-977d-4a9d-a08a-e3b2a1b9a8d4 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:24:18 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-93e81254-daef-413d-949e-6eef52a40357 tempest-GroupSnapshotsTest-1926061586 None] Geting {'code': 200, 'error': None, 'data': {'origin': 'Pool-0/v_7249f8e4-ed4b-4689-aba3-2766c924f42c@s_8a4c296c-977d-4a9d-a08a-e3b2a1b9a8d4', 'relatime': None, 'acltype': None, 'vscan': None, 'full_name': 'Pool-0/s_8a4c296c-977d-4a9d-a08a-e3b2a1b9a8d4', 'userrefs': None, 'primarycache': 'all', 'logbias': 'latency', 'creation': '1681746441', 'sync': 'always', 'is_clone': True, 'dedup': 'off', 'sharenfs': None, 'receive_resume_token': None, 'volsize': '1073741824', 'referenced': '57344', 'sharesmb': None, 'createtxg': '187860', 'reservation': '0', 'scontext': None, 'mountpoint': None, 'casesensitivity': None, 'guid': '7347279528997992856', 'usedbyrefreservation': '0', 'dnodesize': None, 'written': '0', 'logicalused': '0', 'compressratio': '1.00', 'rootcontext': 'none', 'default_scsi_id': '87b142c57b3c093f', 'type': 'volume', 'compression': 'lz4', 'snapdir': None, 'overlay': None, 'encryption': 'off', 'xattr': None, 'volmode': 'default', 'copies': '1', 'snapshot_limit': '18446744073709551615', 'aclinherit': None, 'defcontext': 'none', 'readonly': 'on', 'version': None, 'recordsize': None, 'filesystem_limit': None, 'mounted': None, 'mlslabel': 'none', 'secondarycache': 'all', 'refreservation': '0', 'available': '113926139904', 'used': '0', 'encryptionroot': None, 'exec': None, 'refquota': None, 'refcompressratio': '1.00', 'quota': None, 'utf8only': None, 'keylocation': 'none', 'snapdev': 'hidden', 'snapshot_count': '18446744073709551615', 'fscontext': 'none', 'clones': None, 'canmount': None, 'keystatus': None, 'atime': None, 'usedbysnapshots': '0', 'normalization': None, 'usedbychildren': '0', 'volblocksize': '131072', 'usedbydataset': '0', 'objsetid': '121543', 'name': 's_8a4c296c-977d-4a9d-a08a-e3b2a1b9a8d4', 'defer_destroy': None, 'pbkdf2iters': '0', 'checksum': 'on', 'redundant_metadata': 'all', 'filesystem_count': None, 'devices': None, 'keyformat': 'none', 'setuid': None, 'logicalreferenced': '28672', 'context': 'none', 'zoned': None, 'nbmand': None}} from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/s_8a4c296c-977d-4a9d-a08a-e3b2a1b9a8d4 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:24:18 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-93e81254-daef-413d-949e-6eef52a40357 tempest-GroupSnapshotsTest-1926061586 None] delete snapshot s_8a4c296c-977d-4a9d-a08a-e3b2a1b9a8d4 of volume v_7249f8e4-ed4b-4689-aba3-2766c924f42c {{(pid=84696) delete_snapshot /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:751}} Apr 17 15:24:18 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-93e81254-daef-413d-949e-6eef52a40357 tempest-GroupSnapshotsTest-1926061586 None] Sending pool request DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_7249f8e4-ed4b-4689-aba3-2766c924f42c/snapshots/s_8a4c296c-977d-4a9d-a08a-e3b2a1b9a8d4 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:24:18 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-93e81254-daef-413d-949e-6eef52a40357 tempest-GroupSnapshotsTest-1926061586 None] Sending DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_7249f8e4-ed4b-4689-aba3-2766c924f42c/snapshots/s_8a4c296c-977d-4a9d-a08a-e3b2a1b9a8d4 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:24:18 user sudo[92031]: stack : PWD=/ ; USER=root ; COMMAND=/usr/local/bin/cinder-rootwrap /etc/cinder/rootwrap.conf env LC_ALL=C qemu-img info --output=json /opt/stack/data/cinder/conversion/image_download_b3f0dec1-7758-44a8-82e0-76ada8837428_a5cfdi99 Apr 17 15:24:18 user sudo[92031]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=1001) Apr 17 15:24:19 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-b5809a81-b0b1-43e3-93ac-27311b059c17 tempest-VolumesGetTest-1961901732 None] Geting {'code': 204} from DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_f93cf40a-0d85-4277-91a8-d85037f79e7a {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:24:19 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-b5809a81-b0b1-43e3-93ac-27311b059c17 tempest-VolumesGetTest-1961901732 None] volume v_f93cf40a-0d85-4277-91a8-d85037f79e7a deleted {{(pid=84696) delete_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:346}} Apr 17 15:24:19 user cinder-volume[84696]: DEBUG cinder.quota [None req-b5809a81-b0b1-43e3-93ac-27311b059c17 tempest-VolumesGetTest-1961901732 None] Created reservations ['7c036c45-8aae-43e0-9250-c5f814c818bd', '861ef17c-e512-4348-bf19-10b6692d9b7d', 'b331b223-9fe8-411d-be79-92d324bd51df', 'f33840fc-9568-4051-82dd-bbff48dc0fda'] {{(pid=84696) reserve /opt/stack/cinder/cinder/quota.py:755}} Apr 17 15:24:19 user cinder-volume[84696]: INFO cinder.volume.manager [None req-0b5ed337-fda8-454e-8612-12dea10e7643 tempest-VolumesTransfersTest-54717837 None] Transfer volume completed successfully. Apr 17 15:24:19 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-b5809a81-b0b1-43e3-93ac-27311b059c17 tempest-VolumesGetTest-1961901732 None] Updating volume stats {{(pid=84696) _update_volume_stats /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:724}} Apr 17 15:24:19 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-b5809a81-b0b1-43e3-93ac-27311b059c17 tempest-VolumesGetTest-1961901732 None] Get pool Pool-0 fsprops {{(pid=84696) get_pool_stats /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:894}} Apr 17 15:24:19 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-b5809a81-b0b1-43e3-93ac-27311b059c17 tempest-VolumesGetTest-1961901732 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:24:19 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-b5809a81-b0b1-43e3-93ac-27311b059c17 tempest-VolumesGetTest-1961901732 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:24:19 user sudo[92031]: pam_unix(sudo:session): session closed for user root Apr 17 15:24:19 user cinder-volume[84696]: DEBUG oslo_concurrency.processutils [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] CMD "/usr/bin/python3.10 -m oslo_concurrency.prlimit --as=1073741824 --cpu=60 -- sudo cinder-rootwrap /etc/cinder/rootwrap.conf env LC_ALL=C qemu-img info --output=json /opt/stack/data/cinder/conversion/image_download_b3f0dec1-7758-44a8-82e0-76ada8837428_a5cfdi99" returned: 0 in 1.114s {{(pid=84696) execute /usr/local/lib/python3.10/dist-packages/oslo_concurrency/processutils.py:422}} Apr 17 15:24:19 user cinder-volume[84696]: DEBUG oslo_concurrency.processutils [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] Running cmd (subprocess): /usr/bin/python3.10 -m oslo_concurrency.prlimit --as=1073741824 --cpu=60 -- sudo cinder-rootwrap /etc/cinder/rootwrap.conf env LC_ALL=C qemu-img info --output=json /opt/stack/data/cinder/conversion/image_fetch_b3f0dec1-7758-44a8-82e0-76ada8837428_zt3n6g1xuser@joviandss-2 {{(pid=84696) execute /usr/local/lib/python3.10/dist-packages/oslo_concurrency/processutils.py:384}} Apr 17 15:24:19 user sudo[92050]: stack : PWD=/ ; USER=root ; COMMAND=/usr/local/bin/cinder-rootwrap /etc/cinder/rootwrap.conf env LC_ALL=C qemu-img info --output=json /opt/stack/data/cinder/conversion/image_fetch_b3f0dec1-7758-44a8-82e0-76ada8837428_zt3n6g1xuser@joviandss-2 Apr 17 15:24:19 user sudo[92050]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=1001) Apr 17 15:24:19 user cinder-volume[84696]: DEBUG cinder.coordination [None req-ce906aff-8139-4f35-93dd-d0256428c52f tempest-VolumesGetTest-1961901732 None] Acquiring lock "cinder-f088992b-786a-4641-8bcc-5cb7c819d313-delete_volume" by "delete_volume" {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:192}} Apr 17 15:24:19 user cinder-volume[84696]: DEBUG cinder.coordination [None req-ce906aff-8139-4f35-93dd-d0256428c52f tempest-VolumesGetTest-1961901732 None] Lock "cinder-f088992b-786a-4641-8bcc-5cb7c819d313-delete_volume" acquired by "delete_volume" :: waited 0.035s {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:196}} Apr 17 15:24:19 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-ce906aff-8139-4f35-93dd-d0256428c52f tempest-VolumesGetTest-1961901732 None] remove_export for volume: f088992b-786a-4641-8bcc-5cb7c819d313. {{(pid=84696) remove_export /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:718}} Apr 17 15:24:19 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-ce906aff-8139-4f35-93dd-d0256428c52f tempest-VolumesGetTest-1961901732 None] remove export {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:912}} Apr 17 15:24:19 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-ce906aff-8139-4f35-93dd-d0256428c52f tempest-VolumesGetTest-1961901732 None] detach volume:Volume(_name_id=None,admin_metadata={},attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:24:08Z,deleted=False,deleted_at=None,display_description='This is the new description of volume',display_name='tempest-VolumesGetTest-new-Volume-1517573418',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=f088992b-786a-4641-8bcc-5cb7c819d313,launched_at=2023-04-17T15:24:10Z,metadata={Type='Test'},migration_status=None,multiattach=False,previous_status=None,project_id='a65d14b0a2894a6f90506d86feaefd15',provider_auth='CHAP AgLoKHjM HlWR4VEIrXocSv',provider_geometry=None,provider_id=None,provider_location='172.16.0.220:3260,1 iqn.2010-10.org.openstack:f088992b-786a-4641-8bcc-5cb7c819d313 0',replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:24:08Z,service_uuid='e628352e-c4cf-40eb-8389-cce40663026a',shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='deleting',terminated_at=2023-04-17T15:24:20Z,updated_at=2023-04-17T15:24:20Z,use_quota=True,user_id='6ffa3051570a4721a48a0171b3cc4d16',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c) from target:iqn.2010-10.org.openstack:f088992b-786a-4641-8bcc-5cb7c819d313. {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:913}} Apr 17 15:24:19 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-ce906aff-8139-4f35-93dd-d0256428c52f tempest-VolumesGetTest-1961901732 None] detach volume v_f088992b-786a-4641-8bcc-5cb7c819d313 from target iqn.2010-10.org.openstack:f088992b-786a-4641-8bcc-5cb7c819d313 {{(pid=84696) detach_target_vol /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:606}} Apr 17 15:24:19 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-ce906aff-8139-4f35-93dd-d0256428c52f tempest-VolumesGetTest-1961901732 None] Sending pool request DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:f088992b-786a-4641-8bcc-5cb7c819d313/luns/v_f088992b-786a-4641-8bcc-5cb7c819d313 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:24:19 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-ce906aff-8139-4f35-93dd-d0256428c52f tempest-VolumesGetTest-1961901732 None] Sending DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:f088992b-786a-4641-8bcc-5cb7c819d313/luns/v_f088992b-786a-4641-8bcc-5cb7c819d313 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:24:19 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-ce906aff-8139-4f35-93dd-d0256428c52f tempest-VolumesGetTest-1961901732 None] Geting {'code': 404, 'error': {'class': 'opene.exceptions.ItemNotFoundError', 'message': 'Target iqn.2010-10.org.openstack:f088992b-786a-4641-8bcc-5cb7c819d313 not exists.', 'url': 'http://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:f088992b-786a-4641-8bcc-5cb7c819d313/luns/v_f088992b-786a-4641-8bcc-5cb7c819d313'}, 'data': None} from DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:f088992b-786a-4641-8bcc-5cb7c819d313/luns/v_f088992b-786a-4641-8bcc-5cb7c819d313 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:24:19 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-ce906aff-8139-4f35-93dd-d0256428c52f tempest-VolumesGetTest-1961901732 None] failed to remove resource iqn.2010-10.org.openstack:f088992b-786a-4641-8bcc-5cb7c819d313 because of JDSS resource v_f088992b-786a-4641-8bcc-5cb7c819d313 DNE. {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:920}} Apr 17 15:24:19 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-ce906aff-8139-4f35-93dd-d0256428c52f tempest-VolumesGetTest-1961901732 None] delete target: iqn.2010-10.org.openstack:f088992b-786a-4641-8bcc-5cb7c819d313 {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:930}} Apr 17 15:24:19 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-ce906aff-8139-4f35-93dd-d0256428c52f tempest-VolumesGetTest-1961901732 None] delete target iqn.2010-10.org.openstack:f088992b-786a-4641-8bcc-5cb7c819d313 {{(pid=84696) delete_target /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:442}} Apr 17 15:24:19 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-ce906aff-8139-4f35-93dd-d0256428c52f tempest-VolumesGetTest-1961901732 None] Sending pool request DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:f088992b-786a-4641-8bcc-5cb7c819d313 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:24:19 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-ce906aff-8139-4f35-93dd-d0256428c52f tempest-VolumesGetTest-1961901732 None] Sending DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:f088992b-786a-4641-8bcc-5cb7c819d313 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:24:19 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-ce906aff-8139-4f35-93dd-d0256428c52f tempest-VolumesGetTest-1961901732 None] Geting {'code': 404, 'error': {'class': 'opene.exceptions.ItemNotFoundError', 'message': 'Target iqn.2010-10.org.openstack:f088992b-786a-4641-8bcc-5cb7c819d313 not exists.', 'url': 'http://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:f088992b-786a-4641-8bcc-5cb7c819d313'}, 'data': None} from DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:f088992b-786a-4641-8bcc-5cb7c819d313 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:24:19 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-ce906aff-8139-4f35-93dd-d0256428c52f tempest-VolumesGetTest-1961901732 None] failed to remove resource iqn.2010-10.org.openstack:f088992b-786a-4641-8bcc-5cb7c819d313 because of JDSS resource iqn.2010-10.org.openstack:f088992b-786a-4641-8bcc-5cb7c819d313 DNE. {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:935}} Apr 17 15:24:19 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-ce906aff-8139-4f35-93dd-d0256428c52f tempest-VolumesGetTest-1961901732 None] deleating volume v_f088992b-786a-4641-8bcc-5cb7c819d313 {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 15:24:19 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-ce906aff-8139-4f35-93dd-d0256428c52f tempest-VolumesGetTest-1961901732 None] get snapshots for volume v_f088992b-786a-4641-8bcc-5cb7c819d313 {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 15:24:19 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-ce906aff-8139-4f35-93dd-d0256428c52f tempest-VolumesGetTest-1961901732 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_f088992b-786a-4641-8bcc-5cb7c819d313/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:24:19 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-ce906aff-8139-4f35-93dd-d0256428c52f tempest-VolumesGetTest-1961901732 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_f088992b-786a-4641-8bcc-5cb7c819d313/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:24:20 user cinder-volume[84696]: DEBUG cinder.coordination [None req-f7fe6c6b-6bf3-4a20-9298-2c754b555694 tempest-VolumesTransfersTest-450487945 None] Acquiring lock "cinder-786b4c79-f126-4791-8350-4fed7646bd3b-delete_volume" by "delete_volume" {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:192}} Apr 17 15:24:20 user cinder-volume[84696]: DEBUG cinder.coordination [None req-f7fe6c6b-6bf3-4a20-9298-2c754b555694 tempest-VolumesTransfersTest-450487945 None] Lock "cinder-786b4c79-f126-4791-8350-4fed7646bd3b-delete_volume" acquired by "delete_volume" :: waited 0.016s {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:196}} Apr 17 15:24:20 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-f7fe6c6b-6bf3-4a20-9298-2c754b555694 tempest-VolumesTransfersTest-450487945 None] remove_export for volume: 786b4c79-f126-4791-8350-4fed7646bd3b. {{(pid=84696) remove_export /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:718}} Apr 17 15:24:20 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-f7fe6c6b-6bf3-4a20-9298-2c754b555694 tempest-VolumesTransfersTest-450487945 None] remove export {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:912}} Apr 17 15:24:20 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-f7fe6c6b-6bf3-4a20-9298-2c754b555694 tempest-VolumesTransfersTest-450487945 None] detach volume:Volume(_name_id=None,admin_metadata={},attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:24:16Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesTransfersTest-Volume-604455400',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=786b4c79-f126-4791-8350-4fed7646bd3b,launched_at=2023-04-17T15:24:18Z,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='bcc1b3b77ad342e484eb4c6ad2021a44',provider_auth='CHAP hRITLxVQ 5gYfSyGAhQ1VUP',provider_geometry=None,provider_id=None,provider_location='172.16.0.220:3260,1 iqn.2010-10.org.openstack:786b4c79-f126-4791-8350-4fed7646bd3b 0',replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:24:16Z,service_uuid='e628352e-c4cf-40eb-8389-cce40663026a',shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='deleting',terminated_at=2023-04-17T15:24:20Z,updated_at=2023-04-17T15:24:20Z,use_quota=True,user_id='b6d460eede6242678c4e4274200021f7',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c) from target:iqn.2010-10.org.openstack:786b4c79-f126-4791-8350-4fed7646bd3b. {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:913}} Apr 17 15:24:20 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-f7fe6c6b-6bf3-4a20-9298-2c754b555694 tempest-VolumesTransfersTest-450487945 None] detach volume v_786b4c79-f126-4791-8350-4fed7646bd3b from target iqn.2010-10.org.openstack:786b4c79-f126-4791-8350-4fed7646bd3b {{(pid=84696) detach_target_vol /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:606}} Apr 17 15:24:20 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-f7fe6c6b-6bf3-4a20-9298-2c754b555694 tempest-VolumesTransfersTest-450487945 None] Sending pool request DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:786b4c79-f126-4791-8350-4fed7646bd3b/luns/v_786b4c79-f126-4791-8350-4fed7646bd3b {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:24:20 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-f7fe6c6b-6bf3-4a20-9298-2c754b555694 tempest-VolumesTransfersTest-450487945 None] Sending DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:786b4c79-f126-4791-8350-4fed7646bd3b/luns/v_786b4c79-f126-4791-8350-4fed7646bd3b {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:24:20 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-f7fe6c6b-6bf3-4a20-9298-2c754b555694 tempest-VolumesTransfersTest-450487945 None] Geting {'code': 404, 'error': {'class': 'opene.exceptions.ItemNotFoundError', 'message': 'Target iqn.2010-10.org.openstack:786b4c79-f126-4791-8350-4fed7646bd3b not exists.', 'url': 'http://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:786b4c79-f126-4791-8350-4fed7646bd3b/luns/v_786b4c79-f126-4791-8350-4fed7646bd3b'}, 'data': None} from DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:786b4c79-f126-4791-8350-4fed7646bd3b/luns/v_786b4c79-f126-4791-8350-4fed7646bd3b {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:24:20 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-f7fe6c6b-6bf3-4a20-9298-2c754b555694 tempest-VolumesTransfersTest-450487945 None] failed to remove resource iqn.2010-10.org.openstack:786b4c79-f126-4791-8350-4fed7646bd3b because of JDSS resource v_786b4c79-f126-4791-8350-4fed7646bd3b DNE. {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:920}} Apr 17 15:24:20 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-f7fe6c6b-6bf3-4a20-9298-2c754b555694 tempest-VolumesTransfersTest-450487945 None] delete target: iqn.2010-10.org.openstack:786b4c79-f126-4791-8350-4fed7646bd3b {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:930}} Apr 17 15:24:20 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-f7fe6c6b-6bf3-4a20-9298-2c754b555694 tempest-VolumesTransfersTest-450487945 None] delete target iqn.2010-10.org.openstack:786b4c79-f126-4791-8350-4fed7646bd3b {{(pid=84696) delete_target /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:442}} Apr 17 15:24:20 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-f7fe6c6b-6bf3-4a20-9298-2c754b555694 tempest-VolumesTransfersTest-450487945 None] Sending pool request DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:786b4c79-f126-4791-8350-4fed7646bd3b {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:24:20 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-f7fe6c6b-6bf3-4a20-9298-2c754b555694 tempest-VolumesTransfersTest-450487945 None] Sending DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:786b4c79-f126-4791-8350-4fed7646bd3b {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:24:20 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-f7fe6c6b-6bf3-4a20-9298-2c754b555694 tempest-VolumesTransfersTest-450487945 None] Geting {'code': 404, 'error': {'class': 'opene.exceptions.ItemNotFoundError', 'message': 'Target iqn.2010-10.org.openstack:786b4c79-f126-4791-8350-4fed7646bd3b not exists.', 'url': 'http://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:786b4c79-f126-4791-8350-4fed7646bd3b'}, 'data': None} from DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:786b4c79-f126-4791-8350-4fed7646bd3b {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:24:20 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-f7fe6c6b-6bf3-4a20-9298-2c754b555694 tempest-VolumesTransfersTest-450487945 None] failed to remove resource iqn.2010-10.org.openstack:786b4c79-f126-4791-8350-4fed7646bd3b because of JDSS resource iqn.2010-10.org.openstack:786b4c79-f126-4791-8350-4fed7646bd3b DNE. {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:935}} Apr 17 15:24:20 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-f7fe6c6b-6bf3-4a20-9298-2c754b555694 tempest-VolumesTransfersTest-450487945 None] deleating volume v_786b4c79-f126-4791-8350-4fed7646bd3b {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 15:24:20 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-f7fe6c6b-6bf3-4a20-9298-2c754b555694 tempest-VolumesTransfersTest-450487945 None] get snapshots for volume v_786b4c79-f126-4791-8350-4fed7646bd3b {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 15:24:20 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-f7fe6c6b-6bf3-4a20-9298-2c754b555694 tempest-VolumesTransfersTest-450487945 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_786b4c79-f126-4791-8350-4fed7646bd3b/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:24:20 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-f7fe6c6b-6bf3-4a20-9298-2c754b555694 tempest-VolumesTransfersTest-450487945 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_786b4c79-f126-4791-8350-4fed7646bd3b/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:24:20 user sudo[92050]: pam_unix(sudo:session): session closed for user root Apr 17 15:24:20 user cinder-volume[84696]: DEBUG oslo_concurrency.processutils [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] CMD "/usr/bin/python3.10 -m oslo_concurrency.prlimit --as=1073741824 --cpu=60 -- sudo cinder-rootwrap /etc/cinder/rootwrap.conf env LC_ALL=C qemu-img info --output=json /opt/stack/data/cinder/conversion/image_fetch_b3f0dec1-7758-44a8-82e0-76ada8837428_zt3n6g1xuser@joviandss-2" returned: 0 in 1.077s {{(pid=84696) execute /usr/local/lib/python3.10/dist-packages/oslo_concurrency/processutils.py:422}} Apr 17 15:24:20 user cinder-volume[84696]: DEBUG cinder.image.image_utils [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] b3f0dec1-7758-44a8-82e0-76ada8837428 was qcow2, converting to raw {{(pid=84696) fetch_to_volume_format /opt/stack/cinder/cinder/image/image_utils.py:976}} Apr 17 15:24:20 user cinder-volume[84696]: DEBUG oslo_concurrency.processutils [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] Running cmd (subprocess): sudo cinder-rootwrap /etc/cinder/rootwrap.conf dd count=0 if=/opt/stack/data/cinder/conversion/image_fetch_b3f0dec1-7758-44a8-82e0-76ada8837428_zt3n6g1xuser@joviandss-2 of=/dev/sdb oflag=direct {{(pid=84696) execute /usr/local/lib/python3.10/dist-packages/oslo_concurrency/processutils.py:384}} Apr 17 15:24:20 user sudo[92056]: stack : PWD=/ ; USER=root ; COMMAND=/usr/local/bin/cinder-rootwrap /etc/cinder/rootwrap.conf dd count=0 if=/opt/stack/data/cinder/conversion/image_fetch_b3f0dec1-7758-44a8-82e0-76ada8837428_zt3n6g1xuser@joviandss-2 of=/dev/sdb oflag=direct Apr 17 15:24:20 user sudo[92056]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=1001) Apr 17 15:24:20 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-b5809a81-b0b1-43e3-93ac-27311b059c17 tempest-VolumesGetTest-1961901732 None] Geting {'code': 200, 'error': None, 'data': {'available': '113926115328', 'status': 30, 'import_status': {'import_steps': {'san_setup': True, 'vip_setup': True, 'zfs_import': True, 'mount_system_volume': True, 'nas_setup': True}, 'import_successful': True}, 'scan': None, 'encryption': {'enabled': False}, 'deduplication_rate': '1.00', 'sysvol_upgrade': 'not_needed', 'vdevs': [{'name': 'wwn-0x6000c296b793bf526a107a9d823e947a', 'iostats': {'read': '0', 'write': '0', 'chksum': '0'}, 'disks': [{'origin': 'local', 'slot': '', 'led': 'off', 'name': 'sdn', 'iostats': {'read': '0', 'write': '0', 'chksum': '0'}, 'alias': '', 'health': 'ONLINE', 'sn': '6000c296b793bf526a107a9d823e947a', 'trim_data': {'status': 'trim_unsupported', 'progress': None, 'trim_time': None}, 'path': 'pci-0000:03:00.0-scsi-0:0:14:0', 'model': 'Virtual disk', 'id': 'wwn-0x6000c296b793bf526a107a9d823e947a', 'size': 128849018880}], 'health': 'ONLINE', 'vdev_replacings': [], 'vdev_spares': [], 'type': ''}], 'id': '14181317334440137330', 'health': 'ONLINE', 'iostats': {'read': '0', 'write': '0', 'chksum': '0'}, 'operation': 'none', 'size': '127775277056', 'autotrim': False, 'name': 'Pool-0'}} from GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:24:20 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-b5809a81-b0b1-43e3-93ac-27311b059c17 tempest-VolumesGetTest-1961901732 None] Total capacity: 119, Free 106. {{(pid=84696) _update_volume_stats /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:757}} Apr 17 15:24:20 user cinder-volume[84696]: DEBUG cinder.manager [None req-b5809a81-b0b1-43e3-93ac-27311b059c17 tempest-VolumesGetTest-1961901732 None] Notifying Schedulers of capabilities ... {{(pid=84696) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 17 15:24:20 user cinder-volume[84696]: INFO cinder.volume.manager [None req-b5809a81-b0b1-43e3-93ac-27311b059c17 tempest-VolumesGetTest-1961901732 None] Deleted volume successfully. Apr 17 15:24:20 user cinder-volume[84696]: DEBUG cinder.coordination [None req-b5809a81-b0b1-43e3-93ac-27311b059c17 tempest-VolumesGetTest-1961901732 None] Lock "cinder-f93cf40a-0d85-4277-91a8-d85037f79e7a-delete_volume" released by "delete_volume" :: held 5.495s {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:207}} Apr 17 15:24:20 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-ce906aff-8139-4f35-93dd-d0256428c52f tempest-VolumesGetTest-1961901732 None] Geting {'code': 200, 'error': None, 'data': {'results': 0, 'entries': []}} from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_f088992b-786a-4641-8bcc-5cb7c819d313/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:24:20 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-ce906aff-8139-4f35-93dd-d0256428c52f tempest-VolumesGetTest-1961901732 None] get volume v_f088992b-786a-4641-8bcc-5cb7c819d313 info {{(pid=84696) get_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:239}} Apr 17 15:24:20 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-ce906aff-8139-4f35-93dd-d0256428c52f tempest-VolumesGetTest-1961901732 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_f088992b-786a-4641-8bcc-5cb7c819d313 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:24:20 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-ce906aff-8139-4f35-93dd-d0256428c52f tempest-VolumesGetTest-1961901732 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_f088992b-786a-4641-8bcc-5cb7c819d313 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:24:20 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-ce906aff-8139-4f35-93dd-d0256428c52f tempest-VolumesGetTest-1961901732 None] Geting {'code': 200, 'error': None, 'data': {'origin': None, 'relatime': None, 'acltype': None, 'vscan': None, 'full_name': 'Pool-0/v_f088992b-786a-4641-8bcc-5cb7c819d313', 'userrefs': None, 'primarycache': 'all', 'logbias': 'latency', 'creation': '1681746442', 'sync': 'always', 'is_clone': False, 'dedup': 'off', 'sharenfs': None, 'receive_resume_token': None, 'volsize': '1073741824', 'referenced': '57344', 'sharesmb': None, 'createtxg': '187863', 'reservation': '0', 'scontext': None, 'mountpoint': None, 'casesensitivity': None, 'guid': '5342063421812983704', 'usedbyrefreservation': '0', 'dnodesize': None, 'written': '57344', 'logicalused': '28672', 'compressratio': '1.00', 'rootcontext': 'none', 'default_scsi_id': '3e9ab924afa6234f', 'type': 'volume', 'compression': 'lz4', 'snapdir': None, 'overlay': None, 'encryption': 'off', 'xattr': None, 'volmode': 'default', 'copies': '1', 'snapshot_limit': '18446744073709551615', 'aclinherit': None, 'defcontext': 'none', 'readonly': 'off', 'version': None, 'recordsize': None, 'filesystem_limit': None, 'mounted': None, 'mlslabel': 'none', 'secondarycache': 'all', 'refreservation': '0', 'available': '113926139904', 'san:volume_id': '3e9ab924afa6234f23b4087e9ea6bf3fe50a0145', 'encryptionroot': None, 'exec': None, 'refquota': None, 'refcompressratio': '1.00', 'quota': None, 'utf8only': None, 'keylocation': 'none', 'snapdev': 'hidden', 'snapshot_count': '18446744073709551615', 'fscontext': 'none', 'clones': None, 'canmount': None, 'keystatus': None, 'atime': None, 'usedbysnapshots': '0', 'normalization': None, 'usedbychildren': '0', 'volblocksize': '131072', 'usedbydataset': '57344', 'objsetid': '121554', 'name': 'v_f088992b-786a-4641-8bcc-5cb7c819d313', 'defer_destroy': None, 'pbkdf2iters': '0', 'checksum': 'on', 'redundant_metadata': 'all', 'filesystem_count': None, 'devices': None, 'keyformat': 'none', 'setuid': None, 'used': '57344', 'logicalreferenced': '28672', 'context': 'none', 'zoned': None, 'nbmand': None}} from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_f088992b-786a-4641-8bcc-5cb7c819d313 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:24:20 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-ce906aff-8139-4f35-93dd-d0256428c52f tempest-VolumesGetTest-1961901732 None] delete volume:v_f088992b-786a-4641-8bcc-5cb7c819d313 recursively children:{'force_umount': True} {{(pid=84696) delete_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:335}} Apr 17 15:24:20 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-ce906aff-8139-4f35-93dd-d0256428c52f tempest-VolumesGetTest-1961901732 None] Sending pool request DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_f088992b-786a-4641-8bcc-5cb7c819d313 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:24:20 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-ce906aff-8139-4f35-93dd-d0256428c52f tempest-VolumesGetTest-1961901732 None] Sending DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_f088992b-786a-4641-8bcc-5cb7c819d313 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:24:21 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-93e81254-daef-413d-949e-6eef52a40357 tempest-GroupSnapshotsTest-1926061586 None] Geting {'code': 204} from DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_7249f8e4-ed4b-4689-aba3-2766c924f42c/snapshots/s_8a4c296c-977d-4a9d-a08a-e3b2a1b9a8d4 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:24:21 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-93e81254-daef-413d-949e-6eef52a40357 tempest-GroupSnapshotsTest-1926061586 None] snapshot s_8a4c296c-977d-4a9d-a08a-e3b2a1b9a8d4 deleted {{(pid=84696) delete_snapshot /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:772}} Apr 17 15:24:21 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-93e81254-daef-413d-949e-6eef52a40357 tempest-GroupSnapshotsTest-1926061586 None] deleating snapshot s_a4fd2571-7db4-463a-acbe-9e0377b5e9d4. {{(pid=84696) delete_snapshot /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:649}} Apr 17 15:24:21 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-93e81254-daef-413d-949e-6eef52a40357 tempest-GroupSnapshotsTest-1926061586 None] get snapshots for volume s_a4fd2571-7db4-463a-acbe-9e0377b5e9d4 {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 15:24:21 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-93e81254-daef-413d-949e-6eef52a40357 tempest-GroupSnapshotsTest-1926061586 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/s_a4fd2571-7db4-463a-acbe-9e0377b5e9d4/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:24:21 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-93e81254-daef-413d-949e-6eef52a40357 tempest-GroupSnapshotsTest-1926061586 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/s_a4fd2571-7db4-463a-acbe-9e0377b5e9d4/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:24:21 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-f7fe6c6b-6bf3-4a20-9298-2c754b555694 tempest-VolumesTransfersTest-450487945 None] Geting {'code': 200, 'error': None, 'data': {'results': 0, 'entries': []}} from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_786b4c79-f126-4791-8350-4fed7646bd3b/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:24:21 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-f7fe6c6b-6bf3-4a20-9298-2c754b555694 tempest-VolumesTransfersTest-450487945 None] get volume v_786b4c79-f126-4791-8350-4fed7646bd3b info {{(pid=84696) get_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:239}} Apr 17 15:24:21 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-f7fe6c6b-6bf3-4a20-9298-2c754b555694 tempest-VolumesTransfersTest-450487945 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_786b4c79-f126-4791-8350-4fed7646bd3b {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:24:21 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-f7fe6c6b-6bf3-4a20-9298-2c754b555694 tempest-VolumesTransfersTest-450487945 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_786b4c79-f126-4791-8350-4fed7646bd3b {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:24:21 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-f7fe6c6b-6bf3-4a20-9298-2c754b555694 tempest-VolumesTransfersTest-450487945 None] Geting {'code': 200, 'error': None, 'data': {'origin': None, 'relatime': None, 'acltype': None, 'vscan': None, 'full_name': 'Pool-0/v_786b4c79-f126-4791-8350-4fed7646bd3b', 'userrefs': None, 'primarycache': 'all', 'logbias': 'latency', 'creation': '1681746450', 'sync': 'always', 'is_clone': False, 'dedup': 'off', 'sharenfs': None, 'receive_resume_token': None, 'volsize': '1073741824', 'referenced': '57344', 'sharesmb': None, 'createtxg': '187894', 'reservation': '0', 'scontext': None, 'mountpoint': None, 'casesensitivity': None, 'guid': '11121192858235241069', 'usedbyrefreservation': '0', 'dnodesize': None, 'written': '57344', 'logicalused': '28672', 'compressratio': '1.00', 'rootcontext': 'none', 'default_scsi_id': '46806e735cec5e22', 'type': 'volume', 'compression': 'lz4', 'snapdir': None, 'overlay': None, 'encryption': 'off', 'xattr': None, 'volmode': 'default', 'copies': '1', 'snapshot_limit': '18446744073709551615', 'aclinherit': None, 'defcontext': 'none', 'readonly': 'off', 'version': None, 'recordsize': None, 'filesystem_limit': None, 'mounted': None, 'mlslabel': 'none', 'secondarycache': 'all', 'refreservation': '0', 'available': '113926139904', 'san:volume_id': '46806e735cec5e229723003eb777c00fa54f6b37', 'encryptionroot': None, 'exec': None, 'refquota': None, 'refcompressratio': '1.00', 'quota': None, 'utf8only': None, 'keylocation': 'none', 'snapdev': 'hidden', 'snapshot_count': '18446744073709551615', 'fscontext': 'none', 'clones': None, 'canmount': None, 'keystatus': None, 'atime': None, 'usedbysnapshots': '0', 'normalization': None, 'usedbychildren': '0', 'volblocksize': '131072', 'usedbydataset': '57344', 'objsetid': '123031', 'name': 'v_786b4c79-f126-4791-8350-4fed7646bd3b', 'defer_destroy': None, 'pbkdf2iters': '0', 'checksum': 'on', 'redundant_metadata': 'all', 'filesystem_count': None, 'devices': None, 'keyformat': 'none', 'setuid': None, 'used': '57344', 'logicalreferenced': '28672', 'context': 'none', 'zoned': None, 'nbmand': None}} from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_786b4c79-f126-4791-8350-4fed7646bd3b {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:24:21 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-f7fe6c6b-6bf3-4a20-9298-2c754b555694 tempest-VolumesTransfersTest-450487945 None] delete volume:v_786b4c79-f126-4791-8350-4fed7646bd3b recursively children:{'force_umount': True} {{(pid=84696) delete_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:335}} Apr 17 15:24:21 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-f7fe6c6b-6bf3-4a20-9298-2c754b555694 tempest-VolumesTransfersTest-450487945 None] Sending pool request DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_786b4c79-f126-4791-8350-4fed7646bd3b {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:24:21 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-f7fe6c6b-6bf3-4a20-9298-2c754b555694 tempest-VolumesTransfersTest-450487945 None] Sending DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_786b4c79-f126-4791-8350-4fed7646bd3b {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:24:21 user sudo[92056]: pam_unix(sudo:session): session closed for user root Apr 17 15:24:21 user cinder-volume[84696]: DEBUG oslo_concurrency.processutils [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] CMD "sudo cinder-rootwrap /etc/cinder/rootwrap.conf dd count=0 if=/opt/stack/data/cinder/conversion/image_fetch_b3f0dec1-7758-44a8-82e0-76ada8837428_zt3n6g1xuser@joviandss-2 of=/dev/sdb oflag=direct" returned: 0 in 0.955s {{(pid=84696) execute /usr/local/lib/python3.10/dist-packages/oslo_concurrency/processutils.py:422}} Apr 17 15:24:21 user cinder-volume[84696]: DEBUG oslo_concurrency.processutils [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] Running cmd (subprocess): sudo cinder-rootwrap /etc/cinder/rootwrap.conf qemu-img convert -O raw -t none -f qcow2 /opt/stack/data/cinder/conversion/image_fetch_b3f0dec1-7758-44a8-82e0-76ada8837428_zt3n6g1xuser@joviandss-2 /dev/sdb {{(pid=84696) execute /usr/local/lib/python3.10/dist-packages/oslo_concurrency/processutils.py:384}} Apr 17 15:24:21 user sudo[92067]: stack : PWD=/ ; USER=root ; COMMAND=/usr/local/bin/cinder-rootwrap /etc/cinder/rootwrap.conf qemu-img convert -O raw -t none -f qcow2 /opt/stack/data/cinder/conversion/image_fetch_b3f0dec1-7758-44a8-82e0-76ada8837428_zt3n6g1xuser@joviandss-2 /dev/sdb Apr 17 15:24:21 user sudo[92067]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=1001) Apr 17 15:24:22 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-93e81254-daef-413d-949e-6eef52a40357 tempest-GroupSnapshotsTest-1926061586 None] Geting {'code': 200, 'error': None, 'data': {'results': 0, 'entries': []}} from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/s_a4fd2571-7db4-463a-acbe-9e0377b5e9d4/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:24:22 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-93e81254-daef-413d-949e-6eef52a40357 tempest-GroupSnapshotsTest-1926061586 None] get volume s_a4fd2571-7db4-463a-acbe-9e0377b5e9d4 info {{(pid=84696) get_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:239}} Apr 17 15:24:22 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-93e81254-daef-413d-949e-6eef52a40357 tempest-GroupSnapshotsTest-1926061586 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/s_a4fd2571-7db4-463a-acbe-9e0377b5e9d4 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:24:22 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-93e81254-daef-413d-949e-6eef52a40357 tempest-GroupSnapshotsTest-1926061586 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/s_a4fd2571-7db4-463a-acbe-9e0377b5e9d4 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:24:22 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-93e81254-daef-413d-949e-6eef52a40357 tempest-GroupSnapshotsTest-1926061586 None] Geting {'code': 200, 'error': None, 'data': {'origin': 'Pool-0/v_f6603de5-71d3-468b-a673-ce1ff5cd1285@s_a4fd2571-7db4-463a-acbe-9e0377b5e9d4', 'relatime': None, 'acltype': None, 'vscan': None, 'full_name': 'Pool-0/s_a4fd2571-7db4-463a-acbe-9e0377b5e9d4', 'userrefs': None, 'primarycache': 'all', 'logbias': 'latency', 'creation': '1681746447', 'sync': 'always', 'is_clone': True, 'dedup': 'off', 'sharenfs': None, 'receive_resume_token': None, 'volsize': '1073741824', 'referenced': '57344', 'sharesmb': None, 'createtxg': '187882', 'reservation': '0', 'scontext': None, 'mountpoint': None, 'casesensitivity': None, 'guid': '4861645243345174189', 'usedbyrefreservation': '0', 'dnodesize': None, 'written': '0', 'logicalused': '0', 'compressratio': '1.00', 'rootcontext': 'none', 'default_scsi_id': '4071677aab2e4286', 'type': 'volume', 'compression': 'lz4', 'snapdir': None, 'overlay': None, 'encryption': 'off', 'xattr': None, 'volmode': 'default', 'copies': '1', 'snapshot_limit': '18446744073709551615', 'aclinherit': None, 'defcontext': 'none', 'readonly': 'on', 'version': None, 'recordsize': None, 'filesystem_limit': None, 'mounted': None, 'mlslabel': 'none', 'secondarycache': 'all', 'refreservation': '0', 'available': '113926139904', 'used': '0', 'encryptionroot': None, 'exec': None, 'refquota': None, 'refcompressratio': '1.00', 'quota': None, 'utf8only': None, 'keylocation': 'none', 'snapdev': 'hidden', 'snapshot_count': '18446744073709551615', 'fscontext': 'none', 'clones': None, 'canmount': None, 'keystatus': None, 'atime': None, 'usedbysnapshots': '0', 'normalization': None, 'usedbychildren': '0', 'volblocksize': '131072', 'usedbydataset': '0', 'objsetid': '123265', 'name': 's_a4fd2571-7db4-463a-acbe-9e0377b5e9d4', 'defer_destroy': None, 'pbkdf2iters': '0', 'checksum': 'on', 'redundant_metadata': 'all', 'filesystem_count': None, 'devices': None, 'keyformat': 'none', 'setuid': None, 'logicalreferenced': '28672', 'context': 'none', 'zoned': None, 'nbmand': None}} from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/s_a4fd2571-7db4-463a-acbe-9e0377b5e9d4 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:24:22 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-93e81254-daef-413d-949e-6eef52a40357 tempest-GroupSnapshotsTest-1926061586 None] delete snapshot s_a4fd2571-7db4-463a-acbe-9e0377b5e9d4 of volume v_f6603de5-71d3-468b-a673-ce1ff5cd1285 {{(pid=84696) delete_snapshot /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:751}} Apr 17 15:24:22 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-93e81254-daef-413d-949e-6eef52a40357 tempest-GroupSnapshotsTest-1926061586 None] Sending pool request DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_f6603de5-71d3-468b-a673-ce1ff5cd1285/snapshots/s_a4fd2571-7db4-463a-acbe-9e0377b5e9d4 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:24:22 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-93e81254-daef-413d-949e-6eef52a40357 tempest-GroupSnapshotsTest-1926061586 None] Sending DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_f6603de5-71d3-468b-a673-ce1ff5cd1285/snapshots/s_a4fd2571-7db4-463a-acbe-9e0377b5e9d4 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:24:24 user sudo[92067]: pam_unix(sudo:session): session closed for user root Apr 17 15:24:24 user cinder-volume[84696]: DEBUG oslo_concurrency.processutils [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] CMD "sudo cinder-rootwrap /etc/cinder/rootwrap.conf qemu-img convert -O raw -t none -f qcow2 /opt/stack/data/cinder/conversion/image_fetch_b3f0dec1-7758-44a8-82e0-76ada8837428_zt3n6g1xuser@joviandss-2 /dev/sdb" returned: 0 in 2.929s {{(pid=84696) execute /usr/local/lib/python3.10/dist-packages/oslo_concurrency/processutils.py:422}} Apr 17 15:24:24 user cinder-volume[84696]: DEBUG oslo_concurrency.processutils [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] Running cmd (subprocess): /usr/bin/python3.10 -m oslo_concurrency.prlimit --as=1073741824 --cpu=60 -- sudo cinder-rootwrap /etc/cinder/rootwrap.conf env LC_ALL=C qemu-img info --output=json /opt/stack/data/cinder/conversion/image_fetch_b3f0dec1-7758-44a8-82e0-76ada8837428_zt3n6g1xuser@joviandss-2 {{(pid=84696) execute /usr/local/lib/python3.10/dist-packages/oslo_concurrency/processutils.py:384}} Apr 17 15:24:24 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-ce906aff-8139-4f35-93dd-d0256428c52f tempest-VolumesGetTest-1961901732 None] Geting {'code': 204} from DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_f088992b-786a-4641-8bcc-5cb7c819d313 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:24:24 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-ce906aff-8139-4f35-93dd-d0256428c52f tempest-VolumesGetTest-1961901732 None] volume v_f088992b-786a-4641-8bcc-5cb7c819d313 deleted {{(pid=84696) delete_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:346}} Apr 17 15:24:24 user sudo[92123]: stack : PWD=/ ; USER=root ; COMMAND=/usr/local/bin/cinder-rootwrap /etc/cinder/rootwrap.conf env LC_ALL=C qemu-img info --output=json /opt/stack/data/cinder/conversion/image_fetch_b3f0dec1-7758-44a8-82e0-76ada8837428_zt3n6g1xuser@joviandss-2 Apr 17 15:24:24 user sudo[92123]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=1001) Apr 17 15:24:24 user cinder-volume[84696]: DEBUG cinder.quota [None req-ce906aff-8139-4f35-93dd-d0256428c52f tempest-VolumesGetTest-1961901732 None] Created reservations ['47d9a02d-dd2f-4ab1-8af5-cd91c70182d6', '9be3de47-6ae7-4f88-8069-1fbc4b3eb9a1', '3f8ac92a-78eb-41aa-bb1c-bdfc2a344b5d', '186aa571-87d7-4368-a12f-38f04f52b8b1'] {{(pid=84696) reserve /opt/stack/cinder/cinder/quota.py:755}} Apr 17 15:24:24 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-ce906aff-8139-4f35-93dd-d0256428c52f tempest-VolumesGetTest-1961901732 None] Updating volume stats {{(pid=84696) _update_volume_stats /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:724}} Apr 17 15:24:24 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-ce906aff-8139-4f35-93dd-d0256428c52f tempest-VolumesGetTest-1961901732 None] Get pool Pool-0 fsprops {{(pid=84696) get_pool_stats /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:894}} Apr 17 15:24:24 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-ce906aff-8139-4f35-93dd-d0256428c52f tempest-VolumesGetTest-1961901732 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:24:24 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-ce906aff-8139-4f35-93dd-d0256428c52f tempest-VolumesGetTest-1961901732 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:24:25 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-f7fe6c6b-6bf3-4a20-9298-2c754b555694 tempest-VolumesTransfersTest-450487945 None] Geting {'code': 204} from DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_786b4c79-f126-4791-8350-4fed7646bd3b {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:24:25 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-f7fe6c6b-6bf3-4a20-9298-2c754b555694 tempest-VolumesTransfersTest-450487945 None] volume v_786b4c79-f126-4791-8350-4fed7646bd3b deleted {{(pid=84696) delete_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:346}} Apr 17 15:24:25 user cinder-volume[84696]: DEBUG cinder.quota [None req-f7fe6c6b-6bf3-4a20-9298-2c754b555694 tempest-VolumesTransfersTest-450487945 None] Created reservations ['b5d94aca-b86a-4ce5-a104-0127026ded56', '4776e914-a15d-4300-926a-b8d9b3706d3e', 'b5be909d-2b0a-4c86-a40b-5a38a16c0dd2', 'fffc4c7c-7f5f-44d9-8736-92321c670002'] {{(pid=84696) reserve /opt/stack/cinder/cinder/quota.py:755}} Apr 17 15:24:25 user sudo[92123]: pam_unix(sudo:session): session closed for user root Apr 17 15:24:25 user cinder-volume[84696]: DEBUG oslo_concurrency.processutils [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] CMD "/usr/bin/python3.10 -m oslo_concurrency.prlimit --as=1073741824 --cpu=60 -- sudo cinder-rootwrap /etc/cinder/rootwrap.conf env LC_ALL=C qemu-img info --output=json /opt/stack/data/cinder/conversion/image_fetch_b3f0dec1-7758-44a8-82e0-76ada8837428_zt3n6g1xuser@joviandss-2" returned: 0 in 1.076s {{(pid=84696) execute /usr/local/lib/python3.10/dist-packages/oslo_concurrency/processutils.py:422}} Apr 17 15:24:25 user cinder-volume[84696]: DEBUG cinder.image.image_utils [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] Image conversion details: src /opt/stack/data/cinder/conversion/image_fetch_b3f0dec1-7758-44a8-82e0-76ada8837428_zt3n6g1xuser@joviandss-2, size 112.00 MB, duration 2.94 sec, destination /dev/sdb {{(pid=84696) _convert_image /opt/stack/cinder/cinder/image/image_utils.py:432}} Apr 17 15:24:25 user cinder-volume[84696]: INFO cinder.image.image_utils [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] Converted 112.00 MB image at 38.07 MB/s Apr 17 15:24:25 user cinder-volume[84696]: DEBUG os_brick.initiator.connectors.iscsi [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] ==> disconnect_volume: call "{'args': (, {'target_iqn': 'iqn.2010-10.org.openstack:c74b7878-4936-4015-8b63-142923947310', 'target_portal': '172.16.0.220:3260', 'target_discovered': False, 'auth_method': 'CHAP', 'auth_username': 'PUvXoCNO', 'auth_password': '***', 'target_lun': 0, 'encrypted': False}, {'type': 'block', 'scsi_wwn': '26132396565663337', 'path': '/dev/sdb'}), 'kwargs': {'force': False}}" {{(pid=84696) trace_logging_wrapper /usr/local/lib/python3.10/dist-packages/os_brick/utils.py:175}} Apr 17 15:24:25 user cinder-volume[84696]: DEBUG os_brick.initiator.connectors.base [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] Acquiring lock "connect_volume" by "os_brick.initiator.connectors.iscsi.ISCSIConnector.disconnect_volume" {{(pid=84696) inner /usr/local/lib/python3.10/dist-packages/os_brick/initiator/connectors/base.py:68}} Apr 17 15:24:25 user cinder-volume[84696]: DEBUG os_brick.initiator.connectors.base [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] Lock "connect_volume" acquired by "os_brick.initiator.connectors.iscsi.ISCSIConnector.disconnect_volume" :: waited 0.003s {{(pid=84696) inner /usr/local/lib/python3.10/dist-packages/os_brick/initiator/connectors/base.py:73}} Apr 17 15:24:25 user cinder-volume[84696]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m discoverydb -o show -P 1 {{(pid=91817) execute /usr/local/lib/python3.10/dist-packages/oslo_concurrency/processutils.py:384}} Apr 17 15:24:25 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-f7fe6c6b-6bf3-4a20-9298-2c754b555694 tempest-VolumesTransfersTest-450487945 None] Updating volume stats {{(pid=84696) _update_volume_stats /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:724}} Apr 17 15:24:25 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-f7fe6c6b-6bf3-4a20-9298-2c754b555694 tempest-VolumesTransfersTest-450487945 None] Get pool Pool-0 fsprops {{(pid=84696) get_pool_stats /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:894}} Apr 17 15:24:25 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-f7fe6c6b-6bf3-4a20-9298-2c754b555694 tempest-VolumesTransfersTest-450487945 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:24:25 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-f7fe6c6b-6bf3-4a20-9298-2c754b555694 tempest-VolumesTransfersTest-450487945 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:24:25 user cinder-volume[84696]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m discoverydb -o show -P 1" returned: 0 in 0.037s {{(pid=91817) execute /usr/local/lib/python3.10/dist-packages/oslo_concurrency/processutils.py:422}} Apr 17 15:24:25 user cinder-volume[84696]: DEBUG oslo.privsep.daemon [-] privsep: reply[50719131-92a4-4234-ab41-5a623fe0fdf2]: (4, ('SENDTARGETS:\nNo targets found.\niSNS:\nNo targets found.\nSTATIC:\nTarget: iqn.2010-10.org.openstack:c74b7878-4936-4015-8b63-142923947310\n\tPortal: 172.16.0.220:3260,-1\n\t\tIface Name: default\nFIRMWARE:\nNo targets found.\n', '')) {{(pid=91817) _call_back /usr/local/lib/python3.10/dist-packages/oslo_privsep/daemon.py:501}} Apr 17 15:24:25 user cinder-volume[84696]: DEBUG os_brick.initiator.connectors.iscsi [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] iscsiadm ['-m', 'discoverydb', '-o', 'show', '-P', 1]: stdout=SENDTARGETS: Apr 17 15:24:25 user cinder-volume[84696]: No targets found. Apr 17 15:24:25 user cinder-volume[84696]: iSNS: Apr 17 15:24:25 user cinder-volume[84696]: No targets found. Apr 17 15:24:25 user cinder-volume[84696]: STATIC: Apr 17 15:24:25 user cinder-volume[84696]: Target: iqn.2010-10.org.openstack:c74b7878-4936-4015-8b63-142923947310 Apr 17 15:24:25 user cinder-volume[84696]: Portal: 172.16.0.220:3260,-1 Apr 17 15:24:25 user cinder-volume[84696]: Iface Name: default Apr 17 15:24:25 user cinder-volume[84696]: FIRMWARE: Apr 17 15:24:25 user cinder-volume[84696]: No targets found. Apr 17 15:24:25 user cinder-volume[84696]: stderr= {{(pid=84696) _run_iscsiadm_bare /usr/local/lib/python3.10/dist-packages/os_brick/initiator/connectors/iscsi.py:1182}} Apr 17 15:24:25 user cinder-volume[84696]: DEBUG os_brick.initiator.connectors.iscsi [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] Regex to get portals from discoverydb: ^SENDTARGETS: Apr 17 15:24:25 user cinder-volume[84696]: .*?^DiscoveryAddress: 172.16.0.220,3260.*? Apr 17 15:24:25 user cinder-volume[84696]: (.*?)^(?:DiscoveryAddress|iSNS):.* {{(pid=84696) _get_discoverydb_portals /usr/local/lib/python3.10/dist-packages/os_brick/initiator/connectors/iscsi.py:371}} Apr 17 15:24:25 user cinder-volume[84696]: DEBUG os_brick.initiator.connectors.iscsi [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] Getting connected devices for (ips,iqns,luns)=[('172.16.0.220:3260', 'iqn.2010-10.org.openstack:c74b7878-4936-4015-8b63-142923947310', 0)] {{(pid=84696) _get_connection_devices /usr/local/lib/python3.10/dist-packages/os_brick/initiator/connectors/iscsi.py:819}} Apr 17 15:24:25 user cinder-volume[84696]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m node {{(pid=91817) execute /usr/local/lib/python3.10/dist-packages/oslo_concurrency/processutils.py:384}} Apr 17 15:24:25 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-f7fe6c6b-6bf3-4a20-9298-2c754b555694 tempest-VolumesTransfersTest-450487945 None] Geting {'code': 200, 'error': None, 'data': {'available': '113926115328', 'status': 30, 'import_status': {'import_steps': {'san_setup': True, 'vip_setup': True, 'zfs_import': True, 'mount_system_volume': True, 'nas_setup': True}, 'import_successful': True}, 'scan': None, 'encryption': {'enabled': False}, 'deduplication_rate': '1.00', 'sysvol_upgrade': 'not_needed', 'vdevs': [{'name': 'wwn-0x6000c296b793bf526a107a9d823e947a', 'iostats': {'read': '0', 'write': '0', 'chksum': '0'}, 'disks': [{'origin': 'local', 'slot': '', 'led': 'off', 'name': 'sdn', 'iostats': {'read': '0', 'write': '0', 'chksum': '0'}, 'alias': '', 'health': 'ONLINE', 'sn': '6000c296b793bf526a107a9d823e947a', 'trim_data': {'status': 'trim_unsupported', 'progress': None, 'trim_time': None}, 'path': 'pci-0000:03:00.0-scsi-0:0:14:0', 'model': 'Virtual disk', 'id': 'wwn-0x6000c296b793bf526a107a9d823e947a', 'size': 128849018880}], 'health': 'ONLINE', 'vdev_replacings': [], 'vdev_spares': [], 'type': ''}], 'id': '14181317334440137330', 'health': 'ONLINE', 'iostats': {'read': '0', 'write': '0', 'chksum': '0'}, 'operation': 'none', 'size': '127775277056', 'autotrim': False, 'name': 'Pool-0'}} from GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:24:25 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-f7fe6c6b-6bf3-4a20-9298-2c754b555694 tempest-VolumesTransfersTest-450487945 None] Total capacity: 119, Free 106. {{(pid=84696) _update_volume_stats /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:757}} Apr 17 15:24:25 user cinder-volume[84696]: DEBUG cinder.manager [None req-f7fe6c6b-6bf3-4a20-9298-2c754b555694 tempest-VolumesTransfersTest-450487945 None] Notifying Schedulers of capabilities ... {{(pid=84696) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 17 15:24:25 user cinder-volume[84696]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m node" returned: 0 in 0.022s {{(pid=91817) execute /usr/local/lib/python3.10/dist-packages/oslo_concurrency/processutils.py:422}} Apr 17 15:24:25 user cinder-volume[84696]: DEBUG oslo.privsep.daemon [-] privsep: reply[6ce36afa-6d18-4258-85f1-5e274fc827a6]: (4, ('172.16.0.220:3260,4294967295 iqn.2010-10.org.openstack:c74b7878-4936-4015-8b63-142923947310\n', '')) {{(pid=91817) _call_back /usr/local/lib/python3.10/dist-packages/oslo_privsep/daemon.py:501}} Apr 17 15:24:25 user cinder-volume[84696]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m session {{(pid=91817) execute /usr/local/lib/python3.10/dist-packages/oslo_concurrency/processutils.py:384}} Apr 17 15:24:25 user cinder-volume[84696]: INFO cinder.volume.manager [None req-f7fe6c6b-6bf3-4a20-9298-2c754b555694 tempest-VolumesTransfersTest-450487945 None] Deleted volume successfully. Apr 17 15:24:25 user cinder-volume[84696]: DEBUG cinder.coordination [None req-f7fe6c6b-6bf3-4a20-9298-2c754b555694 tempest-VolumesTransfersTest-450487945 None] Lock "cinder-786b4c79-f126-4791-8350-4fed7646bd3b-delete_volume" released by "delete_volume" :: held 5.441s {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:207}} Apr 17 15:24:25 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-93e81254-daef-413d-949e-6eef52a40357 tempest-GroupSnapshotsTest-1926061586 None] Geting {'code': 204} from DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_f6603de5-71d3-468b-a673-ce1ff5cd1285/snapshots/s_a4fd2571-7db4-463a-acbe-9e0377b5e9d4 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:24:25 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-93e81254-daef-413d-949e-6eef52a40357 tempest-GroupSnapshotsTest-1926061586 None] snapshot s_a4fd2571-7db4-463a-acbe-9e0377b5e9d4 deleted {{(pid=84696) delete_snapshot /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:772}} Apr 17 15:24:25 user cinder-volume[84696]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m session" returned: 0 in 0.057s {{(pid=91817) execute /usr/local/lib/python3.10/dist-packages/oslo_concurrency/processutils.py:422}} Apr 17 15:24:25 user cinder-volume[84696]: DEBUG oslo.privsep.daemon [-] privsep: reply[268ee796-36a3-4a1c-b2f9-66aa8103aa7d]: (4, ('tcp: [1] 172.16.0.220:3260,1 iqn.2010-10.org.openstack:c74b7878-4936-4015-8b63-142923947310 (non-flash)\n', '')) {{(pid=91817) _call_back /usr/local/lib/python3.10/dist-packages/oslo_privsep/daemon.py:501}} Apr 17 15:24:25 user cinder-volume[84696]: DEBUG os_brick.initiator.connectors.iscsi [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] iscsiadm ('-m', 'session'): stdout=tcp: [1] 172.16.0.220:3260,1 iqn.2010-10.org.openstack:c74b7878-4936-4015-8b63-142923947310 (non-flash) Apr 17 15:24:25 user cinder-volume[84696]: stderr= {{(pid=84696) _run_iscsiadm_bare /usr/local/lib/python3.10/dist-packages/os_brick/initiator/connectors/iscsi.py:1182}} Apr 17 15:24:25 user cinder-volume[84696]: DEBUG os_brick.initiator.connectors.iscsi [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] iscsi session list stdout=tcp: [1] 172.16.0.220:3260,1 iqn.2010-10.org.openstack:c74b7878-4936-4015-8b63-142923947310 (non-flash) Apr 17 15:24:25 user cinder-volume[84696]: stderr= {{(pid=84696) _run_iscsi_session /usr/local/lib/python3.10/dist-packages/os_brick/initiator/connectors/iscsi.py:1171}} Apr 17 15:24:25 user cinder-volume[84696]: DEBUG os_brick.initiator.connectors.iscsi [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] Resulting device map defaultdict(. at 0x7f6dd7f649d0>, {('172.16.0.220:3260', 'iqn.2010-10.org.openstack:c74b7878-4936-4015-8b63-142923947310'): ({'sdb'}, set())}) {{(pid=84696) _get_connection_devices /usr/local/lib/python3.10/dist-packages/os_brick/initiator/connectors/iscsi.py:852}} Apr 17 15:24:25 user cinder-volume[84696]: DEBUG os_brick.initiator.linuxscsi [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] Removing single pathed devices sdb {{(pid=84696) remove_connection /usr/local/lib/python3.10/dist-packages/os_brick/initiator/linuxscsi.py:309}} Apr 17 15:24:25 user cinder-volume[84696]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): multipathd show status {{(pid=91817) execute /usr/local/lib/python3.10/dist-packages/oslo_concurrency/processutils.py:384}} Apr 17 15:24:25 user cinder-volume[84696]: DEBUG oslo_concurrency.processutils [-] CMD "multipathd show status" returned: 0 in 0.023s {{(pid=91817) execute /usr/local/lib/python3.10/dist-packages/oslo_concurrency/processutils.py:422}} Apr 17 15:24:25 user cinder-volume[84696]: DEBUG oslo.privsep.daemon [-] privsep: reply[87dfd116-da90-4995-a997-eeb0b55ca21f]: (4, ('path checker states:\nup 1\n\npaths: 0\nbusy: False\n', '')) {{(pid=91817) _call_back /usr/local/lib/python3.10/dist-packages/oslo_privsep/daemon.py:501}} Apr 17 15:24:25 user cinder-volume[84696]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): multipathd del path /dev/sdb {{(pid=91817) execute /usr/local/lib/python3.10/dist-packages/oslo_concurrency/processutils.py:384}} Apr 17 15:24:25 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-c3c39056-e984-4f53-8f45-8ecd6577db98 tempest-VolumesGetTest-1961901732 None] Volume reschedule parameters: True retry: {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 15:24:25 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-c3c39056-e984-4f53-8f45-8ecd6577db98 tempest-VolumesGetTest-1961901732 None] Flow 'volume_create_manager' (e5f4bfd4-9487-4f0c-a4dc-f6a515c75814) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:24:25 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-c3c39056-e984-4f53-8f45-8ecd6577db98 tempest-VolumesGetTest-1961901732 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (613d00ed-767b-4e4a-9c5c-aaa4303fb89c) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:24:25 user cinder-volume[84696]: DEBUG oslo_concurrency.processutils [-] CMD "multipathd del path /dev/sdb" returned: 0 in 0.023s {{(pid=91817) execute /usr/local/lib/python3.10/dist-packages/oslo_concurrency/processutils.py:422}} Apr 17 15:24:25 user cinder-volume[84696]: DEBUG oslo.privsep.daemon [-] privsep: reply[310ad174-425d-46cd-af85-4ac106ac604c]: (4, ('ok\n', '')) {{(pid=91817) _call_back /usr/local/lib/python3.10/dist-packages/oslo_privsep/daemon.py:501}} Apr 17 15:24:25 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=56,cinder:UPDATE=34,cinder:DELETE=3,cinder:INSERT=7 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:24:25 user cinder-volume[84696]: DEBUG os_brick.initiator.linuxscsi [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] Flushing IO for device /dev/sdb {{(pid=84696) flush_device_io /usr/local/lib/python3.10/dist-packages/os_brick/initiator/linuxscsi.py:369}} Apr 17 15:24:25 user cinder-volume[84696]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): blockdev --flushbufs /dev/sdb {{(pid=91817) execute /usr/local/lib/python3.10/dist-packages/oslo_concurrency/processutils.py:384}} Apr 17 15:24:25 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-c3c39056-e984-4f53-8f45-8ecd6577db98 tempest-VolumesGetTest-1961901732 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (613d00ed-767b-4e4a-9c5c-aaa4303fb89c) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:24:25Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesGetTest-Volume-1667230460',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=9a8586a6-f2f5-4bee-a56f-a06b01120c39,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='a65d14b0a2894a6f90506d86feaefd15',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:24:26Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:24:26Z,use_quota=True,user_id='6ffa3051570a4721a48a0171b3cc4d16',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:24:25 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-c3c39056-e984-4f53-8f45-8ecd6577db98 tempest-VolumesGetTest-1961901732 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (f0f33e90-a645-487c-8218-ca9348c35b1e) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:24:25 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-c3c39056-e984-4f53-8f45-8ecd6577db98 tempest-VolumesGetTest-1961901732 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (f0f33e90-a645-487c-8218-ca9348c35b1e) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:24:25 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-c3c39056-e984-4f53-8f45-8ecd6577db98 tempest-VolumesGetTest-1961901732 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (be78b21a-738a-469a-b447-09ad50e22a15) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:24:25 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-c3c39056-e984-4f53-8f45-8ecd6577db98 tempest-VolumesGetTest-1961901732 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (be78b21a-738a-469a-b447-09ad50e22a15) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': '9a8586a6-f2f5-4bee-a56f-a06b01120c39', 'volume_name': 'volume-9a8586a6-f2f5-4bee-a56f-a06b01120c39', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:24:25 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-c3c39056-e984-4f53-8f45-8ecd6577db98 tempest-VolumesGetTest-1961901732 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (c8110e75-badf-46c8-b61f-29c1a2abe723) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:24:25 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-c3c39056-e984-4f53-8f45-8ecd6577db98 tempest-VolumesGetTest-1961901732 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (c8110e75-badf-46c8-b61f-29c1a2abe723) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:24:25 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-c3c39056-e984-4f53-8f45-8ecd6577db98 tempest-VolumesGetTest-1961901732 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (4aa285a1-322a-427f-a375-910ac5cdb6cf) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:24:25 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-c3c39056-e984-4f53-8f45-8ecd6577db98 tempest-VolumesGetTest-1961901732 None] Volume 9a8586a6-f2f5-4bee-a56f-a06b01120c39: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-9a8586a6-f2f5-4bee-a56f-a06b01120c39', 'volume_size': 1} Apr 17 15:24:25 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-c3c39056-e984-4f53-8f45-8ecd6577db98 tempest-VolumesGetTest-1961901732 None] creating volume v_9a8586a6-f2f5-4bee-a56f-a06b01120c39. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 15:24:25 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-c3c39056-e984-4f53-8f45-8ecd6577db98 tempest-VolumesGetTest-1961901732 None] create volume {'name': 'v_9a8586a6-f2f5-4bee-a56f-a06b01120c39', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 15:24:25 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-c3c39056-e984-4f53-8f45-8ecd6577db98 tempest-VolumesGetTest-1961901732 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:24:25 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-c3c39056-e984-4f53-8f45-8ecd6577db98 tempest-VolumesGetTest-1961901732 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:24:25 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-ce906aff-8139-4f35-93dd-d0256428c52f tempest-VolumesGetTest-1961901732 None] Geting {'code': 200, 'error': None, 'data': {'available': '113910112256', 'status': 30, 'import_status': {'import_steps': {'san_setup': True, 'vip_setup': True, 'zfs_import': True, 'mount_system_volume': True, 'nas_setup': True}, 'import_successful': True}, 'scan': None, 'encryption': {'enabled': False}, 'deduplication_rate': '1.00', 'sysvol_upgrade': 'not_needed', 'vdevs': [{'name': 'wwn-0x6000c296b793bf526a107a9d823e947a', 'iostats': {'read': '0', 'write': '0', 'chksum': '0'}, 'disks': [{'origin': 'local', 'slot': '', 'led': 'off', 'name': 'sdn', 'iostats': {'read': '0', 'write': '0', 'chksum': '0'}, 'alias': '', 'health': 'ONLINE', 'sn': '6000c296b793bf526a107a9d823e947a', 'trim_data': {'status': 'trim_unsupported', 'progress': None, 'trim_time': None}, 'path': 'pci-0000:03:00.0-scsi-0:0:14:0', 'model': 'Virtual disk', 'id': 'wwn-0x6000c296b793bf526a107a9d823e947a', 'size': 128849018880}], 'health': 'ONLINE', 'vdev_replacings': [], 'vdev_spares': [], 'type': ''}], 'id': '14181317334440137330', 'health': 'ONLINE', 'iostats': {'read': '0', 'write': '0', 'chksum': '0'}, 'operation': 'none', 'size': '127775277056', 'autotrim': False, 'name': 'Pool-0'}} from GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:24:25 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-ce906aff-8139-4f35-93dd-d0256428c52f tempest-VolumesGetTest-1961901732 None] Total capacity: 119, Free 106. {{(pid=84696) _update_volume_stats /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:757}} Apr 17 15:24:25 user cinder-volume[84696]: DEBUG cinder.manager [None req-ce906aff-8139-4f35-93dd-d0256428c52f tempest-VolumesGetTest-1961901732 None] Notifying Schedulers of capabilities ... {{(pid=84696) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 17 15:24:25 user cinder-volume[84696]: DEBUG oslo_concurrency.processutils [-] CMD "blockdev --flushbufs /dev/sdb" returned: 0 in 0.068s {{(pid=91817) execute /usr/local/lib/python3.10/dist-packages/oslo_concurrency/processutils.py:422}} Apr 17 15:24:25 user cinder-volume[84696]: DEBUG oslo.privsep.daemon [-] privsep: reply[3295d2e3-a2c9-4b7c-ae25-9421152e959d]: (4, ('', '')) {{(pid=91817) _call_back /usr/local/lib/python3.10/dist-packages/oslo_privsep/daemon.py:501}} Apr 17 15:24:25 user cinder-volume[84696]: DEBUG os_brick.initiator.linuxscsi [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] Remove SCSI device /dev/sdb with /sys/block/sdb/device/delete {{(pid=84696) remove_scsi_device /usr/local/lib/python3.10/dist-packages/os_brick/initiator/linuxscsi.py:83}} Apr 17 15:24:26 user cinder-volume[84696]: INFO cinder.volume.manager [None req-ce906aff-8139-4f35-93dd-d0256428c52f tempest-VolumesGetTest-1961901732 None] Deleted volume successfully. Apr 17 15:24:26 user cinder-volume[84696]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): tee -a /sys/block/sdb/device/delete {{(pid=91817) execute /usr/local/lib/python3.10/dist-packages/oslo_concurrency/processutils.py:384}} Apr 17 15:24:26 user cinder-volume[84696]: DEBUG cinder.coordination [None req-ce906aff-8139-4f35-93dd-d0256428c52f tempest-VolumesGetTest-1961901732 None] Lock "cinder-f088992b-786a-4641-8bcc-5cb7c819d313-delete_volume" released by "delete_volume" :: held 6.243s {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:207}} Apr 17 15:24:26 user cinder-volume[84696]: WARNING cinder.quota [None req-93e81254-daef-413d-949e-6eef52a40357 tempest-GroupSnapshotsTest-1926061586 None] Deprecated: Default quota for resource: gigabytes_tempest-GroupSnapshotsTest-volume-type-2030863494 is set by the default quota flag: quota_gigabytes_tempest-GroupSnapshotsTest-volume-type-2030863494, it is now deprecated. Please use the default quota class for default quota. Apr 17 15:24:26 user cinder-volume[84696]: WARNING cinder.quota [None req-93e81254-daef-413d-949e-6eef52a40357 tempest-GroupSnapshotsTest-1926061586 None] Deprecated: Default quota for resource: snapshots_tempest-GroupSnapshotsTest-volume-type-2030863494 is set by the default quota flag: quota_snapshots_tempest-GroupSnapshotsTest-volume-type-2030863494, it is now deprecated. Please use the default quota class for default quota. Apr 17 15:24:26 user cinder-volume[84696]: DEBUG oslo_concurrency.processutils [-] CMD "tee -a /sys/block/sdb/device/delete" returned: 0 in 0.089s {{(pid=91817) execute /usr/local/lib/python3.10/dist-packages/oslo_concurrency/processutils.py:422}} Apr 17 15:24:26 user cinder-volume[84696]: DEBUG oslo.privsep.daemon [-] privsep: reply[4d16010a-5c2c-43bf-a317-8dcd1d392f74]: (4, ('1', '')) {{(pid=91817) _call_back /usr/local/lib/python3.10/dist-packages/oslo_privsep/daemon.py:501}} Apr 17 15:24:26 user cinder-volume[84696]: DEBUG os_brick.initiator.linuxscsi [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] Checking to see if SCSI volumes sdb have been removed. {{(pid=84696) wait_for_volumes_removal /usr/local/lib/python3.10/dist-packages/os_brick/initiator/linuxscsi.py:91}} Apr 17 15:24:26 user cinder-volume[84696]: DEBUG os_brick.initiator.linuxscsi [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] SCSI volumes sdb have been removed. {{(pid=84696) wait_for_volumes_removal /usr/local/lib/python3.10/dist-packages/os_brick/initiator/linuxscsi.py:101}} Apr 17 15:24:26 user cinder-volume[84696]: DEBUG os_brick.initiator.connectors.iscsi [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] Disconnecting from: [('172.16.0.220:3260', 'iqn.2010-10.org.openstack:c74b7878-4936-4015-8b63-142923947310')] {{(pid=84696) _disconnect_connection /usr/local/lib/python3.10/dist-packages/os_brick/initiator/connectors/iscsi.py:1160}} Apr 17 15:24:26 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:UPDATE=30,cinder:SELECT=62,cinder:DELETE=2,cinder:INSERT=6 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:24:26 user cinder-volume[84696]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m node -T iqn.2010-10.org.openstack:c74b7878-4936-4015-8b63-142923947310 -p 172.16.0.220:3260 --op update -n node.startup -v manual {{(pid=91817) execute /usr/local/lib/python3.10/dist-packages/oslo_concurrency/processutils.py:384}} Apr 17 15:24:26 user cinder-volume[84696]: DEBUG cinder.quota [None req-93e81254-daef-413d-949e-6eef52a40357 tempest-GroupSnapshotsTest-1926061586 None] Created reservations ['4d8a378d-40b8-4545-9a06-734bd4cbc89d', 'c007c5ce-0530-4c80-a6b8-2e3af46fde76', '19a13f9a-8901-446c-945a-8c96549645a4', '6c509c1e-394e-4082-8e78-01c017c236e3'] {{(pid=84696) reserve /opt/stack/cinder/cinder/quota.py:755}} Apr 17 15:24:26 user cinder-volume[84696]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m node -T iqn.2010-10.org.openstack:c74b7878-4936-4015-8b63-142923947310 -p 172.16.0.220:3260 --op update -n node.startup -v manual" returned: 0 in 0.023s {{(pid=91817) execute /usr/local/lib/python3.10/dist-packages/oslo_concurrency/processutils.py:422}} Apr 17 15:24:26 user cinder-volume[84696]: DEBUG oslo.privsep.daemon [-] privsep: reply[0997afdc-6dc0-4d6a-ba0a-4ef455f9760a]: (4, ('', '')) {{(pid=91817) _call_back /usr/local/lib/python3.10/dist-packages/oslo_privsep/daemon.py:501}} Apr 17 15:24:26 user cinder-volume[84696]: DEBUG os_brick.initiator.connectors.iscsi [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] iscsiadm ('--op', 'update', '-n', 'node.startup', '-v', 'manual'): stdout= stderr= {{(pid=84696) _run_iscsiadm /usr/local/lib/python3.10/dist-packages/os_brick/initiator/connectors/iscsi.py:1026}} Apr 17 15:24:26 user cinder-volume[84696]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m node -T iqn.2010-10.org.openstack:c74b7878-4936-4015-8b63-142923947310 -p 172.16.0.220:3260 --logout {{(pid=91817) execute /usr/local/lib/python3.10/dist-packages/oslo_concurrency/processutils.py:384}} Apr 17 15:24:26 user cinder-volume[84696]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m node -T iqn.2010-10.org.openstack:c74b7878-4936-4015-8b63-142923947310 -p 172.16.0.220:3260 --logout" returned: 0 in 0.086s {{(pid=91817) execute /usr/local/lib/python3.10/dist-packages/oslo_concurrency/processutils.py:422}} Apr 17 15:24:26 user cinder-volume[84696]: DEBUG oslo.privsep.daemon [-] privsep: reply[a86f8b16-7d5d-4a93-b5bf-7b36ccab6421]: (4, ('Logging out of session [sid: 1, target: iqn.2010-10.org.openstack:c74b7878-4936-4015-8b63-142923947310, portal: 172.16.0.220,3260]\nLogout of [sid: 1, target: iqn.2010-10.org.openstack:c74b7878-4936-4015-8b63-142923947310, portal: 172.16.0.220,3260] successful.\n', '')) {{(pid=91817) _call_back /usr/local/lib/python3.10/dist-packages/oslo_privsep/daemon.py:501}} Apr 17 15:24:26 user cinder-volume[84696]: DEBUG os_brick.initiator.connectors.iscsi [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] iscsiadm ('--logout',): stdout=Logging out of session [sid: 1, target: iqn.2010-10.org.openstack:c74b7878-4936-4015-8b63-142923947310, portal: 172.16.0.220,3260] Apr 17 15:24:26 user cinder-volume[84696]: Logout of [sid: 1, target: iqn.2010-10.org.openstack:c74b7878-4936-4015-8b63-142923947310, portal: 172.16.0.220,3260] successful. Apr 17 15:24:26 user cinder-volume[84696]: stderr= {{(pid=84696) _run_iscsiadm /usr/local/lib/python3.10/dist-packages/os_brick/initiator/connectors/iscsi.py:1026}} Apr 17 15:24:26 user cinder-volume[84696]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m node -T iqn.2010-10.org.openstack:c74b7878-4936-4015-8b63-142923947310 -p 172.16.0.220:3260 --op delete {{(pid=91817) execute /usr/local/lib/python3.10/dist-packages/oslo_concurrency/processutils.py:384}} Apr 17 15:24:26 user cinder-volume[84696]: DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m node -T iqn.2010-10.org.openstack:c74b7878-4936-4015-8b63-142923947310 -p 172.16.0.220:3260 --op delete" returned: 0 in 0.035s {{(pid=91817) execute /usr/local/lib/python3.10/dist-packages/oslo_concurrency/processutils.py:422}} Apr 17 15:24:26 user cinder-volume[84696]: DEBUG oslo.privsep.daemon [-] privsep: reply[3ca1a626-2a05-417e-ba97-5b37d8b68dd2]: (4, ('', '')) {{(pid=91817) _call_back /usr/local/lib/python3.10/dist-packages/oslo_privsep/daemon.py:501}} Apr 17 15:24:26 user cinder-volume[84696]: DEBUG os_brick.initiator.connectors.iscsi [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] iscsiadm ('--op', 'delete'): stdout= stderr= {{(pid=84696) _run_iscsiadm /usr/local/lib/python3.10/dist-packages/os_brick/initiator/connectors/iscsi.py:1026}} Apr 17 15:24:26 user cinder-volume[84696]: DEBUG os_brick.initiator.connectors.base [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] Lock "connect_volume" "released" by "os_brick.initiator.connectors.iscsi.ISCSIConnector.disconnect_volume" :: held 0.700s {{(pid=84696) inner /usr/local/lib/python3.10/dist-packages/os_brick/initiator/connectors/base.py:87}} Apr 17 15:24:26 user cinder-volume[84696]: DEBUG os_brick.initiator.connectors.iscsi [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] <== disconnect_volume: return (703ms) None {{(pid=84696) trace_logging_wrapper /usr/local/lib/python3.10/dist-packages/os_brick/utils.py:202}} Apr 17 15:24:26 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] terminate connection for c74b7878-4936-4015-8b63-142923947310 {{(pid=84696) terminate_connection /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:1032}} Apr 17 15:24:26 user cinder-volume[84696]: DEBUG cinder.volume.driver [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] volume c74b7878-4936-4015-8b63-142923947310: removing export {{(pid=84696) _detach_volume /opt/stack/cinder/cinder/volume/driver.py:560}} Apr 17 15:24:26 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] remove_export for volume: c74b7878-4936-4015-8b63-142923947310. {{(pid=84696) remove_export /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:718}} Apr 17 15:24:26 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] remove export {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:912}} Apr 17 15:24:26 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] detach volume:Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:24:01Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TaggedBootDevicesTest_v242-volume-1367581919',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=c74b7878-4936-4015-8b63-142923947310,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='8f0c848b490b47629051e961c682c5a5',provider_auth='CHAP PUvXoCNO 1H8kFoBshXZL64',provider_geometry=None,provider_id=None,provider_location='172.16.0.220:3260,1 iqn.2010-10.org.openstack:c74b7878-4936-4015-8b63-142923947310 0',replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:24:01Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='downloading',terminated_at=None,updated_at=2023-04-17T15:24:01Z,use_quota=True,user_id='3e4b069991954c57993051fbe8665d35',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c) from target:iqn.2010-10.org.openstack:c74b7878-4936-4015-8b63-142923947310. {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:913}} Apr 17 15:24:26 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] detach volume v_c74b7878-4936-4015-8b63-142923947310 from target iqn.2010-10.org.openstack:c74b7878-4936-4015-8b63-142923947310 {{(pid=84696) detach_target_vol /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:606}} Apr 17 15:24:26 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] Sending pool request DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:c74b7878-4936-4015-8b63-142923947310/luns/v_c74b7878-4936-4015-8b63-142923947310 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:24:26 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] Sending DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:c74b7878-4936-4015-8b63-142923947310/luns/v_c74b7878-4936-4015-8b63-142923947310 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:24:26 user cinder-volume[84696]: DEBUG cinder.quota [None req-93e81254-daef-413d-949e-6eef52a40357 tempest-GroupSnapshotsTest-1926061586 None] Created reservations ['b7c1d06c-29f1-4df1-8198-42126cd55df3', 'c9294f0e-0da3-45a1-9243-4e062ecd4d07', '6f03eb8b-72c2-49b3-b043-e6286d9cf187', 'ff58f4fa-d815-4afa-bca9-5697605bde21'] {{(pid=84696) reserve /opt/stack/cinder/cinder/quota.py:755}} Apr 17 15:24:26 user cinder-volume[84696]: INFO cinder.volume.manager [None req-93e81254-daef-413d-949e-6eef52a40357 tempest-GroupSnapshotsTest-1926061586 None] group_snapshot aa6864c4-c754-46e2-a4a4-18248ad70055: deleted successfully Apr 17 15:24:26 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-d360ab42-9868-4c50-8361-153e346a5879 tempest-VolumesTransfersTest-276179586 None] Volume reschedule parameters: True retry: {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 15:24:26 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-d360ab42-9868-4c50-8361-153e346a5879 tempest-VolumesTransfersTest-276179586 None] Flow 'volume_create_manager' (f1c909e6-32ec-4815-8190-82d7d55b978e) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:24:26 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-d360ab42-9868-4c50-8361-153e346a5879 tempest-VolumesTransfersTest-276179586 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (263c0d98-130b-4166-9cea-c83c21b95b92) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:24:26 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-d360ab42-9868-4c50-8361-153e346a5879 tempest-VolumesTransfersTest-276179586 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (263c0d98-130b-4166-9cea-c83c21b95b92) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:24:26Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesTransfersTest-Volume-1918704143',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=bd3bcf59-72e2-43c9-973a-ab5956dc83b5,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='8b0ce9e109f84b6988b4adf66910910c',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:24:26Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:24:26Z,use_quota=True,user_id='bee3a4dd32ab4d0ba197cf1e28638290',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:24:26 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-d360ab42-9868-4c50-8361-153e346a5879 tempest-VolumesTransfersTest-276179586 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (14732a5d-c5f7-467a-80e4-00fca3c6c60f) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:24:26 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-d360ab42-9868-4c50-8361-153e346a5879 tempest-VolumesTransfersTest-276179586 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (14732a5d-c5f7-467a-80e4-00fca3c6c60f) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:24:26 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-d360ab42-9868-4c50-8361-153e346a5879 tempest-VolumesTransfersTest-276179586 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (1143755f-fb94-479f-bd61-5bc9f87d0457) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:24:26 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-d360ab42-9868-4c50-8361-153e346a5879 tempest-VolumesTransfersTest-276179586 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (1143755f-fb94-479f-bd61-5bc9f87d0457) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': 'bd3bcf59-72e2-43c9-973a-ab5956dc83b5', 'volume_name': 'volume-bd3bcf59-72e2-43c9-973a-ab5956dc83b5', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:24:26 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-d360ab42-9868-4c50-8361-153e346a5879 tempest-VolumesTransfersTest-276179586 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (f33a2e27-2b09-4f15-b003-820114a39810) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:24:26 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-d360ab42-9868-4c50-8361-153e346a5879 tempest-VolumesTransfersTest-276179586 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (f33a2e27-2b09-4f15-b003-820114a39810) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:24:26 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-d360ab42-9868-4c50-8361-153e346a5879 tempest-VolumesTransfersTest-276179586 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (42c066b6-c674-445a-bfa2-2a4ff975d963) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:24:26 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-d360ab42-9868-4c50-8361-153e346a5879 tempest-VolumesTransfersTest-276179586 None] Volume bd3bcf59-72e2-43c9-973a-ab5956dc83b5: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-bd3bcf59-72e2-43c9-973a-ab5956dc83b5', 'volume_size': 1} Apr 17 15:24:26 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-d360ab42-9868-4c50-8361-153e346a5879 tempest-VolumesTransfersTest-276179586 None] creating volume v_bd3bcf59-72e2-43c9-973a-ab5956dc83b5. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 15:24:26 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-d360ab42-9868-4c50-8361-153e346a5879 tempest-VolumesTransfersTest-276179586 None] create volume {'name': 'v_bd3bcf59-72e2-43c9-973a-ab5956dc83b5', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 15:24:26 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-d360ab42-9868-4c50-8361-153e346a5879 tempest-VolumesTransfersTest-276179586 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:24:26 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-d360ab42-9868-4c50-8361-153e346a5879 tempest-VolumesTransfersTest-276179586 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:24:26 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] Geting {'code': 204} from DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:c74b7878-4936-4015-8b63-142923947310/luns/v_c74b7878-4936-4015-8b63-142923947310 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:24:26 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] delete target: iqn.2010-10.org.openstack:c74b7878-4936-4015-8b63-142923947310 {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:930}} Apr 17 15:24:26 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] delete target iqn.2010-10.org.openstack:c74b7878-4936-4015-8b63-142923947310 {{(pid=84696) delete_target /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:442}} Apr 17 15:24:26 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] Sending pool request DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:c74b7878-4936-4015-8b63-142923947310 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:24:26 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] Sending DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:c74b7878-4936-4015-8b63-142923947310 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:24:27 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] Geting {'code': 204} from DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:c74b7878-4936-4015-8b63-142923947310 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:24:27 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] target iqn.2010-10.org.openstack:c74b7878-4936-4015-8b63-142923947310 deleted {{(pid=84696) delete_target /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:447}} Apr 17 15:24:27 user cinder-volume[84696]: DEBUG cinder.volume.volume_utils [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] Downloaded image b3f0dec1-7758-44a8-82e0-76ada8837428 ((None, None)) to volume c74b7878-4936-4015-8b63-142923947310 successfully. {{(pid=84696) copy_image_to_volume /opt/stack/cinder/cinder/volume/volume_utils.py:1232}} Apr 17 15:24:27 user cinder-volume[84696]: DEBUG cinder.image.image_utils [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] Temporary image b3f0dec1-7758-44a8-82e0-76ada8837428 for user 3e4b069991954c57993051fbe8665d35 is deleted. {{(pid=84696) fetch /opt/stack/cinder/cinder/image/image_utils.py:1283}} Apr 17 15:24:27 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-c3c39056-e984-4f53-8f45-8ecd6577db98 tempest-VolumesGetTest-1961901732 None] Geting {'code': 201, 'error': None, 'data': {'full_name': 'Pool-0/v_9a8586a6-f2f5-4bee-a56f-a06b01120c39', 'is_clone': False, 'name': 'v_9a8586a6-f2f5-4bee-a56f-a06b01120c39'}} from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:24:27 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-c3c39056-e984-4f53-8f45-8ecd6577db98 tempest-VolumesGetTest-1961901732 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (4aa285a1-322a-427f-a375-910ac5cdb6cf) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'volume_name': 'volume-9a8586a6-f2f5-4bee-a56f-a06b01120c39', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:24:27 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-c3c39056-e984-4f53-8f45-8ecd6577db98 tempest-VolumesGetTest-1961901732 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeOnFinishTask;volume:create, create.end' (bc335d5a-77f6-4571-9bb0-5ed3cc2f8ee9) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:24:27 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-c3c39056-e984-4f53-8f45-8ecd6577db98 tempest-VolumesGetTest-1961901732 None] Volume volume-9a8586a6-f2f5-4bee-a56f-a06b01120c39 (9a8586a6-f2f5-4bee-a56f-a06b01120c39): created successfully Apr 17 15:24:27 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-c3c39056-e984-4f53-8f45-8ecd6577db98 tempest-VolumesGetTest-1961901732 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeOnFinishTask;volume:create, create.end' (bc335d5a-77f6-4571-9bb0-5ed3cc2f8ee9) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:24:27 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-c3c39056-e984-4f53-8f45-8ecd6577db98 tempest-VolumesGetTest-1961901732 None] Flow 'volume_create_manager' (e5f4bfd4-9487-4f0c-a4dc-f6a515c75814) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:24:27 user cinder-volume[84696]: INFO cinder.volume.manager [None req-c3c39056-e984-4f53-8f45-8ecd6577db98 tempest-VolumesGetTest-1961901732 None] Created volume successfully. Apr 17 15:24:27 user cinder-volume[84696]: DEBUG cinder.quota [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] Created reservations ['4947b108-8a15-41a7-bc8e-12d588257eaf', '98a2ea9f-ec1a-429d-a9ff-bb996b8bd5b9', '46f60cfd-089c-4054-96e0-e3105216d457', '769f1997-7834-45fc-a7fd-12eef6d9faba'] {{(pid=84696) reserve /opt/stack/cinder/cinder/quota.py:755}} Apr 17 15:24:27 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] Creating image volume entry: {'launched_at': None, 'user_id': '3e4b069991954c57993051fbe8665d35', 'cluster_name': None, 'deleted': False, 'scheduled_at': datetime.datetime(2023, 4, 17, 15, 24, 1, tzinfo=datetime.timezone.utc), 'encryption_key_id': None, 'replication_status': None, 'display_description': None, 'availability_zone': 'nova', 'replication_extended_status': None, 'size': 1, 'display_name': 'image-b3f0dec1-7758-44a8-82e0-76ada8837428', 'deleted_at': None, 'provider_id': None, 'ec2_id': None, 'created_at': datetime.datetime(2023, 4, 17, 15, 24, 1, tzinfo=datetime.timezone.utc), 'volume_admin_metadata': [], 'project_id': '2ef07a6150b7438d850c124674d4e245', 'volume_metadata': [], 'group_id': None, 'service_uuid': None, 'provider_geometry': None, 'host': 'user@joviandss-2#joviandss-2', 'provider_auth': 'CHAP PUvXoCNO 1H8kFoBshXZL64', 'consistencygroup_id': None, 'previous_status': None, 'updated_at': datetime.datetime(2023, 4, 17, 15, 24, 1, tzinfo=datetime.timezone.utc), 'shared_targets': True, 'multiattach': False, 'metadata': {}, 'replication_driver_data': None, 'volume_glance_metadata': [{'key': 'signature_verified', 'value': 'False'}], 'bootable': False, 'terminated_at': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'provider_location': '172.16.0.220:3260,1 iqn.2010-10.org.openstack:c74b7878-4936-4015-8b63-142923947310 0', 'attach_status': 'detached', 'status': 'creating', 'source_volid': 'c74b7878-4936-4015-8b63-142923947310'}. {{(pid=84696) _clone_image_volume /opt/stack/cinder/cinder/volume/manager.py:1660}} Apr 17 15:24:27 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] Volume reschedule parameters: False retry: None {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 15:24:27 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] Flow 'volume_create_manager' (b0f8b0ec-ee47-4e74-8b4c-ee08c977d4b0) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:24:27 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (c9758109-be4d-4948-aeba-ae88ee20fdda) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:24:27 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (c9758109-be4d-4948-aeba-ae88ee20fdda) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata={},attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:24:01Z,deleted=False,deleted_at=None,display_description=None,display_name='image-b3f0dec1-7758-44a8-82e0-76ada8837428',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=006f20ae-5306-44fe-afdf-e433f79284e8,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='2ef07a6150b7438d850c124674d4e245',provider_auth='CHAP PUvXoCNO 1H8kFoBshXZL64',provider_geometry=None,provider_id=None,provider_location='172.16.0.220:3260,1 iqn.2010-10.org.openstack:c74b7878-4936-4015-8b63-142923947310 0',replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:24:01Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=c74b7878-4936-4015-8b63-142923947310,status='creating',terminated_at=None,updated_at=2023-04-17T15:24:01Z,use_quota=True,user_id='3e4b069991954c57993051fbe8665d35',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:24:27 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (755193be-08d2-4e87-8b74-64475eace3bb) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:24:27 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (755193be-08d2-4e87-8b74-64475eace3bb) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:24:27 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (3dc5257f-85c3-4793-bffb-5c4fbc35545a) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:24:27 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (3dc5257f-85c3-4793-bffb-5c4fbc35545a) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'source_vol', 'volume_id': '006f20ae-5306-44fe-afdf-e433f79284e8', 'volume_name': 'volume-006f20ae-5306-44fe-afdf-e433f79284e8', 'volume_size': 1, 'source_volid': 'c74b7878-4936-4015-8b63-142923947310', 'source_volstatus': 'downloading'}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:24:27 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (919bdf5f-3aa2-4912-9c62-5291ae3ce116) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:24:27 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (919bdf5f-3aa2-4912-9c62-5291ae3ce116) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:24:27 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (56d3623a-20ae-4488-a3c9-512da4dab680) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:24:27 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] Volume 006f20ae-5306-44fe-afdf-e433f79284e8: being created as source_vol with specification: {'status': 'creating', 'volume_name': 'volume-006f20ae-5306-44fe-afdf-e433f79284e8', 'volume_size': 1, 'source_volid': 'c74b7878-4936-4015-8b63-142923947310', 'source_volstatus': 'downloading'} Apr 17 15:24:27 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-cb0bf660-4169-4302-b2c3-1ebf4db895fa tempest-GroupSnapshotsTest-1926061586 None] remove_export for volume: 7249f8e4-ed4b-4689-aba3-2766c924f42c. {{(pid=84696) remove_export /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:718}} Apr 17 15:24:27 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-cb0bf660-4169-4302-b2c3-1ebf4db895fa tempest-GroupSnapshotsTest-1926061586 None] remove export {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:912}} Apr 17 15:24:27 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-cb0bf660-4169-4302-b2c3-1ebf4db895fa tempest-GroupSnapshotsTest-1926061586 None] detach volume:Volume(_name_id=None,admin_metadata={},attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:23:53Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-volume-231798850',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=286e53b0-285a-4d83-8cd3-cb2b5641b31f,host='user@joviandss-2#joviandss-2',id=7249f8e4-ed4b-4689-aba3-2766c924f42c,launched_at=2023-04-17T15:23:55Z,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='6cae12f3c2614d6fb32f21db60fec8d3',provider_auth='CHAP WRdnaIOw 0Cy26Psq1dOj3B',provider_geometry=None,provider_id=None,provider_location='172.16.0.220:3260,1 iqn.2010-10.org.openstack:7249f8e4-ed4b-4689-aba3-2766c924f42c 0',replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:23:53Z,service_uuid='e628352e-c4cf-40eb-8389-cce40663026a',shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='deleting',terminated_at=None,updated_at=2023-04-17T15:24:28Z,use_quota=True,user_id='560c850c324949d7a214e7a59dd65f8d',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(73c9cbe6-d3bc-4af2-810c-855e0b88bd3f),volume_type_id=73c9cbe6-d3bc-4af2-810c-855e0b88bd3f) from target:iqn.2010-10.org.openstack:7249f8e4-ed4b-4689-aba3-2766c924f42c. {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:913}} Apr 17 15:24:27 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-cb0bf660-4169-4302-b2c3-1ebf4db895fa tempest-GroupSnapshotsTest-1926061586 None] detach volume v_7249f8e4-ed4b-4689-aba3-2766c924f42c from target iqn.2010-10.org.openstack:7249f8e4-ed4b-4689-aba3-2766c924f42c {{(pid=84696) detach_target_vol /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:606}} Apr 17 15:24:27 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-cb0bf660-4169-4302-b2c3-1ebf4db895fa tempest-GroupSnapshotsTest-1926061586 None] Sending pool request DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:7249f8e4-ed4b-4689-aba3-2766c924f42c/luns/v_7249f8e4-ed4b-4689-aba3-2766c924f42c {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:24:27 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-cb0bf660-4169-4302-b2c3-1ebf4db895fa tempest-GroupSnapshotsTest-1926061586 None] Sending DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:7249f8e4-ed4b-4689-aba3-2766c924f42c/luns/v_7249f8e4-ed4b-4689-aba3-2766c924f42c {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:24:27 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] cloned volume c74b7878-4936-4015-8b63-142923947310 to 006f20ae-5306-44fe-afdf-e433f79284e8 {{(pid=84696) create_cloned_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:521}} Apr 17 15:24:27 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] cloning v_c74b7878-4936-4015-8b63-142923947310 to v_006f20ae-5306-44fe-afdf-e433f79284e8 {{(pid=84696) _clone_object /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:456}} Apr 17 15:24:27 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] create snapshot v_006f20ae-5306-44fe-afdf-e433f79284e8 {{(pid=84696) create_snapshot /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:635}} Apr 17 15:24:27 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_c74b7878-4936-4015-8b63-142923947310/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:24:27 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_c74b7878-4936-4015-8b63-142923947310/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:24:27 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-cb0bf660-4169-4302-b2c3-1ebf4db895fa tempest-GroupSnapshotsTest-1926061586 None] Geting {'code': 404, 'error': {'class': 'opene.exceptions.ItemNotFoundError', 'message': 'Target iqn.2010-10.org.openstack:7249f8e4-ed4b-4689-aba3-2766c924f42c not exists.', 'url': 'http://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:7249f8e4-ed4b-4689-aba3-2766c924f42c/luns/v_7249f8e4-ed4b-4689-aba3-2766c924f42c'}, 'data': None} from DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:7249f8e4-ed4b-4689-aba3-2766c924f42c/luns/v_7249f8e4-ed4b-4689-aba3-2766c924f42c {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:24:27 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-cb0bf660-4169-4302-b2c3-1ebf4db895fa tempest-GroupSnapshotsTest-1926061586 None] failed to remove resource iqn.2010-10.org.openstack:7249f8e4-ed4b-4689-aba3-2766c924f42c because of JDSS resource v_7249f8e4-ed4b-4689-aba3-2766c924f42c DNE. {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:920}} Apr 17 15:24:27 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-cb0bf660-4169-4302-b2c3-1ebf4db895fa tempest-GroupSnapshotsTest-1926061586 None] delete target: iqn.2010-10.org.openstack:7249f8e4-ed4b-4689-aba3-2766c924f42c {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:930}} Apr 17 15:24:27 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-cb0bf660-4169-4302-b2c3-1ebf4db895fa tempest-GroupSnapshotsTest-1926061586 None] delete target iqn.2010-10.org.openstack:7249f8e4-ed4b-4689-aba3-2766c924f42c {{(pid=84696) delete_target /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:442}} Apr 17 15:24:27 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-cb0bf660-4169-4302-b2c3-1ebf4db895fa tempest-GroupSnapshotsTest-1926061586 None] Sending pool request DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:7249f8e4-ed4b-4689-aba3-2766c924f42c {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:24:27 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-cb0bf660-4169-4302-b2c3-1ebf4db895fa tempest-GroupSnapshotsTest-1926061586 None] Sending DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:7249f8e4-ed4b-4689-aba3-2766c924f42c {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:24:28 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-cb0bf660-4169-4302-b2c3-1ebf4db895fa tempest-GroupSnapshotsTest-1926061586 None] Geting {'code': 404, 'error': {'class': 'opene.exceptions.ItemNotFoundError', 'message': 'Target iqn.2010-10.org.openstack:7249f8e4-ed4b-4689-aba3-2766c924f42c not exists.', 'url': 'http://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:7249f8e4-ed4b-4689-aba3-2766c924f42c'}, 'data': None} from DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:7249f8e4-ed4b-4689-aba3-2766c924f42c {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:24:28 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-cb0bf660-4169-4302-b2c3-1ebf4db895fa tempest-GroupSnapshotsTest-1926061586 None] failed to remove resource iqn.2010-10.org.openstack:7249f8e4-ed4b-4689-aba3-2766c924f42c because of JDSS resource iqn.2010-10.org.openstack:7249f8e4-ed4b-4689-aba3-2766c924f42c DNE. {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:935}} Apr 17 15:24:28 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-cb0bf660-4169-4302-b2c3-1ebf4db895fa tempest-GroupSnapshotsTest-1926061586 None] deleating volume v_7249f8e4-ed4b-4689-aba3-2766c924f42c {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 15:24:28 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-cb0bf660-4169-4302-b2c3-1ebf4db895fa tempest-GroupSnapshotsTest-1926061586 None] get snapshots for volume v_7249f8e4-ed4b-4689-aba3-2766c924f42c {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 15:24:28 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-cb0bf660-4169-4302-b2c3-1ebf4db895fa tempest-GroupSnapshotsTest-1926061586 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_7249f8e4-ed4b-4689-aba3-2766c924f42c/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:24:28 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-cb0bf660-4169-4302-b2c3-1ebf4db895fa tempest-GroupSnapshotsTest-1926061586 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_7249f8e4-ed4b-4689-aba3-2766c924f42c/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:24:28 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-d360ab42-9868-4c50-8361-153e346a5879 tempest-VolumesTransfersTest-276179586 None] Geting {'code': 201, 'error': None, 'data': {'full_name': 'Pool-0/v_bd3bcf59-72e2-43c9-973a-ab5956dc83b5', 'is_clone': False, 'name': 'v_bd3bcf59-72e2-43c9-973a-ab5956dc83b5'}} from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:24:28 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-d360ab42-9868-4c50-8361-153e346a5879 tempest-VolumesTransfersTest-276179586 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (42c066b6-c674-445a-bfa2-2a4ff975d963) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'volume_name': 'volume-bd3bcf59-72e2-43c9-973a-ab5956dc83b5', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:24:28 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-d360ab42-9868-4c50-8361-153e346a5879 tempest-VolumesTransfersTest-276179586 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeOnFinishTask;volume:create, create.end' (b22935e3-29ad-4948-a579-db7d9cbcaf40) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:24:28 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-d360ab42-9868-4c50-8361-153e346a5879 tempest-VolumesTransfersTest-276179586 None] Volume volume-bd3bcf59-72e2-43c9-973a-ab5956dc83b5 (bd3bcf59-72e2-43c9-973a-ab5956dc83b5): created successfully Apr 17 15:24:28 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-d360ab42-9868-4c50-8361-153e346a5879 tempest-VolumesTransfersTest-276179586 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeOnFinishTask;volume:create, create.end' (b22935e3-29ad-4948-a579-db7d9cbcaf40) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:24:28 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-d360ab42-9868-4c50-8361-153e346a5879 tempest-VolumesTransfersTest-276179586 None] Flow 'volume_create_manager' (f1c909e6-32ec-4815-8190-82d7d55b978e) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:24:28 user cinder-volume[84696]: INFO cinder.volume.manager [None req-d360ab42-9868-4c50-8361-153e346a5879 tempest-VolumesTransfersTest-276179586 None] Created volume successfully. Apr 17 15:24:28 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-ffc97846-e2d4-4147-ab93-bc7ab1830c90 tempest-VolumesGetTest-1961901732 None] Volume reschedule parameters: True retry: {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 15:24:28 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-ffc97846-e2d4-4147-ab93-bc7ab1830c90 tempest-VolumesGetTest-1961901732 None] Flow 'volume_create_manager' (d530dba1-3eb3-4477-af75-0538fdc7f6d4) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:24:28 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-ffc97846-e2d4-4147-ab93-bc7ab1830c90 tempest-VolumesGetTest-1961901732 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (7547c076-f357-4e6b-88ae-1042312658f2) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:24:28 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-ffc97846-e2d4-4147-ab93-bc7ab1830c90 tempest-VolumesGetTest-1961901732 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (7547c076-f357-4e6b-88ae-1042312658f2) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:24:28Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesGetTest-Volume-1605278566',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=fa8e71d4-54a7-4e0d-9a26-50cc7946ce11,launched_at=None,metadata={Type='Test'},migration_status=None,multiattach=False,previous_status=None,project_id='a65d14b0a2894a6f90506d86feaefd15',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:24:29Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=9a8586a6-f2f5-4bee-a56f-a06b01120c39,status='creating',terminated_at=None,updated_at=2023-04-17T15:24:29Z,use_quota=True,user_id='6ffa3051570a4721a48a0171b3cc4d16',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:24:28 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-ffc97846-e2d4-4147-ab93-bc7ab1830c90 tempest-VolumesGetTest-1961901732 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (8b274a73-1080-4a00-876c-112e10a02781) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:24:28 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-ffc97846-e2d4-4147-ab93-bc7ab1830c90 tempest-VolumesGetTest-1961901732 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (8b274a73-1080-4a00-876c-112e10a02781) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:24:28 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-ffc97846-e2d4-4147-ab93-bc7ab1830c90 tempest-VolumesGetTest-1961901732 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (c3d22515-fe31-481b-9aeb-26467980c194) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:24:28 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-ffc97846-e2d4-4147-ab93-bc7ab1830c90 tempest-VolumesGetTest-1961901732 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (c3d22515-fe31-481b-9aeb-26467980c194) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'source_vol', 'volume_id': 'fa8e71d4-54a7-4e0d-9a26-50cc7946ce11', 'volume_name': 'volume-fa8e71d4-54a7-4e0d-9a26-50cc7946ce11', 'volume_size': 1, 'source_volid': '9a8586a6-f2f5-4bee-a56f-a06b01120c39', 'source_volstatus': 'available'}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:24:28 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-ffc97846-e2d4-4147-ab93-bc7ab1830c90 tempest-VolumesGetTest-1961901732 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (64ba181d-bb81-43ee-bb2e-45bad57bc7e7) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:24:28 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-ffc97846-e2d4-4147-ab93-bc7ab1830c90 tempest-VolumesGetTest-1961901732 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (64ba181d-bb81-43ee-bb2e-45bad57bc7e7) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:24:28 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-ffc97846-e2d4-4147-ab93-bc7ab1830c90 tempest-VolumesGetTest-1961901732 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (e230de85-0720-498a-805a-b1d9ebb28db9) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:24:28 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-ffc97846-e2d4-4147-ab93-bc7ab1830c90 tempest-VolumesGetTest-1961901732 None] Volume fa8e71d4-54a7-4e0d-9a26-50cc7946ce11: being created as source_vol with specification: {'status': 'creating', 'volume_name': 'volume-fa8e71d4-54a7-4e0d-9a26-50cc7946ce11', 'volume_size': 1, 'source_volid': '9a8586a6-f2f5-4bee-a56f-a06b01120c39', 'source_volstatus': 'available'} Apr 17 15:24:28 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-ffc97846-e2d4-4147-ab93-bc7ab1830c90 tempest-VolumesGetTest-1961901732 None] cloned volume 9a8586a6-f2f5-4bee-a56f-a06b01120c39 to fa8e71d4-54a7-4e0d-9a26-50cc7946ce11 {{(pid=84696) create_cloned_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:521}} Apr 17 15:24:28 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-ffc97846-e2d4-4147-ab93-bc7ab1830c90 tempest-VolumesGetTest-1961901732 None] cloning v_9a8586a6-f2f5-4bee-a56f-a06b01120c39 to v_fa8e71d4-54a7-4e0d-9a26-50cc7946ce11 {{(pid=84696) _clone_object /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:456}} Apr 17 15:24:28 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-ffc97846-e2d4-4147-ab93-bc7ab1830c90 tempest-VolumesGetTest-1961901732 None] create snapshot v_fa8e71d4-54a7-4e0d-9a26-50cc7946ce11 {{(pid=84696) create_snapshot /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:635}} Apr 17 15:24:28 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-ffc97846-e2d4-4147-ab93-bc7ab1830c90 tempest-VolumesGetTest-1961901732 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_9a8586a6-f2f5-4bee-a56f-a06b01120c39/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:24:28 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-ffc97846-e2d4-4147-ab93-bc7ab1830c90 tempest-VolumesGetTest-1961901732 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_9a8586a6-f2f5-4bee-a56f-a06b01120c39/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:24:28 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-cb0bf660-4169-4302-b2c3-1ebf4db895fa tempest-GroupSnapshotsTest-1926061586 None] Geting {'code': 200, 'error': None, 'data': {'results': 0, 'entries': []}} from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_7249f8e4-ed4b-4689-aba3-2766c924f42c/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:24:28 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-cb0bf660-4169-4302-b2c3-1ebf4db895fa tempest-GroupSnapshotsTest-1926061586 None] get volume v_7249f8e4-ed4b-4689-aba3-2766c924f42c info {{(pid=84696) get_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:239}} Apr 17 15:24:28 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-cb0bf660-4169-4302-b2c3-1ebf4db895fa tempest-GroupSnapshotsTest-1926061586 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_7249f8e4-ed4b-4689-aba3-2766c924f42c {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:24:28 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-cb0bf660-4169-4302-b2c3-1ebf4db895fa tempest-GroupSnapshotsTest-1926061586 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_7249f8e4-ed4b-4689-aba3-2766c924f42c {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:24:28 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-cb0bf660-4169-4302-b2c3-1ebf4db895fa tempest-GroupSnapshotsTest-1926061586 None] Geting {'code': 200, 'error': None, 'data': {'origin': None, 'relatime': None, 'acltype': None, 'vscan': None, 'full_name': 'Pool-0/v_7249f8e4-ed4b-4689-aba3-2766c924f42c', 'userrefs': None, 'primarycache': 'all', 'logbias': 'latency', 'creation': '1681746427', 'sync': 'always', 'is_clone': False, 'dedup': 'off', 'sharenfs': None, 'receive_resume_token': None, 'volsize': '1073741824', 'referenced': '57344', 'sharesmb': None, 'createtxg': '187836', 'reservation': '0', 'scontext': None, 'mountpoint': None, 'casesensitivity': None, 'guid': '14143552662154981910', 'usedbyrefreservation': '0', 'dnodesize': None, 'written': '57344', 'logicalused': '28672', 'compressratio': '1.00', 'rootcontext': 'none', 'default_scsi_id': '9903bf0cec764c49', 'type': 'volume', 'compression': 'lz4', 'snapdir': None, 'overlay': None, 'encryption': 'off', 'xattr': None, 'volmode': 'default', 'copies': '1', 'snapshot_limit': '18446744073709551615', 'aclinherit': None, 'defcontext': 'none', 'readonly': 'off', 'version': None, 'recordsize': None, 'filesystem_limit': None, 'mounted': None, 'mlslabel': 'none', 'secondarycache': 'all', 'refreservation': '0', 'available': '113909948416', 'san:volume_id': '9903bf0cec764c498f02fea7c2be8aa61da26ec7', 'encryptionroot': None, 'exec': None, 'refquota': None, 'refcompressratio': '1.00', 'quota': None, 'utf8only': None, 'keylocation': 'none', 'snapdev': 'hidden', 'snapshot_count': '18446744073709551615', 'fscontext': 'none', 'clones': None, 'canmount': None, 'keystatus': None, 'atime': None, 'usedbysnapshots': '0', 'normalization': None, 'usedbychildren': '0', 'volblocksize': '131072', 'usedbydataset': '57344', 'objsetid': '121833', 'name': 'v_7249f8e4-ed4b-4689-aba3-2766c924f42c', 'defer_destroy': None, 'pbkdf2iters': '0', 'checksum': 'on', 'redundant_metadata': 'all', 'filesystem_count': None, 'devices': None, 'keyformat': 'none', 'setuid': None, 'used': '57344', 'logicalreferenced': '28672', 'context': 'none', 'zoned': None, 'nbmand': None}} from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_7249f8e4-ed4b-4689-aba3-2766c924f42c {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:24:28 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-cb0bf660-4169-4302-b2c3-1ebf4db895fa tempest-GroupSnapshotsTest-1926061586 None] delete volume:v_7249f8e4-ed4b-4689-aba3-2766c924f42c recursively children:{'force_umount': True} {{(pid=84696) delete_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:335}} Apr 17 15:24:28 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-cb0bf660-4169-4302-b2c3-1ebf4db895fa tempest-GroupSnapshotsTest-1926061586 None] Sending pool request DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_7249f8e4-ed4b-4689-aba3-2766c924f42c {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:24:28 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-cb0bf660-4169-4302-b2c3-1ebf4db895fa tempest-GroupSnapshotsTest-1926061586 None] Sending DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_7249f8e4-ed4b-4689-aba3-2766c924f42c {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:24:29 user cinder-volume[84696]: DEBUG cinder.coordination [None req-733c2b3a-fb60-4425-8809-eceac0525109 tempest-VolumesTransfersTest-450487945 None] Acquiring lock "cinder-bd3bcf59-72e2-43c9-973a-ab5956dc83b5-delete_volume" by "delete_volume" {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:192}} Apr 17 15:24:29 user cinder-volume[84696]: DEBUG cinder.coordination [None req-733c2b3a-fb60-4425-8809-eceac0525109 tempest-VolumesTransfersTest-450487945 None] Lock "cinder-bd3bcf59-72e2-43c9-973a-ab5956dc83b5-delete_volume" acquired by "delete_volume" :: waited 0.015s {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:196}} Apr 17 15:24:29 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-733c2b3a-fb60-4425-8809-eceac0525109 tempest-VolumesTransfersTest-450487945 None] remove_export for volume: bd3bcf59-72e2-43c9-973a-ab5956dc83b5. {{(pid=84696) remove_export /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:718}} Apr 17 15:24:29 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-733c2b3a-fb60-4425-8809-eceac0525109 tempest-VolumesTransfersTest-450487945 None] remove export {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:912}} Apr 17 15:24:29 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-733c2b3a-fb60-4425-8809-eceac0525109 tempest-VolumesTransfersTest-450487945 None] detach volume:Volume(_name_id=None,admin_metadata={},attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:24:26Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesTransfersTest-Volume-1918704143',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=bd3bcf59-72e2-43c9-973a-ab5956dc83b5,launched_at=2023-04-17T15:24:28Z,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='8b0ce9e109f84b6988b4adf66910910c',provider_auth='CHAP hAZtnPjU tbgIlNdTqQoS9A',provider_geometry=None,provider_id=None,provider_location='172.16.0.220:3260,1 iqn.2010-10.org.openstack:bd3bcf59-72e2-43c9-973a-ab5956dc83b5 0',replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:24:26Z,service_uuid='e628352e-c4cf-40eb-8389-cce40663026a',shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='deleting',terminated_at=2023-04-17T15:24:29Z,updated_at=2023-04-17T15:24:29Z,use_quota=True,user_id='bee3a4dd32ab4d0ba197cf1e28638290',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c) from target:iqn.2010-10.org.openstack:bd3bcf59-72e2-43c9-973a-ab5956dc83b5. {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:913}} Apr 17 15:24:29 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-733c2b3a-fb60-4425-8809-eceac0525109 tempest-VolumesTransfersTest-450487945 None] detach volume v_bd3bcf59-72e2-43c9-973a-ab5956dc83b5 from target iqn.2010-10.org.openstack:bd3bcf59-72e2-43c9-973a-ab5956dc83b5 {{(pid=84696) detach_target_vol /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:606}} Apr 17 15:24:29 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-733c2b3a-fb60-4425-8809-eceac0525109 tempest-VolumesTransfersTest-450487945 None] Sending pool request DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:bd3bcf59-72e2-43c9-973a-ab5956dc83b5/luns/v_bd3bcf59-72e2-43c9-973a-ab5956dc83b5 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:24:29 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-733c2b3a-fb60-4425-8809-eceac0525109 tempest-VolumesTransfersTest-450487945 None] Sending DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:bd3bcf59-72e2-43c9-973a-ab5956dc83b5/luns/v_bd3bcf59-72e2-43c9-973a-ab5956dc83b5 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:24:29 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-733c2b3a-fb60-4425-8809-eceac0525109 tempest-VolumesTransfersTest-450487945 None] Geting {'code': 404, 'error': {'class': 'opene.exceptions.ItemNotFoundError', 'message': 'Target iqn.2010-10.org.openstack:bd3bcf59-72e2-43c9-973a-ab5956dc83b5 not exists.', 'url': 'http://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:bd3bcf59-72e2-43c9-973a-ab5956dc83b5/luns/v_bd3bcf59-72e2-43c9-973a-ab5956dc83b5'}, 'data': None} from DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:bd3bcf59-72e2-43c9-973a-ab5956dc83b5/luns/v_bd3bcf59-72e2-43c9-973a-ab5956dc83b5 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:24:29 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-733c2b3a-fb60-4425-8809-eceac0525109 tempest-VolumesTransfersTest-450487945 None] failed to remove resource iqn.2010-10.org.openstack:bd3bcf59-72e2-43c9-973a-ab5956dc83b5 because of JDSS resource v_bd3bcf59-72e2-43c9-973a-ab5956dc83b5 DNE. {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:920}} Apr 17 15:24:29 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-733c2b3a-fb60-4425-8809-eceac0525109 tempest-VolumesTransfersTest-450487945 None] delete target: iqn.2010-10.org.openstack:bd3bcf59-72e2-43c9-973a-ab5956dc83b5 {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:930}} Apr 17 15:24:29 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-733c2b3a-fb60-4425-8809-eceac0525109 tempest-VolumesTransfersTest-450487945 None] delete target iqn.2010-10.org.openstack:bd3bcf59-72e2-43c9-973a-ab5956dc83b5 {{(pid=84696) delete_target /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:442}} Apr 17 15:24:29 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-733c2b3a-fb60-4425-8809-eceac0525109 tempest-VolumesTransfersTest-450487945 None] Sending pool request DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:bd3bcf59-72e2-43c9-973a-ab5956dc83b5 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:24:29 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-733c2b3a-fb60-4425-8809-eceac0525109 tempest-VolumesTransfersTest-450487945 None] Sending DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:bd3bcf59-72e2-43c9-973a-ab5956dc83b5 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:24:29 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-733c2b3a-fb60-4425-8809-eceac0525109 tempest-VolumesTransfersTest-450487945 None] Geting {'code': 404, 'error': {'class': 'opene.exceptions.ItemNotFoundError', 'message': 'Target iqn.2010-10.org.openstack:bd3bcf59-72e2-43c9-973a-ab5956dc83b5 not exists.', 'url': 'http://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:bd3bcf59-72e2-43c9-973a-ab5956dc83b5'}, 'data': None} from DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:bd3bcf59-72e2-43c9-973a-ab5956dc83b5 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:24:29 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-733c2b3a-fb60-4425-8809-eceac0525109 tempest-VolumesTransfersTest-450487945 None] failed to remove resource iqn.2010-10.org.openstack:bd3bcf59-72e2-43c9-973a-ab5956dc83b5 because of JDSS resource iqn.2010-10.org.openstack:bd3bcf59-72e2-43c9-973a-ab5956dc83b5 DNE. {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:935}} Apr 17 15:24:29 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-733c2b3a-fb60-4425-8809-eceac0525109 tempest-VolumesTransfersTest-450487945 None] deleating volume v_bd3bcf59-72e2-43c9-973a-ab5956dc83b5 {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 15:24:29 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-733c2b3a-fb60-4425-8809-eceac0525109 tempest-VolumesTransfersTest-450487945 None] get snapshots for volume v_bd3bcf59-72e2-43c9-973a-ab5956dc83b5 {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 15:24:29 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-733c2b3a-fb60-4425-8809-eceac0525109 tempest-VolumesTransfersTest-450487945 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_bd3bcf59-72e2-43c9-973a-ab5956dc83b5/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:24:29 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-733c2b3a-fb60-4425-8809-eceac0525109 tempest-VolumesTransfersTest-450487945 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_bd3bcf59-72e2-43c9-973a-ab5956dc83b5/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:24:29 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] Geting {'code': 200, 'error': None, 'data': {'name': 'v_006f20ae-5306-44fe-afdf-e433f79284e8'}} from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_c74b7878-4936-4015-8b63-142923947310/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:24:29 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] create volume v_006f20ae-5306-44fe-afdf-e433f79284e8 from snapshot v_006f20ae-5306-44fe-afdf-e433f79284e8 {{(pid=84696) create_volume_from_snapshot /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:674}} Apr 17 15:24:29 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_c74b7878-4936-4015-8b63-142923947310/clone {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:24:29 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_c74b7878-4936-4015-8b63-142923947310/clone {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:24:30 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-733c2b3a-fb60-4425-8809-eceac0525109 tempest-VolumesTransfersTest-450487945 None] Geting {'code': 200, 'error': None, 'data': {'results': 0, 'entries': []}} from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_bd3bcf59-72e2-43c9-973a-ab5956dc83b5/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:24:30 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-733c2b3a-fb60-4425-8809-eceac0525109 tempest-VolumesTransfersTest-450487945 None] get volume v_bd3bcf59-72e2-43c9-973a-ab5956dc83b5 info {{(pid=84696) get_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:239}} Apr 17 15:24:30 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-733c2b3a-fb60-4425-8809-eceac0525109 tempest-VolumesTransfersTest-450487945 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_bd3bcf59-72e2-43c9-973a-ab5956dc83b5 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:24:30 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-733c2b3a-fb60-4425-8809-eceac0525109 tempest-VolumesTransfersTest-450487945 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_bd3bcf59-72e2-43c9-973a-ab5956dc83b5 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:24:30 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-733c2b3a-fb60-4425-8809-eceac0525109 tempest-VolumesTransfersTest-450487945 None] Geting {'code': 200, 'error': None, 'data': {'origin': None, 'relatime': None, 'acltype': None, 'vscan': None, 'full_name': 'Pool-0/v_bd3bcf59-72e2-43c9-973a-ab5956dc83b5', 'userrefs': None, 'primarycache': 'all', 'logbias': 'latency', 'creation': '1681746461', 'sync': 'always', 'is_clone': False, 'dedup': 'off', 'sharenfs': None, 'receive_resume_token': None, 'volsize': '1073741824', 'referenced': '57344', 'sharesmb': None, 'createtxg': '187918', 'reservation': '0', 'scontext': None, 'mountpoint': None, 'casesensitivity': None, 'guid': '17873668656675123053', 'usedbyrefreservation': '0', 'dnodesize': None, 'written': '57344', 'logicalused': '28672', 'compressratio': '1.00', 'rootcontext': 'none', 'default_scsi_id': '41eb1f83f60d4bd6', 'type': 'volume', 'compression': 'lz4', 'snapdir': None, 'overlay': None, 'encryption': 'off', 'xattr': None, 'volmode': 'default', 'copies': '1', 'snapshot_limit': '18446744073709551615', 'aclinherit': None, 'defcontext': 'none', 'readonly': 'off', 'version': None, 'recordsize': None, 'filesystem_limit': None, 'mounted': None, 'mlslabel': 'none', 'secondarycache': 'all', 'refreservation': '0', 'available': '113909891072', 'san:volume_id': '41eb1f83f60d4bd69bcc1667ac8fad821865a064', 'encryptionroot': None, 'exec': None, 'refquota': None, 'refcompressratio': '1.00', 'quota': None, 'utf8only': None, 'keylocation': 'none', 'snapdev': 'hidden', 'snapshot_count': '18446744073709551615', 'fscontext': 'none', 'clones': None, 'canmount': None, 'keystatus': None, 'atime': None, 'usedbysnapshots': '0', 'normalization': None, 'usedbychildren': '0', 'volblocksize': '131072', 'usedbydataset': '57344', 'objsetid': '122784', 'name': 'v_bd3bcf59-72e2-43c9-973a-ab5956dc83b5', 'defer_destroy': None, 'pbkdf2iters': '0', 'checksum': 'on', 'redundant_metadata': 'all', 'filesystem_count': None, 'devices': None, 'keyformat': 'none', 'setuid': None, 'used': '57344', 'logicalreferenced': '28672', 'context': 'none', 'zoned': None, 'nbmand': None}} from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_bd3bcf59-72e2-43c9-973a-ab5956dc83b5 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:24:30 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-733c2b3a-fb60-4425-8809-eceac0525109 tempest-VolumesTransfersTest-450487945 None] delete volume:v_bd3bcf59-72e2-43c9-973a-ab5956dc83b5 recursively children:{'force_umount': True} {{(pid=84696) delete_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:335}} Apr 17 15:24:30 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-733c2b3a-fb60-4425-8809-eceac0525109 tempest-VolumesTransfersTest-450487945 None] Sending pool request DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_bd3bcf59-72e2-43c9-973a-ab5956dc83b5 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:24:30 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-733c2b3a-fb60-4425-8809-eceac0525109 tempest-VolumesTransfersTest-450487945 None] Sending DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_bd3bcf59-72e2-43c9-973a-ab5956dc83b5 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:24:30 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-ffc97846-e2d4-4147-ab93-bc7ab1830c90 tempest-VolumesGetTest-1961901732 None] Geting {'code': 200, 'error': None, 'data': {'name': 'v_fa8e71d4-54a7-4e0d-9a26-50cc7946ce11'}} from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_9a8586a6-f2f5-4bee-a56f-a06b01120c39/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:24:30 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-ffc97846-e2d4-4147-ab93-bc7ab1830c90 tempest-VolumesGetTest-1961901732 None] create volume v_fa8e71d4-54a7-4e0d-9a26-50cc7946ce11 from snapshot v_fa8e71d4-54a7-4e0d-9a26-50cc7946ce11 {{(pid=84696) create_volume_from_snapshot /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:674}} Apr 17 15:24:30 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-ffc97846-e2d4-4147-ab93-bc7ab1830c90 tempest-VolumesGetTest-1961901732 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_9a8586a6-f2f5-4bee-a56f-a06b01120c39/clone {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:24:30 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-ffc97846-e2d4-4147-ab93-bc7ab1830c90 tempest-VolumesGetTest-1961901732 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_9a8586a6-f2f5-4bee-a56f-a06b01120c39/clone {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:24:31 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] Geting {'code': 200, 'error': None, 'data': {'origin': 'Pool-0/v_c74b7878-4936-4015-8b63-142923947310@v_006f20ae-5306-44fe-afdf-e433f79284e8', 'is_clone': True, 'full_name': 'Pool-0/v_006f20ae-5306-44fe-afdf-e433f79284e8', 'name': 'v_006f20ae-5306-44fe-afdf-e433f79284e8'}} from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_c74b7878-4936-4015-8b63-142923947310/clone {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:24:31 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] get volume v_006f20ae-5306-44fe-afdf-e433f79284e8 info {{(pid=84696) get_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:239}} Apr 17 15:24:31 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:24:31 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:24:31 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] Geting {'code': 200, 'error': None, 'data': {'origin': 'Pool-0/v_c74b7878-4936-4015-8b63-142923947310@v_006f20ae-5306-44fe-afdf-e433f79284e8', 'relatime': None, 'acltype': None, 'vscan': None, 'full_name': 'Pool-0/v_006f20ae-5306-44fe-afdf-e433f79284e8', 'userrefs': None, 'primarycache': 'all', 'logbias': 'latency', 'creation': '1681746463', 'sync': 'always', 'is_clone': True, 'dedup': 'off', 'sharenfs': None, 'receive_resume_token': None, 'volsize': '1073741824', 'referenced': '16285696', 'sharesmb': None, 'createtxg': '187926', 'reservation': '0', 'scontext': None, 'mountpoint': None, 'casesensitivity': None, 'guid': '17063470514764443530', 'usedbyrefreservation': '0', 'dnodesize': None, 'written': '8192', 'logicalused': '4096', 'compressratio': '1.00', 'rootcontext': 'none', 'default_scsi_id': 'acbee1e7ecbae27b', 'type': 'volume', 'compression': 'lz4', 'snapdir': None, 'overlay': None, 'encryption': 'off', 'xattr': None, 'volmode': 'default', 'copies': '1', 'snapshot_limit': '18446744073709551615', 'aclinherit': None, 'defcontext': 'none', 'readonly': 'off', 'version': None, 'recordsize': None, 'filesystem_limit': None, 'mounted': None, 'mlslabel': 'none', 'secondarycache': 'all', 'refreservation': '0', 'available': '113909960704', 'used': '8192', 'encryptionroot': None, 'exec': None, 'refquota': None, 'refcompressratio': '1.17', 'quota': None, 'utf8only': None, 'keylocation': 'none', 'snapdev': 'hidden', 'snapshot_count': '18446744073709551615', 'fscontext': 'none', 'clones': None, 'canmount': None, 'keystatus': None, 'atime': None, 'usedbysnapshots': '0', 'normalization': None, 'usedbychildren': '0', 'volblocksize': '131072', 'usedbydataset': '8192', 'objsetid': '121594', 'name': 'v_006f20ae-5306-44fe-afdf-e433f79284e8', 'defer_destroy': None, 'pbkdf2iters': '0', 'checksum': 'on', 'redundant_metadata': 'all', 'filesystem_count': None, 'devices': None, 'keyformat': 'none', 'setuid': None, 'logicalreferenced': '19042304', 'context': 'none', 'zoned': None, 'nbmand': None}} from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:24:31 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (56d3623a-20ae-4488-a3c9-512da4dab680) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'volume_name': 'volume-006f20ae-5306-44fe-afdf-e433f79284e8', 'volume_size': 1, 'source_volid': 'c74b7878-4936-4015-8b63-142923947310', 'source_volstatus': 'downloading'}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:24:31 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeOnFinishTask;volume:create, create.end' (f490fa3f-260a-40ff-9bde-a1ca5018235f) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:24:31 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=59,cinder:INSERT=13,cinder:UPDATE=26,cinder:DELETE=2 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:24:31 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] Volume volume-006f20ae-5306-44fe-afdf-e433f79284e8 (006f20ae-5306-44fe-afdf-e433f79284e8): created successfully Apr 17 15:24:31 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeOnFinishTask;volume:create, create.end' (f490fa3f-260a-40ff-9bde-a1ca5018235f) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:24:31 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] Flow 'volume_create_manager' (b0f8b0ec-ee47-4e74-8b4c-ee08c977d4b0) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:24:31 user cinder-volume[84696]: INFO cinder.volume.manager [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] Created volume successfully. Apr 17 15:24:31 user cinder-volume[84696]: DEBUG cinder.image.cache [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] Creating new image-volume cache entry for image b3f0dec1-7758-44a8-82e0-76ada8837428 on user@joviandss-2#joviandss-2 {{(pid=84696) create_cache_entry /opt/stack/cinder/cinder/image/cache.py:104}} Apr 17 15:24:31 user cinder-volume[84696]: DEBUG cinder.image.cache [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] New image-volume cache entry created: {'id': 1, 'image_id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'volume_id': '006f20ae-5306-44fe-afdf-e433f79284e8', 'host': 'user@joviandss-2#joviandss-2', 'size': 1, 'image_updated_at': datetime.datetime(2023, 4, 17, 15, 19, 55), 'last_used': datetime.datetime(2023, 4, 17, 15, 24, 31, 778926)}. {{(pid=84696) create_cache_entry /opt/stack/cinder/cinder/image/cache.py:129}} Apr 17 15:24:31 user cinder-volume[84696]: DEBUG cinder.coordination [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] Lock "cinder-b3f0dec1-7758-44a8-82e0-76ada8837428" released by "_prepare_image_cache_entry" :: held 30.352s {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:207}} Apr 17 15:24:31 user cinder-volume[84696]: DEBUG cinder.volume.volume_utils [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] Marking volume c74b7878-4936-4015-8b63-142923947310 as bootable. {{(pid=84696) enable_bootable_flag /opt/stack/cinder/cinder/volume/volume_utils.py:1140}} Apr 17 15:24:31 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-cb0bf660-4169-4302-b2c3-1ebf4db895fa tempest-GroupSnapshotsTest-1926061586 None] Geting {'code': 204} from DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_7249f8e4-ed4b-4689-aba3-2766c924f42c {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:24:31 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-cb0bf660-4169-4302-b2c3-1ebf4db895fa tempest-GroupSnapshotsTest-1926061586 None] volume v_7249f8e4-ed4b-4689-aba3-2766c924f42c deleted {{(pid=84696) delete_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:346}} Apr 17 15:24:31 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-cb0bf660-4169-4302-b2c3-1ebf4db895fa tempest-GroupSnapshotsTest-1926061586 None] remove_export for volume: f6603de5-71d3-468b-a673-ce1ff5cd1285. {{(pid=84696) remove_export /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:718}} Apr 17 15:24:31 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-cb0bf660-4169-4302-b2c3-1ebf4db895fa tempest-GroupSnapshotsTest-1926061586 None] remove export {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:912}} Apr 17 15:24:31 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-cb0bf660-4169-4302-b2c3-1ebf4db895fa tempest-GroupSnapshotsTest-1926061586 None] detach volume:Volume(_name_id=None,admin_metadata={},attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:23:56Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-volume-591301068',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=286e53b0-285a-4d83-8cd3-cb2b5641b31f,host='user@joviandss-2#joviandss-2',id=f6603de5-71d3-468b-a673-ce1ff5cd1285,launched_at=2023-04-17T15:23:58Z,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='6cae12f3c2614d6fb32f21db60fec8d3',provider_auth='CHAP IJceEwRD Gck6vpZ8P9UFhI',provider_geometry=None,provider_id=None,provider_location='172.16.0.220:3260,1 iqn.2010-10.org.openstack:f6603de5-71d3-468b-a673-ce1ff5cd1285 0',replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:23:56Z,service_uuid='e628352e-c4cf-40eb-8389-cce40663026a',shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='deleting',terminated_at=None,updated_at=2023-04-17T15:24:28Z,use_quota=True,user_id='560c850c324949d7a214e7a59dd65f8d',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(73c9cbe6-d3bc-4af2-810c-855e0b88bd3f),volume_type_id=73c9cbe6-d3bc-4af2-810c-855e0b88bd3f) from target:iqn.2010-10.org.openstack:f6603de5-71d3-468b-a673-ce1ff5cd1285. {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:913}} Apr 17 15:24:31 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-cb0bf660-4169-4302-b2c3-1ebf4db895fa tempest-GroupSnapshotsTest-1926061586 None] detach volume v_f6603de5-71d3-468b-a673-ce1ff5cd1285 from target iqn.2010-10.org.openstack:f6603de5-71d3-468b-a673-ce1ff5cd1285 {{(pid=84696) detach_target_vol /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:606}} Apr 17 15:24:31 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-cb0bf660-4169-4302-b2c3-1ebf4db895fa tempest-GroupSnapshotsTest-1926061586 None] Sending pool request DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:f6603de5-71d3-468b-a673-ce1ff5cd1285/luns/v_f6603de5-71d3-468b-a673-ce1ff5cd1285 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:24:31 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-cb0bf660-4169-4302-b2c3-1ebf4db895fa tempest-GroupSnapshotsTest-1926061586 None] Sending DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:f6603de5-71d3-468b-a673-ce1ff5cd1285/luns/v_f6603de5-71d3-468b-a673-ce1ff5cd1285 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:24:31 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] Copying metadata from image b3f0dec1-7758-44a8-82e0-76ada8837428 to c74b7878-4936-4015-8b63-142923947310. {{(pid=84696) _handle_bootable_volume_glance_meta /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:452}} Apr 17 15:24:31 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] Creating volume glance metadata for volume c74b7878-4936-4015-8b63-142923947310 backed by image b3f0dec1-7758-44a8-82e0-76ada8837428 with: {'hw_rng_model': 'virtio', 'owner_specified.openstack.md5': '', 'owner_specified.openstack.object': 'images/cirros-0.5.2-x86_64-disk', 'owner_specified.openstack.sha256': '', 'image_id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'image_name': 'cirros-0.5.2-x86_64-disk', 'checksum': 'b874c39491a2377b8490f5f1e89761a4', 'container_format': 'bare', 'disk_format': 'qcow2', 'min_disk': 0, 'min_ram': 0, 'size': 16300544}. {{(pid=84696) _capture_volume_image_metadata /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:683}} Apr 17 15:24:31 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-cb0bf660-4169-4302-b2c3-1ebf4db895fa tempest-GroupSnapshotsTest-1926061586 None] Geting {'code': 404, 'error': {'class': 'opene.exceptions.ItemNotFoundError', 'message': 'Target iqn.2010-10.org.openstack:f6603de5-71d3-468b-a673-ce1ff5cd1285 not exists.', 'url': 'http://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:f6603de5-71d3-468b-a673-ce1ff5cd1285/luns/v_f6603de5-71d3-468b-a673-ce1ff5cd1285'}, 'data': None} from DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:f6603de5-71d3-468b-a673-ce1ff5cd1285/luns/v_f6603de5-71d3-468b-a673-ce1ff5cd1285 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:24:31 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-cb0bf660-4169-4302-b2c3-1ebf4db895fa tempest-GroupSnapshotsTest-1926061586 None] failed to remove resource iqn.2010-10.org.openstack:f6603de5-71d3-468b-a673-ce1ff5cd1285 because of JDSS resource v_f6603de5-71d3-468b-a673-ce1ff5cd1285 DNE. {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:920}} Apr 17 15:24:31 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-cb0bf660-4169-4302-b2c3-1ebf4db895fa tempest-GroupSnapshotsTest-1926061586 None] delete target: iqn.2010-10.org.openstack:f6603de5-71d3-468b-a673-ce1ff5cd1285 {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:930}} Apr 17 15:24:31 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-cb0bf660-4169-4302-b2c3-1ebf4db895fa tempest-GroupSnapshotsTest-1926061586 None] delete target iqn.2010-10.org.openstack:f6603de5-71d3-468b-a673-ce1ff5cd1285 {{(pid=84696) delete_target /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:442}} Apr 17 15:24:31 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-cb0bf660-4169-4302-b2c3-1ebf4db895fa tempest-GroupSnapshotsTest-1926061586 None] Sending pool request DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:f6603de5-71d3-468b-a673-ce1ff5cd1285 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:24:31 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-cb0bf660-4169-4302-b2c3-1ebf4db895fa tempest-GroupSnapshotsTest-1926061586 None] Sending DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:f6603de5-71d3-468b-a673-ce1ff5cd1285 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:24:31 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-cb0bf660-4169-4302-b2c3-1ebf4db895fa tempest-GroupSnapshotsTest-1926061586 None] Geting {'code': 404, 'error': {'class': 'opene.exceptions.ItemNotFoundError', 'message': 'Target iqn.2010-10.org.openstack:f6603de5-71d3-468b-a673-ce1ff5cd1285 not exists.', 'url': 'http://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:f6603de5-71d3-468b-a673-ce1ff5cd1285'}, 'data': None} from DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:f6603de5-71d3-468b-a673-ce1ff5cd1285 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:24:31 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-cb0bf660-4169-4302-b2c3-1ebf4db895fa tempest-GroupSnapshotsTest-1926061586 None] failed to remove resource iqn.2010-10.org.openstack:f6603de5-71d3-468b-a673-ce1ff5cd1285 because of JDSS resource iqn.2010-10.org.openstack:f6603de5-71d3-468b-a673-ce1ff5cd1285 DNE. {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:935}} Apr 17 15:24:31 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-cb0bf660-4169-4302-b2c3-1ebf4db895fa tempest-GroupSnapshotsTest-1926061586 None] deleating volume v_f6603de5-71d3-468b-a673-ce1ff5cd1285 {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 15:24:31 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-cb0bf660-4169-4302-b2c3-1ebf4db895fa tempest-GroupSnapshotsTest-1926061586 None] get snapshots for volume v_f6603de5-71d3-468b-a673-ce1ff5cd1285 {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 15:24:31 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-cb0bf660-4169-4302-b2c3-1ebf4db895fa tempest-GroupSnapshotsTest-1926061586 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_f6603de5-71d3-468b-a673-ce1ff5cd1285/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:24:31 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-cb0bf660-4169-4302-b2c3-1ebf4db895fa tempest-GroupSnapshotsTest-1926061586 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_f6603de5-71d3-468b-a673-ce1ff5cd1285/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:24:31 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (a0bec28b-ba7f-4d26-8d24-66f65b71c718) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'volume_name': 'volume-c74b7878-4936-4015-8b63-142923947310', 'volume_size': 1, 'image_id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'image_location': (None, None), 'image_meta': {'name': 'cirros-0.5.2-x86_64-disk', 'disk_format': 'qcow2', 'container_format': 'bare', 'visibility': 'public', 'size': 16300544, 'virtual_size': 117440512, 'status': 'active', 'checksum': 'b874c39491a2377b8490f5f1e89761a4', 'protected': False, 'min_ram': 0, 'min_disk': 0, 'owner': 'a43c211b4ed542538799349489b1d884', 'os_hidden': False, 'os_hash_algo': 'sha512', 'os_hash_value': '6b813aa46bb90b4da216a4d19376593fa3f4fc7e617f03a92b7fe11e9a3981cbe8f0959dbebe36225e5f53dc4492341a4863cac4ed1ee0909f3fc78ef9c3e869', 'id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'created_at': datetime.datetime(2023, 4, 17, 15, 19, 53, tzinfo=datetime.timezone.utc), 'updated_at': datetime.datetime(2023, 4, 17, 15, 19, 55, tzinfo=datetime.timezone.utc), 'tags': [], 'file': '/v2/images/b3f0dec1-7758-44a8-82e0-76ada8837428/file', 'properties': {'hw_rng_model': 'virtio', 'owner_specified.openstack.md5': '', 'owner_specified.openstack.object': 'images/cirros-0.5.2-x86_64-disk', 'owner_specified.openstack.sha256': ''}}, 'image_service': }' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:24:31 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeOnFinishTask;volume:create, create.end' (55030a9c-c5f4-4856-a964-37c5820e2d1f) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:24:31 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] Volume volume-c74b7878-4936-4015-8b63-142923947310 (c74b7878-4936-4015-8b63-142923947310): created successfully Apr 17 15:24:31 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeOnFinishTask;volume:create, create.end' (55030a9c-c5f4-4856-a964-37c5820e2d1f) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:24:31 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] Flow 'volume_create_manager' (8483a0a5-3732-4994-aae0-0c0dcfbdbc19) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:24:31 user cinder-volume[84696]: INFO cinder.volume.manager [None req-564420f3-c6f7-4553-9d71-dade6adb0bfb tempest-TaggedBootDevicesTest_v242-1317742003 None] Created volume successfully. Apr 17 15:24:32 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-ffc97846-e2d4-4147-ab93-bc7ab1830c90 tempest-VolumesGetTest-1961901732 None] Geting {'code': 200, 'error': None, 'data': {'origin': 'Pool-0/v_9a8586a6-f2f5-4bee-a56f-a06b01120c39@v_fa8e71d4-54a7-4e0d-9a26-50cc7946ce11', 'is_clone': True, 'full_name': 'Pool-0/v_fa8e71d4-54a7-4e0d-9a26-50cc7946ce11', 'name': 'v_fa8e71d4-54a7-4e0d-9a26-50cc7946ce11'}} from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_9a8586a6-f2f5-4bee-a56f-a06b01120c39/clone {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:24:32 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-ffc97846-e2d4-4147-ab93-bc7ab1830c90 tempest-VolumesGetTest-1961901732 None] get volume v_fa8e71d4-54a7-4e0d-9a26-50cc7946ce11 info {{(pid=84696) get_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:239}} Apr 17 15:24:32 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-ffc97846-e2d4-4147-ab93-bc7ab1830c90 tempest-VolumesGetTest-1961901732 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_fa8e71d4-54a7-4e0d-9a26-50cc7946ce11 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:24:32 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-ffc97846-e2d4-4147-ab93-bc7ab1830c90 tempest-VolumesGetTest-1961901732 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_fa8e71d4-54a7-4e0d-9a26-50cc7946ce11 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:24:32 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-ffc97846-e2d4-4147-ab93-bc7ab1830c90 tempest-VolumesGetTest-1961901732 None] Geting {'code': 200, 'error': None, 'data': {'origin': 'Pool-0/v_9a8586a6-f2f5-4bee-a56f-a06b01120c39@v_fa8e71d4-54a7-4e0d-9a26-50cc7946ce11', 'relatime': None, 'acltype': None, 'vscan': None, 'full_name': 'Pool-0/v_fa8e71d4-54a7-4e0d-9a26-50cc7946ce11', 'userrefs': None, 'primarycache': 'all', 'logbias': 'latency', 'creation': '1681746464', 'sync': 'always', 'is_clone': True, 'dedup': 'off', 'sharenfs': None, 'receive_resume_token': None, 'volsize': '1073741824', 'referenced': '57344', 'sharesmb': None, 'createtxg': '187930', 'reservation': '0', 'scontext': None, 'mountpoint': None, 'casesensitivity': None, 'guid': '1118096299032255136', 'usedbyrefreservation': '0', 'dnodesize': None, 'written': '0', 'logicalused': '0', 'compressratio': '1.00', 'rootcontext': 'none', 'default_scsi_id': '405b5facf109dfda', 'type': 'volume', 'compression': 'lz4', 'snapdir': None, 'overlay': None, 'encryption': 'off', 'xattr': None, 'volmode': 'default', 'copies': '1', 'snapshot_limit': '18446744073709551615', 'aclinherit': None, 'defcontext': 'none', 'readonly': 'off', 'version': None, 'recordsize': None, 'filesystem_limit': None, 'mounted': None, 'mlslabel': 'none', 'secondarycache': 'all', 'refreservation': '0', 'available': '113909948416', 'used': '0', 'encryptionroot': None, 'exec': None, 'refquota': None, 'refcompressratio': '1.00', 'quota': None, 'utf8only': None, 'keylocation': 'none', 'snapdev': 'hidden', 'snapshot_count': '18446744073709551615', 'fscontext': 'none', 'clones': None, 'canmount': None, 'keystatus': None, 'atime': None, 'usedbysnapshots': '0', 'normalization': None, 'usedbychildren': '0', 'volblocksize': '131072', 'usedbydataset': '0', 'objsetid': '123401', 'name': 'v_fa8e71d4-54a7-4e0d-9a26-50cc7946ce11', 'defer_destroy': None, 'pbkdf2iters': '0', 'checksum': 'on', 'redundant_metadata': 'all', 'filesystem_count': None, 'devices': None, 'keyformat': 'none', 'setuid': None, 'logicalreferenced': '28672', 'context': 'none', 'zoned': None, 'nbmand': None}} from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_fa8e71d4-54a7-4e0d-9a26-50cc7946ce11 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:24:32 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-ffc97846-e2d4-4147-ab93-bc7ab1830c90 tempest-VolumesGetTest-1961901732 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (e230de85-0720-498a-805a-b1d9ebb28db9) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'volume_name': 'volume-fa8e71d4-54a7-4e0d-9a26-50cc7946ce11', 'volume_size': 1, 'source_volid': '9a8586a6-f2f5-4bee-a56f-a06b01120c39', 'source_volstatus': 'available'}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:24:32 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-ffc97846-e2d4-4147-ab93-bc7ab1830c90 tempest-VolumesGetTest-1961901732 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeOnFinishTask;volume:create, create.end' (1890a26a-f1f2-4872-8ec3-3196486c77f1) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:24:32 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-ffc97846-e2d4-4147-ab93-bc7ab1830c90 tempest-VolumesGetTest-1961901732 None] Volume volume-fa8e71d4-54a7-4e0d-9a26-50cc7946ce11 (fa8e71d4-54a7-4e0d-9a26-50cc7946ce11): created successfully Apr 17 15:24:32 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-ffc97846-e2d4-4147-ab93-bc7ab1830c90 tempest-VolumesGetTest-1961901732 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeOnFinishTask;volume:create, create.end' (1890a26a-f1f2-4872-8ec3-3196486c77f1) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:24:32 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-ffc97846-e2d4-4147-ab93-bc7ab1830c90 tempest-VolumesGetTest-1961901732 None] Flow 'volume_create_manager' (d530dba1-3eb3-4477-af75-0538fdc7f6d4) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:24:32 user cinder-volume[84696]: INFO cinder.volume.manager [None req-ffc97846-e2d4-4147-ab93-bc7ab1830c90 tempest-VolumesGetTest-1961901732 None] Created volume successfully. Apr 17 15:24:32 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-cb0bf660-4169-4302-b2c3-1ebf4db895fa tempest-GroupSnapshotsTest-1926061586 None] Geting {'code': 200, 'error': None, 'data': {'results': 0, 'entries': []}} from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_f6603de5-71d3-468b-a673-ce1ff5cd1285/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:24:32 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-cb0bf660-4169-4302-b2c3-1ebf4db895fa tempest-GroupSnapshotsTest-1926061586 None] get volume v_f6603de5-71d3-468b-a673-ce1ff5cd1285 info {{(pid=84696) get_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:239}} Apr 17 15:24:32 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-cb0bf660-4169-4302-b2c3-1ebf4db895fa tempest-GroupSnapshotsTest-1926061586 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_f6603de5-71d3-468b-a673-ce1ff5cd1285 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:24:32 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-cb0bf660-4169-4302-b2c3-1ebf4db895fa tempest-GroupSnapshotsTest-1926061586 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_f6603de5-71d3-468b-a673-ce1ff5cd1285 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:24:32 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-cb0bf660-4169-4302-b2c3-1ebf4db895fa tempest-GroupSnapshotsTest-1926061586 None] Geting {'code': 200, 'error': None, 'data': {'origin': None, 'relatime': None, 'acltype': None, 'vscan': None, 'full_name': 'Pool-0/v_f6603de5-71d3-468b-a673-ce1ff5cd1285', 'userrefs': None, 'primarycache': 'all', 'logbias': 'latency', 'creation': '1681746430', 'sync': 'always', 'is_clone': False, 'dedup': 'off', 'sharenfs': None, 'receive_resume_token': None, 'volsize': '1073741824', 'referenced': '57344', 'sharesmb': None, 'createtxg': '187843', 'reservation': '0', 'scontext': None, 'mountpoint': None, 'casesensitivity': None, 'guid': '9693421205875437733', 'usedbyrefreservation': '0', 'dnodesize': None, 'written': '57344', 'logicalused': '28672', 'compressratio': '1.00', 'rootcontext': 'none', 'default_scsi_id': 'd2c169919903c9a4', 'type': 'volume', 'compression': 'lz4', 'snapdir': None, 'overlay': None, 'encryption': 'off', 'xattr': None, 'volmode': 'default', 'copies': '1', 'snapshot_limit': '18446744073709551615', 'aclinherit': None, 'defcontext': 'none', 'readonly': 'off', 'version': None, 'recordsize': None, 'filesystem_limit': None, 'mounted': None, 'mlslabel': 'none', 'secondarycache': 'all', 'refreservation': '0', 'available': '113909993472', 'san:volume_id': 'd2c169919903c9a41910148ef797167a4dc8cda9', 'encryptionroot': None, 'exec': None, 'refquota': None, 'refcompressratio': '1.00', 'quota': None, 'utf8only': None, 'keylocation': 'none', 'snapdev': 'hidden', 'snapshot_count': '18446744073709551615', 'fscontext': 'none', 'clones': None, 'canmount': None, 'keystatus': None, 'atime': None, 'usedbysnapshots': '0', 'normalization': None, 'usedbychildren': '0', 'volblocksize': '131072', 'usedbydataset': '57344', 'objsetid': '121850', 'name': 'v_f6603de5-71d3-468b-a673-ce1ff5cd1285', 'defer_destroy': None, 'pbkdf2iters': '0', 'checksum': 'on', 'redundant_metadata': 'all', 'filesystem_count': None, 'devices': None, 'keyformat': 'none', 'setuid': None, 'used': '57344', 'logicalreferenced': '28672', 'context': 'none', 'zoned': None, 'nbmand': None}} from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_f6603de5-71d3-468b-a673-ce1ff5cd1285 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:24:32 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-cb0bf660-4169-4302-b2c3-1ebf4db895fa tempest-GroupSnapshotsTest-1926061586 None] delete volume:v_f6603de5-71d3-468b-a673-ce1ff5cd1285 recursively children:{'force_umount': True} {{(pid=84696) delete_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:335}} Apr 17 15:24:32 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-cb0bf660-4169-4302-b2c3-1ebf4db895fa tempest-GroupSnapshotsTest-1926061586 None] Sending pool request DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_f6603de5-71d3-468b-a673-ce1ff5cd1285 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:24:32 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-cb0bf660-4169-4302-b2c3-1ebf4db895fa tempest-GroupSnapshotsTest-1926061586 None] Sending DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_f6603de5-71d3-468b-a673-ce1ff5cd1285 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:24:33 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=63,cinder:INSERT=9,cinder:UPDATE=27,cinder:DELETE=1 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:24:33 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-f476fc78-3409-48a7-af27-d8fd19176ff4 tempest-TaggedBootDevicesTest_v242-1317742003 None] Volume reschedule parameters: True retry: {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 15:24:33 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f476fc78-3409-48a7-af27-d8fd19176ff4 tempest-TaggedBootDevicesTest_v242-1317742003 None] Flow 'volume_create_manager' (1b5bd7e6-1489-483e-8755-7bb3ea36831b) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:24:33 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f476fc78-3409-48a7-af27-d8fd19176ff4 tempest-TaggedBootDevicesTest_v242-1317742003 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (817f01c0-15b1-4023-b737-b2a9c6e2da3a) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:24:33 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f476fc78-3409-48a7-af27-d8fd19176ff4 tempest-TaggedBootDevicesTest_v242-1317742003 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (817f01c0-15b1-4023-b737-b2a9c6e2da3a) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:24:33Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TaggedBootDevicesTest_v242-volume-1325806681',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=18475864-c202-4937-b18f-cc3e16c6d89f,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='8f0c848b490b47629051e961c682c5a5',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:24:33Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:24:33Z,use_quota=True,user_id='3e4b069991954c57993051fbe8665d35',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:24:33 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f476fc78-3409-48a7-af27-d8fd19176ff4 tempest-TaggedBootDevicesTest_v242-1317742003 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (7e3fb2c7-d31c-4adb-953f-0173138f1d8d) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:24:33 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f476fc78-3409-48a7-af27-d8fd19176ff4 tempest-TaggedBootDevicesTest_v242-1317742003 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (7e3fb2c7-d31c-4adb-953f-0173138f1d8d) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:24:33 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f476fc78-3409-48a7-af27-d8fd19176ff4 tempest-TaggedBootDevicesTest_v242-1317742003 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (f13ddba7-3d97-4bd6-b1f9-b2d22002e764) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:24:33 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f476fc78-3409-48a7-af27-d8fd19176ff4 tempest-TaggedBootDevicesTest_v242-1317742003 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (f13ddba7-3d97-4bd6-b1f9-b2d22002e764) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': '18475864-c202-4937-b18f-cc3e16c6d89f', 'volume_name': 'volume-18475864-c202-4937-b18f-cc3e16c6d89f', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:24:33 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f476fc78-3409-48a7-af27-d8fd19176ff4 tempest-TaggedBootDevicesTest_v242-1317742003 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (6728f951-f8b7-4f21-aba3-adf0bd687b4d) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:24:33 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f476fc78-3409-48a7-af27-d8fd19176ff4 tempest-TaggedBootDevicesTest_v242-1317742003 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (6728f951-f8b7-4f21-aba3-adf0bd687b4d) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:24:33 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f476fc78-3409-48a7-af27-d8fd19176ff4 tempest-TaggedBootDevicesTest_v242-1317742003 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (3b1a8c3a-9517-4bb4-be7c-59c523e03559) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:24:33 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-f476fc78-3409-48a7-af27-d8fd19176ff4 tempest-TaggedBootDevicesTest_v242-1317742003 None] Volume 18475864-c202-4937-b18f-cc3e16c6d89f: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-18475864-c202-4937-b18f-cc3e16c6d89f', 'volume_size': 1} Apr 17 15:24:33 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-f476fc78-3409-48a7-af27-d8fd19176ff4 tempest-TaggedBootDevicesTest_v242-1317742003 None] creating volume v_18475864-c202-4937-b18f-cc3e16c6d89f. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 15:24:33 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-f476fc78-3409-48a7-af27-d8fd19176ff4 tempest-TaggedBootDevicesTest_v242-1317742003 None] create volume {'name': 'v_18475864-c202-4937-b18f-cc3e16c6d89f', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 15:24:33 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-f476fc78-3409-48a7-af27-d8fd19176ff4 tempest-TaggedBootDevicesTest_v242-1317742003 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:24:33 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-f476fc78-3409-48a7-af27-d8fd19176ff4 tempest-TaggedBootDevicesTest_v242-1317742003 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:24:33 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-e706d105-4a69-45fd-a85e-bbcb2970af50 tempest-VolumesGetTest-1961901732 None] Volume reschedule parameters: True retry: {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 15:24:33 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e706d105-4a69-45fd-a85e-bbcb2970af50 tempest-VolumesGetTest-1961901732 None] Flow 'volume_create_manager' (8c7bfe27-72dc-40b1-917a-ddd47c37ef3d) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:24:33 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e706d105-4a69-45fd-a85e-bbcb2970af50 tempest-VolumesGetTest-1961901732 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (26564cca-49b4-45a8-adb3-9e35ac88bfe0) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:24:33 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e706d105-4a69-45fd-a85e-bbcb2970af50 tempest-VolumesGetTest-1961901732 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (26564cca-49b4-45a8-adb3-9e35ac88bfe0) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:24:34Z,deleted=False,deleted_at=None,display_description='tempest-@#$%^* description-43480765',display_name=None,ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=a90b7acc-e20a-4e97-9004-f18d7a2b4c98,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='a65d14b0a2894a6f90506d86feaefd15',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:24:34Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:24:34Z,use_quota=True,user_id='6ffa3051570a4721a48a0171b3cc4d16',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:24:33 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e706d105-4a69-45fd-a85e-bbcb2970af50 tempest-VolumesGetTest-1961901732 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (2ad39337-c57e-4d6c-92c9-3ca5f8ba4a6d) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:24:33 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e706d105-4a69-45fd-a85e-bbcb2970af50 tempest-VolumesGetTest-1961901732 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (2ad39337-c57e-4d6c-92c9-3ca5f8ba4a6d) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:24:33 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e706d105-4a69-45fd-a85e-bbcb2970af50 tempest-VolumesGetTest-1961901732 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (0f9b9f5d-a7ad-4de7-8176-28fe389e6f38) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:24:33 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e706d105-4a69-45fd-a85e-bbcb2970af50 tempest-VolumesGetTest-1961901732 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (0f9b9f5d-a7ad-4de7-8176-28fe389e6f38) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': 'a90b7acc-e20a-4e97-9004-f18d7a2b4c98', 'volume_name': 'volume-a90b7acc-e20a-4e97-9004-f18d7a2b4c98', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:24:33 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e706d105-4a69-45fd-a85e-bbcb2970af50 tempest-VolumesGetTest-1961901732 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (5c1f95df-5675-44d2-aa28-b677241fc83a) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:24:33 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e706d105-4a69-45fd-a85e-bbcb2970af50 tempest-VolumesGetTest-1961901732 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (5c1f95df-5675-44d2-aa28-b677241fc83a) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:24:33 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e706d105-4a69-45fd-a85e-bbcb2970af50 tempest-VolumesGetTest-1961901732 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (7a361a33-c1e0-4e04-babf-3d95fb9153c2) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:24:33 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-e706d105-4a69-45fd-a85e-bbcb2970af50 tempest-VolumesGetTest-1961901732 None] Volume a90b7acc-e20a-4e97-9004-f18d7a2b4c98: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-a90b7acc-e20a-4e97-9004-f18d7a2b4c98', 'volume_size': 1} Apr 17 15:24:33 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-e706d105-4a69-45fd-a85e-bbcb2970af50 tempest-VolumesGetTest-1961901732 None] creating volume v_a90b7acc-e20a-4e97-9004-f18d7a2b4c98. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 15:24:33 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-e706d105-4a69-45fd-a85e-bbcb2970af50 tempest-VolumesGetTest-1961901732 None] create volume {'name': 'v_a90b7acc-e20a-4e97-9004-f18d7a2b4c98', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 15:24:33 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-e706d105-4a69-45fd-a85e-bbcb2970af50 tempest-VolumesGetTest-1961901732 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:24:33 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-e706d105-4a69-45fd-a85e-bbcb2970af50 tempest-VolumesGetTest-1961901732 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:24:34 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-733c2b3a-fb60-4425-8809-eceac0525109 tempest-VolumesTransfersTest-450487945 None] Geting {'code': 204} from DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_bd3bcf59-72e2-43c9-973a-ab5956dc83b5 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:24:34 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-733c2b3a-fb60-4425-8809-eceac0525109 tempest-VolumesTransfersTest-450487945 None] volume v_bd3bcf59-72e2-43c9-973a-ab5956dc83b5 deleted {{(pid=84696) delete_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:346}} Apr 17 15:24:34 user cinder-volume[84696]: DEBUG cinder.quota [None req-733c2b3a-fb60-4425-8809-eceac0525109 tempest-VolumesTransfersTest-450487945 None] Created reservations ['92fbb131-dc60-41dc-aac6-b00d1a24b491', 'bd8e45cf-690a-4383-a338-b0ab8bed4e1d', 'cfbd1b98-362a-4bd9-afd8-4f8549517ba4', '3a3caae0-e6da-4914-9bff-96ab357c4022'] {{(pid=84696) reserve /opt/stack/cinder/cinder/quota.py:755}} Apr 17 15:24:34 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-733c2b3a-fb60-4425-8809-eceac0525109 tempest-VolumesTransfersTest-450487945 None] Updating volume stats {{(pid=84696) _update_volume_stats /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:724}} Apr 17 15:24:34 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-733c2b3a-fb60-4425-8809-eceac0525109 tempest-VolumesTransfersTest-450487945 None] Get pool Pool-0 fsprops {{(pid=84696) get_pool_stats /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:894}} Apr 17 15:24:34 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-733c2b3a-fb60-4425-8809-eceac0525109 tempest-VolumesTransfersTest-450487945 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:24:34 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-733c2b3a-fb60-4425-8809-eceac0525109 tempest-VolumesTransfersTest-450487945 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:24:35 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-733c2b3a-fb60-4425-8809-eceac0525109 tempest-VolumesTransfersTest-450487945 None] Geting {'code': 200, 'error': None, 'data': {'available': '113909993472', 'status': 30, 'import_status': {'import_steps': {'san_setup': True, 'vip_setup': True, 'zfs_import': True, 'mount_system_volume': True, 'nas_setup': True}, 'import_successful': True}, 'scan': None, 'encryption': {'enabled': False}, 'deduplication_rate': '1.00', 'sysvol_upgrade': 'not_needed', 'vdevs': [{'name': 'wwn-0x6000c296b793bf526a107a9d823e947a', 'iostats': {'read': '0', 'write': '0', 'chksum': '0'}, 'disks': [{'origin': 'local', 'slot': '', 'led': 'off', 'name': 'sdn', 'iostats': {'read': '0', 'write': '0', 'chksum': '0'}, 'alias': '', 'health': 'ONLINE', 'sn': '6000c296b793bf526a107a9d823e947a', 'trim_data': {'status': 'trim_unsupported', 'progress': None, 'trim_time': None}, 'path': 'pci-0000:03:00.0-scsi-0:0:14:0', 'model': 'Virtual disk', 'id': 'wwn-0x6000c296b793bf526a107a9d823e947a', 'size': 128849018880}], 'health': 'ONLINE', 'vdev_replacings': [], 'vdev_spares': [], 'type': ''}], 'id': '14181317334440137330', 'health': 'ONLINE', 'iostats': {'read': '0', 'write': '0', 'chksum': '0'}, 'operation': 'none', 'size': '127775277056', 'autotrim': False, 'name': 'Pool-0'}} from GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:24:35 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-733c2b3a-fb60-4425-8809-eceac0525109 tempest-VolumesTransfersTest-450487945 None] Total capacity: 119, Free 106. {{(pid=84696) _update_volume_stats /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:757}} Apr 17 15:24:35 user cinder-volume[84696]: DEBUG cinder.manager [None req-733c2b3a-fb60-4425-8809-eceac0525109 tempest-VolumesTransfersTest-450487945 None] Notifying Schedulers of capabilities ... {{(pid=84696) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 17 15:24:35 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-f476fc78-3409-48a7-af27-d8fd19176ff4 tempest-TaggedBootDevicesTest_v242-1317742003 None] Geting {'code': 201, 'error': None, 'data': {'full_name': 'Pool-0/v_18475864-c202-4937-b18f-cc3e16c6d89f', 'is_clone': False, 'name': 'v_18475864-c202-4937-b18f-cc3e16c6d89f'}} from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:24:35 user cinder-volume[84696]: INFO cinder.volume.manager [None req-733c2b3a-fb60-4425-8809-eceac0525109 tempest-VolumesTransfersTest-450487945 None] Deleted volume successfully. Apr 17 15:24:35 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f476fc78-3409-48a7-af27-d8fd19176ff4 tempest-TaggedBootDevicesTest_v242-1317742003 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (3b1a8c3a-9517-4bb4-be7c-59c523e03559) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'volume_name': 'volume-18475864-c202-4937-b18f-cc3e16c6d89f', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:24:35 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f476fc78-3409-48a7-af27-d8fd19176ff4 tempest-TaggedBootDevicesTest_v242-1317742003 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeOnFinishTask;volume:create, create.end' (8afad89b-e16d-4850-9573-050708a9d3c5) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:24:35 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-f476fc78-3409-48a7-af27-d8fd19176ff4 tempest-TaggedBootDevicesTest_v242-1317742003 None] Volume volume-18475864-c202-4937-b18f-cc3e16c6d89f (18475864-c202-4937-b18f-cc3e16c6d89f): created successfully Apr 17 15:24:35 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f476fc78-3409-48a7-af27-d8fd19176ff4 tempest-TaggedBootDevicesTest_v242-1317742003 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeOnFinishTask;volume:create, create.end' (8afad89b-e16d-4850-9573-050708a9d3c5) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:24:35 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f476fc78-3409-48a7-af27-d8fd19176ff4 tempest-TaggedBootDevicesTest_v242-1317742003 None] Flow 'volume_create_manager' (1b5bd7e6-1489-483e-8755-7bb3ea36831b) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:24:35 user cinder-volume[84696]: DEBUG cinder.coordination [None req-733c2b3a-fb60-4425-8809-eceac0525109 tempest-VolumesTransfersTest-450487945 None] Lock "cinder-bd3bcf59-72e2-43c9-973a-ab5956dc83b5-delete_volume" released by "delete_volume" :: held 6.232s {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:207}} Apr 17 15:24:35 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-e706d105-4a69-45fd-a85e-bbcb2970af50 tempest-VolumesGetTest-1961901732 None] Geting {'code': 201, 'error': None, 'data': {'full_name': 'Pool-0/v_a90b7acc-e20a-4e97-9004-f18d7a2b4c98', 'is_clone': False, 'name': 'v_a90b7acc-e20a-4e97-9004-f18d7a2b4c98'}} from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:24:35 user cinder-volume[84696]: INFO cinder.volume.manager [None req-f476fc78-3409-48a7-af27-d8fd19176ff4 tempest-TaggedBootDevicesTest_v242-1317742003 None] Created volume successfully. Apr 17 15:24:35 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e706d105-4a69-45fd-a85e-bbcb2970af50 tempest-VolumesGetTest-1961901732 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (7a361a33-c1e0-4e04-babf-3d95fb9153c2) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'volume_name': 'volume-a90b7acc-e20a-4e97-9004-f18d7a2b4c98', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:24:35 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e706d105-4a69-45fd-a85e-bbcb2970af50 tempest-VolumesGetTest-1961901732 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeOnFinishTask;volume:create, create.end' (8f2e7e17-fea9-4d8e-993c-acf525744638) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:24:35 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-e706d105-4a69-45fd-a85e-bbcb2970af50 tempest-VolumesGetTest-1961901732 None] Volume volume-a90b7acc-e20a-4e97-9004-f18d7a2b4c98 (a90b7acc-e20a-4e97-9004-f18d7a2b4c98): created successfully Apr 17 15:24:35 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e706d105-4a69-45fd-a85e-bbcb2970af50 tempest-VolumesGetTest-1961901732 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeOnFinishTask;volume:create, create.end' (8f2e7e17-fea9-4d8e-993c-acf525744638) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:24:35 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e706d105-4a69-45fd-a85e-bbcb2970af50 tempest-VolumesGetTest-1961901732 None] Flow 'volume_create_manager' (8c7bfe27-72dc-40b1-917a-ddd47c37ef3d) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:24:35 user cinder-volume[84696]: INFO cinder.volume.manager [None req-e706d105-4a69-45fd-a85e-bbcb2970af50 tempest-VolumesGetTest-1961901732 None] Created volume successfully. Apr 17 15:24:35 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-cb0bf660-4169-4302-b2c3-1ebf4db895fa tempest-GroupSnapshotsTest-1926061586 None] Geting {'code': 204} from DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_f6603de5-71d3-468b-a673-ce1ff5cd1285 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:24:35 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-cb0bf660-4169-4302-b2c3-1ebf4db895fa tempest-GroupSnapshotsTest-1926061586 None] volume v_f6603de5-71d3-468b-a673-ce1ff5cd1285 deleted {{(pid=84696) delete_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:346}} Apr 17 15:24:35 user cinder-volume[84696]: DEBUG cinder.quota [None req-cb0bf660-4169-4302-b2c3-1ebf4db895fa tempest-GroupSnapshotsTest-1926061586 None] Created reservations ['549f5e04-aafd-48cd-a553-8745b89551b7'] {{(pid=84696) reserve /opt/stack/cinder/cinder/quota.py:755}} Apr 17 15:24:35 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-dafb15d5-01b7-4803-8d19-6ae0af274656 tempest-TaggedBootDevicesTest_v242-1317742003 None] Volume reschedule parameters: True retry: {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 15:24:35 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-dafb15d5-01b7-4803-8d19-6ae0af274656 tempest-TaggedBootDevicesTest_v242-1317742003 None] Flow 'volume_create_manager' (6d8d1e82-9651-4b91-ae75-3d12918bbc9e) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:24:35 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-dafb15d5-01b7-4803-8d19-6ae0af274656 tempest-TaggedBootDevicesTest_v242-1317742003 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (dae5e954-81b8-4fd3-b0a9-3b278aa09d9d) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:24:35 user cinder-volume[84696]: WARNING cinder.quota [None req-cb0bf660-4169-4302-b2c3-1ebf4db895fa tempest-GroupSnapshotsTest-1926061586 None] Deprecated: Default quota for resource: volumes_tempest-GroupSnapshotsTest-volume-type-2030863494 is set by the default quota flag: quota_volumes_tempest-GroupSnapshotsTest-volume-type-2030863494, it is now deprecated. Please use the default quota class for default quota. Apr 17 15:24:35 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-dafb15d5-01b7-4803-8d19-6ae0af274656 tempest-TaggedBootDevicesTest_v242-1317742003 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (dae5e954-81b8-4fd3-b0a9-3b278aa09d9d) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:24:36Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TaggedBootDevicesTest_v242-volume-1252584131',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=65a7c5eb-3af6-4afd-918f-ccb69a68641c,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='8f0c848b490b47629051e961c682c5a5',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:24:36Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:24:36Z,use_quota=True,user_id='3e4b069991954c57993051fbe8665d35',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:24:35 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-dafb15d5-01b7-4803-8d19-6ae0af274656 tempest-TaggedBootDevicesTest_v242-1317742003 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (91f4fece-95e7-49f8-9461-6d5b6edb534a) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:24:35 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-dafb15d5-01b7-4803-8d19-6ae0af274656 tempest-TaggedBootDevicesTest_v242-1317742003 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (91f4fece-95e7-49f8-9461-6d5b6edb534a) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:24:35 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-dafb15d5-01b7-4803-8d19-6ae0af274656 tempest-TaggedBootDevicesTest_v242-1317742003 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (bff57e89-bdc6-4a45-8d9d-eeeecba995a8) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:24:35 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-dafb15d5-01b7-4803-8d19-6ae0af274656 tempest-TaggedBootDevicesTest_v242-1317742003 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (bff57e89-bdc6-4a45-8d9d-eeeecba995a8) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': '65a7c5eb-3af6-4afd-918f-ccb69a68641c', 'volume_name': 'volume-65a7c5eb-3af6-4afd-918f-ccb69a68641c', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:24:35 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-dafb15d5-01b7-4803-8d19-6ae0af274656 tempest-TaggedBootDevicesTest_v242-1317742003 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (ad1e1892-1d33-4266-bb3c-23a8660155a1) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:24:35 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-dafb15d5-01b7-4803-8d19-6ae0af274656 tempest-TaggedBootDevicesTest_v242-1317742003 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (ad1e1892-1d33-4266-bb3c-23a8660155a1) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:24:35 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-dafb15d5-01b7-4803-8d19-6ae0af274656 tempest-TaggedBootDevicesTest_v242-1317742003 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (6db73a2e-23ee-4e63-be3f-72fb46ec4169) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:24:35 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-dafb15d5-01b7-4803-8d19-6ae0af274656 tempest-TaggedBootDevicesTest_v242-1317742003 None] Volume 65a7c5eb-3af6-4afd-918f-ccb69a68641c: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-65a7c5eb-3af6-4afd-918f-ccb69a68641c', 'volume_size': 1} Apr 17 15:24:35 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-dafb15d5-01b7-4803-8d19-6ae0af274656 tempest-TaggedBootDevicesTest_v242-1317742003 None] creating volume v_65a7c5eb-3af6-4afd-918f-ccb69a68641c. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 15:24:35 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-dafb15d5-01b7-4803-8d19-6ae0af274656 tempest-TaggedBootDevicesTest_v242-1317742003 None] create volume {'name': 'v_65a7c5eb-3af6-4afd-918f-ccb69a68641c', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 15:24:35 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-dafb15d5-01b7-4803-8d19-6ae0af274656 tempest-TaggedBootDevicesTest_v242-1317742003 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:24:35 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-dafb15d5-01b7-4803-8d19-6ae0af274656 tempest-TaggedBootDevicesTest_v242-1317742003 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:24:35 user cinder-volume[84696]: DEBUG cinder.quota [None req-cb0bf660-4169-4302-b2c3-1ebf4db895fa tempest-GroupSnapshotsTest-1926061586 None] Created reservations ['766daceb-f7fa-4cce-b66c-61c4ab2a680c', '85a06de2-17ff-48b4-aa22-f7cdd688fe77', '2b2e3562-d4d8-4ea0-bc60-cd7b6e976b1c', 'aeb25709-c1ca-49ac-9d11-5b4501531213'] {{(pid=84696) reserve /opt/stack/cinder/cinder/quota.py:755}} Apr 17 15:24:36 user cinder-volume[84696]: DEBUG cinder.quota [None req-cb0bf660-4169-4302-b2c3-1ebf4db895fa tempest-GroupSnapshotsTest-1926061586 None] Created reservations ['b44b3fc1-ede1-4474-8484-fde86f7f75f4', '8093ede2-d079-4eb2-8c47-c04cd5c52fe4', '5e6dd487-9251-490a-bf90-9fdc7976544d', '0589c5cb-54d0-494f-b858-7d5b052947eb'] {{(pid=84696) reserve /opt/stack/cinder/cinder/quota.py:755}} Apr 17 15:24:36 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=51,cinder:UPDATE=27,cinder:INSERT=18,cinder:DELETE=4 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:24:36 user cinder-volume[84696]: DEBUG cinder.coordination [None req-8ef70554-d4a6-4df0-9a96-76278ba9c85b tempest-VolumesGetTest-1961901732 None] Acquiring lock "cinder-a90b7acc-e20a-4e97-9004-f18d7a2b4c98-delete_volume" by "delete_volume" {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:192}} Apr 17 15:24:36 user cinder-volume[84696]: DEBUG cinder.coordination [None req-8ef70554-d4a6-4df0-9a96-76278ba9c85b tempest-VolumesGetTest-1961901732 None] Lock "cinder-a90b7acc-e20a-4e97-9004-f18d7a2b4c98-delete_volume" acquired by "delete_volume" :: waited 0.035s {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:196}} Apr 17 15:24:36 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-cb0bf660-4169-4302-b2c3-1ebf4db895fa tempest-GroupSnapshotsTest-1926061586 None] Updating volume stats {{(pid=84696) _update_volume_stats /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:724}} Apr 17 15:24:36 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-cb0bf660-4169-4302-b2c3-1ebf4db895fa tempest-GroupSnapshotsTest-1926061586 None] Get pool Pool-0 fsprops {{(pid=84696) get_pool_stats /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:894}} Apr 17 15:24:36 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-cb0bf660-4169-4302-b2c3-1ebf4db895fa tempest-GroupSnapshotsTest-1926061586 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:24:36 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-cb0bf660-4169-4302-b2c3-1ebf4db895fa tempest-GroupSnapshotsTest-1926061586 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:24:36 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-8ef70554-d4a6-4df0-9a96-76278ba9c85b tempest-VolumesGetTest-1961901732 None] remove_export for volume: a90b7acc-e20a-4e97-9004-f18d7a2b4c98. {{(pid=84696) remove_export /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:718}} Apr 17 15:24:36 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-8ef70554-d4a6-4df0-9a96-76278ba9c85b tempest-VolumesGetTest-1961901732 None] remove export {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:912}} Apr 17 15:24:36 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-8ef70554-d4a6-4df0-9a96-76278ba9c85b tempest-VolumesGetTest-1961901732 None] detach volume:Volume(_name_id=None,admin_metadata={},attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:24:34Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesGetTest-Volume-1605278566',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=a90b7acc-e20a-4e97-9004-f18d7a2b4c98,launched_at=2023-04-17T15:24:35Z,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='a65d14b0a2894a6f90506d86feaefd15',provider_auth='CHAP wpsAUMvK HkPiwt1YbgIuUf',provider_geometry=None,provider_id=None,provider_location='172.16.0.220:3260,1 iqn.2010-10.org.openstack:a90b7acc-e20a-4e97-9004-f18d7a2b4c98 0',replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:24:34Z,service_uuid='e628352e-c4cf-40eb-8389-cce40663026a',shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='deleting',terminated_at=2023-04-17T15:24:36Z,updated_at=2023-04-17T15:24:36Z,use_quota=True,user_id='6ffa3051570a4721a48a0171b3cc4d16',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c) from target:iqn.2010-10.org.openstack:a90b7acc-e20a-4e97-9004-f18d7a2b4c98. {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:913}} Apr 17 15:24:36 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-8ef70554-d4a6-4df0-9a96-76278ba9c85b tempest-VolumesGetTest-1961901732 None] detach volume v_a90b7acc-e20a-4e97-9004-f18d7a2b4c98 from target iqn.2010-10.org.openstack:a90b7acc-e20a-4e97-9004-f18d7a2b4c98 {{(pid=84696) detach_target_vol /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:606}} Apr 17 15:24:36 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-8ef70554-d4a6-4df0-9a96-76278ba9c85b tempest-VolumesGetTest-1961901732 None] Sending pool request DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:a90b7acc-e20a-4e97-9004-f18d7a2b4c98/luns/v_a90b7acc-e20a-4e97-9004-f18d7a2b4c98 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:24:36 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-8ef70554-d4a6-4df0-9a96-76278ba9c85b tempest-VolumesGetTest-1961901732 None] Sending DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:a90b7acc-e20a-4e97-9004-f18d7a2b4c98/luns/v_a90b7acc-e20a-4e97-9004-f18d7a2b4c98 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:24:36 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-8ef70554-d4a6-4df0-9a96-76278ba9c85b tempest-VolumesGetTest-1961901732 None] Geting {'code': 404, 'error': {'class': 'opene.exceptions.ItemNotFoundError', 'message': 'Target iqn.2010-10.org.openstack:a90b7acc-e20a-4e97-9004-f18d7a2b4c98 not exists.', 'url': 'http://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:a90b7acc-e20a-4e97-9004-f18d7a2b4c98/luns/v_a90b7acc-e20a-4e97-9004-f18d7a2b4c98'}, 'data': None} from DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:a90b7acc-e20a-4e97-9004-f18d7a2b4c98/luns/v_a90b7acc-e20a-4e97-9004-f18d7a2b4c98 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:24:36 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-8ef70554-d4a6-4df0-9a96-76278ba9c85b tempest-VolumesGetTest-1961901732 None] failed to remove resource iqn.2010-10.org.openstack:a90b7acc-e20a-4e97-9004-f18d7a2b4c98 because of JDSS resource v_a90b7acc-e20a-4e97-9004-f18d7a2b4c98 DNE. {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:920}} Apr 17 15:24:36 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-8ef70554-d4a6-4df0-9a96-76278ba9c85b tempest-VolumesGetTest-1961901732 None] delete target: iqn.2010-10.org.openstack:a90b7acc-e20a-4e97-9004-f18d7a2b4c98 {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:930}} Apr 17 15:24:36 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-8ef70554-d4a6-4df0-9a96-76278ba9c85b tempest-VolumesGetTest-1961901732 None] delete target iqn.2010-10.org.openstack:a90b7acc-e20a-4e97-9004-f18d7a2b4c98 {{(pid=84696) delete_target /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:442}} Apr 17 15:24:36 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-8ef70554-d4a6-4df0-9a96-76278ba9c85b tempest-VolumesGetTest-1961901732 None] Sending pool request DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:a90b7acc-e20a-4e97-9004-f18d7a2b4c98 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:24:36 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-8ef70554-d4a6-4df0-9a96-76278ba9c85b tempest-VolumesGetTest-1961901732 None] Sending DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:a90b7acc-e20a-4e97-9004-f18d7a2b4c98 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:24:36 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-8ef70554-d4a6-4df0-9a96-76278ba9c85b tempest-VolumesGetTest-1961901732 None] Geting {'code': 404, 'error': {'class': 'opene.exceptions.ItemNotFoundError', 'message': 'Target iqn.2010-10.org.openstack:a90b7acc-e20a-4e97-9004-f18d7a2b4c98 not exists.', 'url': 'http://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:a90b7acc-e20a-4e97-9004-f18d7a2b4c98'}, 'data': None} from DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:a90b7acc-e20a-4e97-9004-f18d7a2b4c98 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:24:36 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-8ef70554-d4a6-4df0-9a96-76278ba9c85b tempest-VolumesGetTest-1961901732 None] failed to remove resource iqn.2010-10.org.openstack:a90b7acc-e20a-4e97-9004-f18d7a2b4c98 because of JDSS resource iqn.2010-10.org.openstack:a90b7acc-e20a-4e97-9004-f18d7a2b4c98 DNE. {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:935}} Apr 17 15:24:36 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-8ef70554-d4a6-4df0-9a96-76278ba9c85b tempest-VolumesGetTest-1961901732 None] deleating volume v_a90b7acc-e20a-4e97-9004-f18d7a2b4c98 {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 15:24:36 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-8ef70554-d4a6-4df0-9a96-76278ba9c85b tempest-VolumesGetTest-1961901732 None] get snapshots for volume v_a90b7acc-e20a-4e97-9004-f18d7a2b4c98 {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 15:24:36 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-8ef70554-d4a6-4df0-9a96-76278ba9c85b tempest-VolumesGetTest-1961901732 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_a90b7acc-e20a-4e97-9004-f18d7a2b4c98/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:24:36 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-8ef70554-d4a6-4df0-9a96-76278ba9c85b tempest-VolumesGetTest-1961901732 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_a90b7acc-e20a-4e97-9004-f18d7a2b4c98/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:24:37 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-8ef70554-d4a6-4df0-9a96-76278ba9c85b tempest-VolumesGetTest-1961901732 None] Geting {'code': 200, 'error': None, 'data': {'results': 0, 'entries': []}} from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_a90b7acc-e20a-4e97-9004-f18d7a2b4c98/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:24:37 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-8ef70554-d4a6-4df0-9a96-76278ba9c85b tempest-VolumesGetTest-1961901732 None] get volume v_a90b7acc-e20a-4e97-9004-f18d7a2b4c98 info {{(pid=84696) get_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:239}} Apr 17 15:24:37 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-8ef70554-d4a6-4df0-9a96-76278ba9c85b tempest-VolumesGetTest-1961901732 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_a90b7acc-e20a-4e97-9004-f18d7a2b4c98 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:24:37 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-8ef70554-d4a6-4df0-9a96-76278ba9c85b tempest-VolumesGetTest-1961901732 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_a90b7acc-e20a-4e97-9004-f18d7a2b4c98 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:24:37 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-8ef70554-d4a6-4df0-9a96-76278ba9c85b tempest-VolumesGetTest-1961901732 None] Geting {'code': 200, 'error': None, 'data': {'origin': None, 'relatime': None, 'acltype': None, 'vscan': None, 'full_name': 'Pool-0/v_a90b7acc-e20a-4e97-9004-f18d7a2b4c98', 'userrefs': None, 'primarycache': 'all', 'logbias': 'latency', 'creation': '1681746468', 'sync': 'always', 'is_clone': False, 'dedup': 'off', 'sharenfs': None, 'receive_resume_token': None, 'volsize': '1073741824', 'referenced': '57344', 'sharesmb': None, 'createtxg': '187940', 'reservation': '0', 'scontext': None, 'mountpoint': None, 'casesensitivity': None, 'guid': '3776772538524829345', 'usedbyrefreservation': '0', 'dnodesize': None, 'written': '57344', 'logicalused': '28672', 'compressratio': '1.00', 'rootcontext': 'none', 'default_scsi_id': '2be324070199c8ef', 'type': 'volume', 'compression': 'lz4', 'snapdir': None, 'overlay': None, 'encryption': 'off', 'xattr': None, 'volmode': 'default', 'copies': '1', 'snapshot_limit': '18446744073709551615', 'aclinherit': None, 'defcontext': 'none', 'readonly': 'off', 'version': None, 'recordsize': None, 'filesystem_limit': None, 'mounted': None, 'mlslabel': 'none', 'secondarycache': 'all', 'refreservation': '0', 'available': '113909817344', 'san:volume_id': '2be324070199c8efca27e5d695afc0a0df21a536', 'encryptionroot': None, 'exec': None, 'refquota': None, 'refcompressratio': '1.00', 'quota': None, 'utf8only': None, 'keylocation': 'none', 'snapdev': 'hidden', 'snapshot_count': '18446744073709551615', 'fscontext': 'none', 'clones': None, 'canmount': None, 'keystatus': None, 'atime': None, 'usedbysnapshots': '0', 'normalization': None, 'usedbychildren': '0', 'volblocksize': '131072', 'usedbydataset': '57344', 'objsetid': '123054', 'name': 'v_a90b7acc-e20a-4e97-9004-f18d7a2b4c98', 'defer_destroy': None, 'pbkdf2iters': '0', 'checksum': 'on', 'redundant_metadata': 'all', 'filesystem_count': None, 'devices': None, 'keyformat': 'none', 'setuid': None, 'used': '57344', 'logicalreferenced': '28672', 'context': 'none', 'zoned': None, 'nbmand': None}} from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_a90b7acc-e20a-4e97-9004-f18d7a2b4c98 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:24:37 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-8ef70554-d4a6-4df0-9a96-76278ba9c85b tempest-VolumesGetTest-1961901732 None] delete volume:v_a90b7acc-e20a-4e97-9004-f18d7a2b4c98 recursively children:{'force_umount': True} {{(pid=84696) delete_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:335}} Apr 17 15:24:37 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-8ef70554-d4a6-4df0-9a96-76278ba9c85b tempest-VolumesGetTest-1961901732 None] Sending pool request DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_a90b7acc-e20a-4e97-9004-f18d7a2b4c98 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:24:37 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-8ef70554-d4a6-4df0-9a96-76278ba9c85b tempest-VolumesGetTest-1961901732 None] Sending DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_a90b7acc-e20a-4e97-9004-f18d7a2b4c98 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:24:37 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-cb0bf660-4169-4302-b2c3-1ebf4db895fa tempest-GroupSnapshotsTest-1926061586 None] Geting {'code': 200, 'error': None, 'data': {'available': '113909911552', 'status': 30, 'import_status': {'import_steps': {'san_setup': True, 'vip_setup': True, 'zfs_import': True, 'mount_system_volume': True, 'nas_setup': True}, 'import_successful': True}, 'scan': None, 'encryption': {'enabled': False}, 'deduplication_rate': '1.00', 'sysvol_upgrade': 'not_needed', 'vdevs': [{'name': 'wwn-0x6000c296b793bf526a107a9d823e947a', 'iostats': {'read': '0', 'write': '0', 'chksum': '0'}, 'disks': [{'origin': 'local', 'slot': '', 'led': 'off', 'name': 'sdn', 'iostats': {'read': '0', 'write': '0', 'chksum': '0'}, 'alias': '', 'health': 'ONLINE', 'sn': '6000c296b793bf526a107a9d823e947a', 'trim_data': {'status': 'trim_unsupported', 'progress': None, 'trim_time': None}, 'path': 'pci-0000:03:00.0-scsi-0:0:14:0', 'model': 'Virtual disk', 'id': 'wwn-0x6000c296b793bf526a107a9d823e947a', 'size': 128849018880}], 'health': 'ONLINE', 'vdev_replacings': [], 'vdev_spares': [], 'type': ''}], 'id': '14181317334440137330', 'health': 'ONLINE', 'iostats': {'read': '0', 'write': '0', 'chksum': '0'}, 'operation': 'none', 'size': '127775277056', 'autotrim': False, 'name': 'Pool-0'}} from GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:24:37 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-cb0bf660-4169-4302-b2c3-1ebf4db895fa tempest-GroupSnapshotsTest-1926061586 None] Total capacity: 119, Free 106. {{(pid=84696) _update_volume_stats /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:757}} Apr 17 15:24:37 user cinder-volume[84696]: DEBUG cinder.manager [None req-cb0bf660-4169-4302-b2c3-1ebf4db895fa tempest-GroupSnapshotsTest-1926061586 None] Notifying Schedulers of capabilities ... {{(pid=84696) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 17 15:24:37 user cinder-volume[84696]: INFO cinder.volume.manager [None req-cb0bf660-4169-4302-b2c3-1ebf4db895fa tempest-GroupSnapshotsTest-1926061586 None] Delete group completed successfully. Apr 17 15:24:37 user cinder-volume[84696]: INFO cinder.volume.manager [None req-b10140cb-27c9-4b38-8d11-d2ade8c0d299 tempest-GroupSnapshotsTest-1926061586 None] Group tempest-GroupSnapshotsTest-Group-1860289922: creating Apr 17 15:24:37 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-dafb15d5-01b7-4803-8d19-6ae0af274656 tempest-TaggedBootDevicesTest_v242-1317742003 None] Geting {'code': 201, 'error': None, 'data': {'full_name': 'Pool-0/v_65a7c5eb-3af6-4afd-918f-ccb69a68641c', 'is_clone': False, 'name': 'v_65a7c5eb-3af6-4afd-918f-ccb69a68641c'}} from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:24:37 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-dafb15d5-01b7-4803-8d19-6ae0af274656 tempest-TaggedBootDevicesTest_v242-1317742003 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (6db73a2e-23ee-4e63-be3f-72fb46ec4169) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'volume_name': 'volume-65a7c5eb-3af6-4afd-918f-ccb69a68641c', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:24:37 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-dafb15d5-01b7-4803-8d19-6ae0af274656 tempest-TaggedBootDevicesTest_v242-1317742003 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeOnFinishTask;volume:create, create.end' (60ca214a-fa48-4be9-9ccf-aee0cea6f0dc) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:24:37 user cinder-volume[84696]: INFO cinder.volume.manager [None req-b10140cb-27c9-4b38-8d11-d2ade8c0d299 tempest-GroupSnapshotsTest-1926061586 None] Group tempest-GroupSnapshotsTest-Group-1860289922: created successfully Apr 17 15:24:37 user cinder-volume[84696]: INFO cinder.volume.manager [None req-b10140cb-27c9-4b38-8d11-d2ade8c0d299 tempest-GroupSnapshotsTest-1926061586 None] Create group completed successfully. Apr 17 15:24:37 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-dafb15d5-01b7-4803-8d19-6ae0af274656 tempest-TaggedBootDevicesTest_v242-1317742003 None] Volume volume-65a7c5eb-3af6-4afd-918f-ccb69a68641c (65a7c5eb-3af6-4afd-918f-ccb69a68641c): created successfully Apr 17 15:24:37 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-dafb15d5-01b7-4803-8d19-6ae0af274656 tempest-TaggedBootDevicesTest_v242-1317742003 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeOnFinishTask;volume:create, create.end' (60ca214a-fa48-4be9-9ccf-aee0cea6f0dc) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:24:37 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-dafb15d5-01b7-4803-8d19-6ae0af274656 tempest-TaggedBootDevicesTest_v242-1317742003 None] Flow 'volume_create_manager' (6d8d1e82-9651-4b91-ae75-3d12918bbc9e) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:24:37 user cinder-volume[84696]: INFO cinder.volume.manager [None req-dafb15d5-01b7-4803-8d19-6ae0af274656 tempest-TaggedBootDevicesTest_v242-1317742003 None] Created volume successfully. Apr 17 15:24:39 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-1cf647d2-902a-4e0a-b37a-c4223cf1233b tempest-GroupSnapshotsTest-1926061586 None] Volume reschedule parameters: True retry: {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 15:24:39 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-1cf647d2-902a-4e0a-b37a-c4223cf1233b tempest-GroupSnapshotsTest-1926061586 None] Flow 'volume_create_manager' (a984e0a6-ba58-4bfd-9f47-1635f5dc8341) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:24:39 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-1cf647d2-902a-4e0a-b37a-c4223cf1233b tempest-GroupSnapshotsTest-1926061586 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (b4e00833-3d9b-44d4-98e7-26ce610605aa) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:24:39 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-1cf647d2-902a-4e0a-b37a-c4223cf1233b tempest-GroupSnapshotsTest-1926061586 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (b4e00833-3d9b-44d4-98e7-26ce610605aa) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:24:39Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-volume-1159247536',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=83f168f7-009f-4ecb-8cda-bee4f351f951,host='user@joviandss-2#joviandss-2',id=7a71a6fd-8167-40ab-b057-5f82e0f55947,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='6cae12f3c2614d6fb32f21db60fec8d3',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:24:39Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:24:39Z,use_quota=True,user_id='560c850c324949d7a214e7a59dd65f8d',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(2fbbd4b4-c823-4693-8ac9-0db5fcafba0b),volume_type_id=2fbbd4b4-c823-4693-8ac9-0db5fcafba0b)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:24:39 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-1cf647d2-902a-4e0a-b37a-c4223cf1233b tempest-GroupSnapshotsTest-1926061586 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (8586578c-ec67-47aa-a607-81efa46086db) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:24:39 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-1cf647d2-902a-4e0a-b37a-c4223cf1233b tempest-GroupSnapshotsTest-1926061586 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (8586578c-ec67-47aa-a607-81efa46086db) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:24:39 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-1cf647d2-902a-4e0a-b37a-c4223cf1233b tempest-GroupSnapshotsTest-1926061586 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (6b074e23-82c4-43fb-bbb7-481131090948) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:24:39 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-1cf647d2-902a-4e0a-b37a-c4223cf1233b tempest-GroupSnapshotsTest-1926061586 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (6b074e23-82c4-43fb-bbb7-481131090948) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': '7a71a6fd-8167-40ab-b057-5f82e0f55947', 'volume_name': 'volume-7a71a6fd-8167-40ab-b057-5f82e0f55947', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:24:39 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-1cf647d2-902a-4e0a-b37a-c4223cf1233b tempest-GroupSnapshotsTest-1926061586 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (fd0b010d-e633-4458-8883-1be529afaab3) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:24:39 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-1cf647d2-902a-4e0a-b37a-c4223cf1233b tempest-GroupSnapshotsTest-1926061586 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (fd0b010d-e633-4458-8883-1be529afaab3) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:24:39 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-1cf647d2-902a-4e0a-b37a-c4223cf1233b tempest-GroupSnapshotsTest-1926061586 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (ea5000a6-36d3-4322-bcb2-521ff10049f1) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:24:39 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-1cf647d2-902a-4e0a-b37a-c4223cf1233b tempest-GroupSnapshotsTest-1926061586 None] Volume 7a71a6fd-8167-40ab-b057-5f82e0f55947: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-7a71a6fd-8167-40ab-b057-5f82e0f55947', 'volume_size': 1} Apr 17 15:24:39 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-1cf647d2-902a-4e0a-b37a-c4223cf1233b tempest-GroupSnapshotsTest-1926061586 None] creating volume v_7a71a6fd-8167-40ab-b057-5f82e0f55947. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 15:24:39 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-1cf647d2-902a-4e0a-b37a-c4223cf1233b tempest-GroupSnapshotsTest-1926061586 None] create volume {'name': 'v_7a71a6fd-8167-40ab-b057-5f82e0f55947', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 15:24:39 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-1cf647d2-902a-4e0a-b37a-c4223cf1233b tempest-GroupSnapshotsTest-1926061586 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:24:39 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-1cf647d2-902a-4e0a-b37a-c4223cf1233b tempest-GroupSnapshotsTest-1926061586 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:24:40 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-8ef70554-d4a6-4df0-9a96-76278ba9c85b tempest-VolumesGetTest-1961901732 None] Geting {'code': 204} from DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_a90b7acc-e20a-4e97-9004-f18d7a2b4c98 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:24:40 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-8ef70554-d4a6-4df0-9a96-76278ba9c85b tempest-VolumesGetTest-1961901732 None] volume v_a90b7acc-e20a-4e97-9004-f18d7a2b4c98 deleted {{(pid=84696) delete_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:346}} Apr 17 15:24:40 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=60,cinder:INSERT=4,cinder:UPDATE=34,cinder:DELETE=2 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:24:40 user cinder-volume[84696]: DEBUG cinder.quota [None req-8ef70554-d4a6-4df0-9a96-76278ba9c85b tempest-VolumesGetTest-1961901732 None] Created reservations ['570f100f-842d-4f85-83d7-6cd7038ee15d', 'b55b3b98-13a9-40c2-8897-1826bbf5a7bf', 'd3de1c4e-4be1-42e3-bbee-8ae777b6a548', '4f36ab97-4963-49b9-8bd3-ba0e26e1766d'] {{(pid=84696) reserve /opt/stack/cinder/cinder/quota.py:755}} Apr 17 15:24:40 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-8ef70554-d4a6-4df0-9a96-76278ba9c85b tempest-VolumesGetTest-1961901732 None] Updating volume stats {{(pid=84696) _update_volume_stats /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:724}} Apr 17 15:24:40 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-8ef70554-d4a6-4df0-9a96-76278ba9c85b tempest-VolumesGetTest-1961901732 None] Get pool Pool-0 fsprops {{(pid=84696) get_pool_stats /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:894}} Apr 17 15:24:40 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-8ef70554-d4a6-4df0-9a96-76278ba9c85b tempest-VolumesGetTest-1961901732 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:24:40 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-8ef70554-d4a6-4df0-9a96-76278ba9c85b tempest-VolumesGetTest-1961901732 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:24:40 user cinder-volume[84696]: DEBUG cinder.coordination [None req-54dde4b2-8343-4ef5-8f0f-d617a0a46a99 tempest-VolumesGetTest-1961901732 None] Acquiring lock "cinder-fa8e71d4-54a7-4e0d-9a26-50cc7946ce11-delete_volume" by "delete_volume" {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:192}} Apr 17 15:24:40 user cinder-volume[84696]: DEBUG cinder.coordination [None req-54dde4b2-8343-4ef5-8f0f-d617a0a46a99 tempest-VolumesGetTest-1961901732 None] Lock "cinder-fa8e71d4-54a7-4e0d-9a26-50cc7946ce11-delete_volume" acquired by "delete_volume" :: waited 0.011s {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:196}} Apr 17 15:24:40 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-54dde4b2-8343-4ef5-8f0f-d617a0a46a99 tempest-VolumesGetTest-1961901732 None] remove_export for volume: fa8e71d4-54a7-4e0d-9a26-50cc7946ce11. {{(pid=84696) remove_export /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:718}} Apr 17 15:24:40 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-54dde4b2-8343-4ef5-8f0f-d617a0a46a99 tempest-VolumesGetTest-1961901732 None] remove export {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:912}} Apr 17 15:24:40 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-54dde4b2-8343-4ef5-8f0f-d617a0a46a99 tempest-VolumesGetTest-1961901732 None] detach volume:Volume(_name_id=None,admin_metadata={},attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:24:28Z,deleted=False,deleted_at=None,display_description='This is the new description of volume',display_name='tempest-VolumesGetTest-new-Volume-318866890',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=fa8e71d4-54a7-4e0d-9a26-50cc7946ce11,launched_at=2023-04-17T15:24:32Z,metadata={Type='Test'},migration_status=None,multiattach=False,previous_status=None,project_id='a65d14b0a2894a6f90506d86feaefd15',provider_auth='CHAP BEWjJaGL rsWY2Bf08U7IHP',provider_geometry=None,provider_id=None,provider_location='172.16.0.220:3260,1 iqn.2010-10.org.openstack:fa8e71d4-54a7-4e0d-9a26-50cc7946ce11 0',replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:24:29Z,service_uuid='e628352e-c4cf-40eb-8389-cce40663026a',shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=9a8586a6-f2f5-4bee-a56f-a06b01120c39,status='deleting',terminated_at=2023-04-17T15:24:41Z,updated_at=2023-04-17T15:24:41Z,use_quota=True,user_id='6ffa3051570a4721a48a0171b3cc4d16',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c) from target:iqn.2010-10.org.openstack:fa8e71d4-54a7-4e0d-9a26-50cc7946ce11. {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:913}} Apr 17 15:24:40 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-54dde4b2-8343-4ef5-8f0f-d617a0a46a99 tempest-VolumesGetTest-1961901732 None] detach volume v_fa8e71d4-54a7-4e0d-9a26-50cc7946ce11 from target iqn.2010-10.org.openstack:fa8e71d4-54a7-4e0d-9a26-50cc7946ce11 {{(pid=84696) detach_target_vol /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:606}} Apr 17 15:24:40 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-54dde4b2-8343-4ef5-8f0f-d617a0a46a99 tempest-VolumesGetTest-1961901732 None] Sending pool request DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:fa8e71d4-54a7-4e0d-9a26-50cc7946ce11/luns/v_fa8e71d4-54a7-4e0d-9a26-50cc7946ce11 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:24:40 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-54dde4b2-8343-4ef5-8f0f-d617a0a46a99 tempest-VolumesGetTest-1961901732 None] Sending DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:fa8e71d4-54a7-4e0d-9a26-50cc7946ce11/luns/v_fa8e71d4-54a7-4e0d-9a26-50cc7946ce11 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:24:40 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-54dde4b2-8343-4ef5-8f0f-d617a0a46a99 tempest-VolumesGetTest-1961901732 None] Geting {'code': 404, 'error': {'class': 'opene.exceptions.ItemNotFoundError', 'message': 'Target iqn.2010-10.org.openstack:fa8e71d4-54a7-4e0d-9a26-50cc7946ce11 not exists.', 'url': 'http://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:fa8e71d4-54a7-4e0d-9a26-50cc7946ce11/luns/v_fa8e71d4-54a7-4e0d-9a26-50cc7946ce11'}, 'data': None} from DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:fa8e71d4-54a7-4e0d-9a26-50cc7946ce11/luns/v_fa8e71d4-54a7-4e0d-9a26-50cc7946ce11 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:24:40 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-54dde4b2-8343-4ef5-8f0f-d617a0a46a99 tempest-VolumesGetTest-1961901732 None] failed to remove resource iqn.2010-10.org.openstack:fa8e71d4-54a7-4e0d-9a26-50cc7946ce11 because of JDSS resource v_fa8e71d4-54a7-4e0d-9a26-50cc7946ce11 DNE. {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:920}} Apr 17 15:24:40 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-54dde4b2-8343-4ef5-8f0f-d617a0a46a99 tempest-VolumesGetTest-1961901732 None] delete target: iqn.2010-10.org.openstack:fa8e71d4-54a7-4e0d-9a26-50cc7946ce11 {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:930}} Apr 17 15:24:40 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-54dde4b2-8343-4ef5-8f0f-d617a0a46a99 tempest-VolumesGetTest-1961901732 None] delete target iqn.2010-10.org.openstack:fa8e71d4-54a7-4e0d-9a26-50cc7946ce11 {{(pid=84696) delete_target /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:442}} Apr 17 15:24:40 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-54dde4b2-8343-4ef5-8f0f-d617a0a46a99 tempest-VolumesGetTest-1961901732 None] Sending pool request DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:fa8e71d4-54a7-4e0d-9a26-50cc7946ce11 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:24:40 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-54dde4b2-8343-4ef5-8f0f-d617a0a46a99 tempest-VolumesGetTest-1961901732 None] Sending DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:fa8e71d4-54a7-4e0d-9a26-50cc7946ce11 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:24:40 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-54dde4b2-8343-4ef5-8f0f-d617a0a46a99 tempest-VolumesGetTest-1961901732 None] Geting {'code': 404, 'error': {'class': 'opene.exceptions.ItemNotFoundError', 'message': 'Target iqn.2010-10.org.openstack:fa8e71d4-54a7-4e0d-9a26-50cc7946ce11 not exists.', 'url': 'http://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:fa8e71d4-54a7-4e0d-9a26-50cc7946ce11'}, 'data': None} from DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:fa8e71d4-54a7-4e0d-9a26-50cc7946ce11 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:24:40 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-54dde4b2-8343-4ef5-8f0f-d617a0a46a99 tempest-VolumesGetTest-1961901732 None] failed to remove resource iqn.2010-10.org.openstack:fa8e71d4-54a7-4e0d-9a26-50cc7946ce11 because of JDSS resource iqn.2010-10.org.openstack:fa8e71d4-54a7-4e0d-9a26-50cc7946ce11 DNE. {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:935}} Apr 17 15:24:40 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-54dde4b2-8343-4ef5-8f0f-d617a0a46a99 tempest-VolumesGetTest-1961901732 None] deleating volume v_fa8e71d4-54a7-4e0d-9a26-50cc7946ce11 {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 15:24:40 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-54dde4b2-8343-4ef5-8f0f-d617a0a46a99 tempest-VolumesGetTest-1961901732 None] get snapshots for volume v_fa8e71d4-54a7-4e0d-9a26-50cc7946ce11 {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 15:24:40 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-54dde4b2-8343-4ef5-8f0f-d617a0a46a99 tempest-VolumesGetTest-1961901732 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_fa8e71d4-54a7-4e0d-9a26-50cc7946ce11/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:24:40 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-54dde4b2-8343-4ef5-8f0f-d617a0a46a99 tempest-VolumesGetTest-1961901732 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_fa8e71d4-54a7-4e0d-9a26-50cc7946ce11/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:24:41 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-274966fd-0736-4b94-9044-21d5db896776 tempest-VolumesExtendTest-93540250 None] Volume reschedule parameters: True retry: {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 15:24:41 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-274966fd-0736-4b94-9044-21d5db896776 tempest-VolumesExtendTest-93540250 None] Flow 'volume_create_manager' (5d41bdb9-721d-4cc7-b18e-75868a6ee8f5) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:24:41 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-274966fd-0736-4b94-9044-21d5db896776 tempest-VolumesExtendTest-93540250 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (6e0274e1-cc42-4b89-85b6-4cee217c2205) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:24:41 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-1cf647d2-902a-4e0a-b37a-c4223cf1233b tempest-GroupSnapshotsTest-1926061586 None] Geting {'code': 201, 'error': None, 'data': {'full_name': 'Pool-0/v_7a71a6fd-8167-40ab-b057-5f82e0f55947', 'is_clone': False, 'name': 'v_7a71a6fd-8167-40ab-b057-5f82e0f55947'}} from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:24:41 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-274966fd-0736-4b94-9044-21d5db896776 tempest-VolumesExtendTest-93540250 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (6e0274e1-cc42-4b89-85b6-4cee217c2205) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:24:41Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesExtendTest-Volume-1116151627',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=4b0ebe2e-141a-40f1-bcb5-83708d103f54,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='ef25ef72b5a64e2c8a60d3d139367fd1',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:24:41Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:24:41Z,use_quota=True,user_id='d473e2b76b554904a4d625f35a78c038',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:24:41 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-274966fd-0736-4b94-9044-21d5db896776 tempest-VolumesExtendTest-93540250 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (f9168a3d-af14-4053-a618-fdbf96999ab1) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:24:41 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-274966fd-0736-4b94-9044-21d5db896776 tempest-VolumesExtendTest-93540250 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (f9168a3d-af14-4053-a618-fdbf96999ab1) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:24:41 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-274966fd-0736-4b94-9044-21d5db896776 tempest-VolumesExtendTest-93540250 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (1248d4d7-4f85-485e-998f-d88e9d2358fb) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:24:41 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-1cf647d2-902a-4e0a-b37a-c4223cf1233b tempest-GroupSnapshotsTest-1926061586 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (ea5000a6-36d3-4322-bcb2-521ff10049f1) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'volume_name': 'volume-7a71a6fd-8167-40ab-b057-5f82e0f55947', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:24:41 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-1cf647d2-902a-4e0a-b37a-c4223cf1233b tempest-GroupSnapshotsTest-1926061586 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeOnFinishTask;volume:create, create.end' (d8e60ef9-3400-4a08-8617-468a42660705) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:24:41 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-1cf647d2-902a-4e0a-b37a-c4223cf1233b tempest-GroupSnapshotsTest-1926061586 None] Volume volume-7a71a6fd-8167-40ab-b057-5f82e0f55947 (7a71a6fd-8167-40ab-b057-5f82e0f55947): created successfully Apr 17 15:24:41 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-1cf647d2-902a-4e0a-b37a-c4223cf1233b tempest-GroupSnapshotsTest-1926061586 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeOnFinishTask;volume:create, create.end' (d8e60ef9-3400-4a08-8617-468a42660705) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:24:41 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-1cf647d2-902a-4e0a-b37a-c4223cf1233b tempest-GroupSnapshotsTest-1926061586 None] Flow 'volume_create_manager' (a984e0a6-ba58-4bfd-9f47-1635f5dc8341) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:24:41 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-8ef70554-d4a6-4df0-9a96-76278ba9c85b tempest-VolumesGetTest-1961901732 None] Geting {'code': 200, 'error': None, 'data': {'available': '113909805056', 'status': 30, 'import_status': {'import_steps': {'san_setup': True, 'vip_setup': True, 'zfs_import': True, 'mount_system_volume': True, 'nas_setup': True}, 'import_successful': True}, 'scan': None, 'encryption': {'enabled': False}, 'deduplication_rate': '1.00', 'sysvol_upgrade': 'not_needed', 'vdevs': [{'name': 'wwn-0x6000c296b793bf526a107a9d823e947a', 'iostats': {'read': '0', 'write': '0', 'chksum': '0'}, 'disks': [{'origin': 'local', 'slot': '', 'led': 'off', 'name': 'sdn', 'iostats': {'read': '0', 'write': '0', 'chksum': '0'}, 'alias': '', 'health': 'ONLINE', 'sn': '6000c296b793bf526a107a9d823e947a', 'trim_data': {'status': 'trim_unsupported', 'progress': None, 'trim_time': None}, 'path': 'pci-0000:03:00.0-scsi-0:0:14:0', 'model': 'Virtual disk', 'id': 'wwn-0x6000c296b793bf526a107a9d823e947a', 'size': 128849018880}], 'health': 'ONLINE', 'vdev_replacings': [], 'vdev_spares': [], 'type': ''}], 'id': '14181317334440137330', 'health': 'ONLINE', 'iostats': {'read': '0', 'write': '0', 'chksum': '0'}, 'operation': 'none', 'size': '127775277056', 'autotrim': False, 'name': 'Pool-0'}} from GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:24:41 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-8ef70554-d4a6-4df0-9a96-76278ba9c85b tempest-VolumesGetTest-1961901732 None] Total capacity: 119, Free 106. {{(pid=84696) _update_volume_stats /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:757}} Apr 17 15:24:41 user cinder-volume[84696]: DEBUG cinder.manager [None req-8ef70554-d4a6-4df0-9a96-76278ba9c85b tempest-VolumesGetTest-1961901732 None] Notifying Schedulers of capabilities ... {{(pid=84696) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 17 15:24:41 user cinder-volume[84696]: INFO cinder.volume.manager [None req-1cf647d2-902a-4e0a-b37a-c4223cf1233b tempest-GroupSnapshotsTest-1926061586 None] Created volume successfully. Apr 17 15:24:41 user cinder-volume[84696]: INFO cinder.volume.manager [None req-8ef70554-d4a6-4df0-9a96-76278ba9c85b tempest-VolumesGetTest-1961901732 None] Deleted volume successfully. Apr 17 15:24:41 user cinder-volume[84696]: DEBUG cinder.coordination [None req-8ef70554-d4a6-4df0-9a96-76278ba9c85b tempest-VolumesGetTest-1961901732 None] Lock "cinder-a90b7acc-e20a-4e97-9004-f18d7a2b4c98-delete_volume" released by "delete_volume" :: held 5.036s {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:207}} Apr 17 15:24:41 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-274966fd-0736-4b94-9044-21d5db896776 tempest-VolumesExtendTest-93540250 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (1248d4d7-4f85-485e-998f-d88e9d2358fb) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'image', 'volume_id': '4b0ebe2e-141a-40f1-bcb5-83708d103f54', 'volume_name': 'volume-4b0ebe2e-141a-40f1-bcb5-83708d103f54', 'volume_size': 1, 'image_id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'image_location': (None, None), 'image_meta': {'name': 'cirros-0.5.2-x86_64-disk', 'disk_format': 'qcow2', 'container_format': 'bare', 'visibility': 'public', 'size': 16300544, 'virtual_size': 117440512, 'status': 'active', 'checksum': 'b874c39491a2377b8490f5f1e89761a4', 'protected': False, 'min_ram': 0, 'min_disk': 0, 'owner': 'a43c211b4ed542538799349489b1d884', 'os_hidden': False, 'os_hash_algo': 'sha512', 'os_hash_value': '6b813aa46bb90b4da216a4d19376593fa3f4fc7e617f03a92b7fe11e9a3981cbe8f0959dbebe36225e5f53dc4492341a4863cac4ed1ee0909f3fc78ef9c3e869', 'id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'created_at': datetime.datetime(2023, 4, 17, 15, 19, 53, tzinfo=datetime.timezone.utc), 'updated_at': datetime.datetime(2023, 4, 17, 15, 19, 55, tzinfo=datetime.timezone.utc), 'tags': [], 'file': '/v2/images/b3f0dec1-7758-44a8-82e0-76ada8837428/file', 'properties': {'hw_rng_model': 'virtio', 'owner_specified.openstack.md5': '', 'owner_specified.openstack.object': 'images/cirros-0.5.2-x86_64-disk', 'owner_specified.openstack.sha256': ''}}, 'image_service': }' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:24:41 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-274966fd-0736-4b94-9044-21d5db896776 tempest-VolumesExtendTest-93540250 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (7055ad4d-4d15-4031-830b-3a79fd91f0e8) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:24:41 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-274966fd-0736-4b94-9044-21d5db896776 tempest-VolumesExtendTest-93540250 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (7055ad4d-4d15-4031-830b-3a79fd91f0e8) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:24:41 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-274966fd-0736-4b94-9044-21d5db896776 tempest-VolumesExtendTest-93540250 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (29ba5341-ef6b-4c3b-8b09-179ebb0a2af9) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:24:41 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-274966fd-0736-4b94-9044-21d5db896776 tempest-VolumesExtendTest-93540250 None] Volume 4b0ebe2e-141a-40f1-bcb5-83708d103f54: being created as image with specification: {'status': 'creating', 'volume_name': 'volume-4b0ebe2e-141a-40f1-bcb5-83708d103f54', 'volume_size': 1, 'image_id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'image_location': (None, None), 'image_meta': {'name': 'cirros-0.5.2-x86_64-disk', 'disk_format': 'qcow2', 'container_format': 'bare', 'visibility': 'public', 'size': 16300544, 'virtual_size': 117440512, 'status': 'active', 'checksum': 'b874c39491a2377b8490f5f1e89761a4', 'protected': False, 'min_ram': 0, 'min_disk': 0, 'owner': 'a43c211b4ed542538799349489b1d884', 'os_hidden': False, 'os_hash_algo': 'sha512', 'os_hash_value': '6b813aa46bb90b4da216a4d19376593fa3f4fc7e617f03a92b7fe11e9a3981cbe8f0959dbebe36225e5f53dc4492341a4863cac4ed1ee0909f3fc78ef9c3e869', 'id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'created_at': datetime.datetime(2023, 4, 17, 15, 19, 53, tzinfo=datetime.timezone.utc), 'updated_at': datetime.datetime(2023, 4, 17, 15, 19, 55, tzinfo=datetime.timezone.utc), 'tags': [], 'file': '/v2/images/b3f0dec1-7758-44a8-82e0-76ada8837428/file', 'properties': {'hw_rng_model': 'virtio', 'owner_specified.openstack.md5': '', 'owner_specified.openstack.object': 'images/cirros-0.5.2-x86_64-disk', 'owner_specified.openstack.sha256': ''}}, 'image_service': } Apr 17 15:24:41 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-274966fd-0736-4b94-9044-21d5db896776 tempest-VolumesExtendTest-93540250 None] Cloning 4b0ebe2e-141a-40f1-bcb5-83708d103f54 from image b3f0dec1-7758-44a8-82e0-76ada8837428 at location (None, None). {{(pid=84696) _create_from_image /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1071}} Apr 17 15:24:41 user cinder-volume[84696]: DEBUG cinder.coordination [None req-274966fd-0736-4b94-9044-21d5db896776 tempest-VolumesExtendTest-93540250 None] Acquiring lock "cinder-b3f0dec1-7758-44a8-82e0-76ada8837428" by "_prepare_image_cache_entry" {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:192}} Apr 17 15:24:41 user cinder-volume[84696]: DEBUG cinder.coordination [None req-274966fd-0736-4b94-9044-21d5db896776 tempest-VolumesExtendTest-93540250 None] Lock "cinder-b3f0dec1-7758-44a8-82e0-76ada8837428" acquired by "_prepare_image_cache_entry" :: waited 0.021s {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:196}} Apr 17 15:24:41 user cinder-volume[84696]: DEBUG cinder.image.cache [None req-274966fd-0736-4b94-9044-21d5db896776 tempest-VolumesExtendTest-93540250 None] Found image-volume cache entry: {'id': 1, 'image_id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'volume_id': '006f20ae-5306-44fe-afdf-e433f79284e8', 'host': 'user@joviandss-2#joviandss-2', 'size': 1, 'image_updated_at': datetime.datetime(2023, 4, 17, 15, 19, 55), 'last_used': datetime.datetime(2023, 4, 17, 15, 24, 41, 341105)}. {{(pid=84696) get_entry /opt/stack/cinder/cinder/image/cache.py:76}} Apr 17 15:24:41 user cinder-volume[84696]: DEBUG cinder.image.cache [None req-274966fd-0736-4b94-9044-21d5db896776 tempest-VolumesExtendTest-93540250 None] Image-volume cache entry image_update_at = 2023-04-17 15:19:55+00:00, requested image updated_at = 2023-04-17 15:19:55+00:00. {{(pid=84696) _should_update_entry /opt/stack/cinder/cinder/image/cache.py:259}} Apr 17 15:24:41 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-274966fd-0736-4b94-9044-21d5db896776 tempest-VolumesExtendTest-93540250 None] Found cache entry for image = b3f0dec1-7758-44a8-82e0-76ada8837428 on host user@joviandss-2#joviandss-2. {{(pid=84696) _prepare_image_cache_entry /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:886}} Apr 17 15:24:41 user cinder-volume[84696]: DEBUG cinder.coordination [None req-274966fd-0736-4b94-9044-21d5db896776 tempest-VolumesExtendTest-93540250 None] Lock "cinder-b3f0dec1-7758-44a8-82e0-76ada8837428" released by "_prepare_image_cache_entry" :: held 0.034s {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:207}} Apr 17 15:24:41 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-274966fd-0736-4b94-9044-21d5db896776 tempest-VolumesExtendTest-93540250 None] Attempting to retrieve cache entry for image = b3f0dec1-7758-44a8-82e0-76ada8837428 on host user@joviandss-2#joviandss-2. {{(pid=84696) _create_from_image_cache /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:827}} Apr 17 15:24:41 user cinder-volume[84696]: DEBUG cinder.image.cache [None req-274966fd-0736-4b94-9044-21d5db896776 tempest-VolumesExtendTest-93540250 None] Found image-volume cache entry: {'id': 1, 'image_id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'volume_id': '006f20ae-5306-44fe-afdf-e433f79284e8', 'host': 'user@joviandss-2#joviandss-2', 'size': 1, 'image_updated_at': datetime.datetime(2023, 4, 17, 15, 19, 55), 'last_used': datetime.datetime(2023, 4, 17, 15, 24, 41, 365002)}. {{(pid=84696) get_entry /opt/stack/cinder/cinder/image/cache.py:76}} Apr 17 15:24:41 user cinder-volume[84696]: DEBUG cinder.image.cache [None req-274966fd-0736-4b94-9044-21d5db896776 tempest-VolumesExtendTest-93540250 None] Image-volume cache entry image_update_at = 2023-04-17 15:19:55+00:00, requested image updated_at = 2023-04-17 15:19:55+00:00. {{(pid=84696) _should_update_entry /opt/stack/cinder/cinder/image/cache.py:259}} Apr 17 15:24:41 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-274966fd-0736-4b94-9044-21d5db896776 tempest-VolumesExtendTest-93540250 None] Creating from source image-volume 006f20ae-5306-44fe-afdf-e433f79284e8 {{(pid=84696) _create_from_image_cache /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:841}} Apr 17 15:24:41 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-274966fd-0736-4b94-9044-21d5db896776 tempest-VolumesExtendTest-93540250 None] cloned volume 006f20ae-5306-44fe-afdf-e433f79284e8 to 4b0ebe2e-141a-40f1-bcb5-83708d103f54 {{(pid=84696) create_cloned_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:521}} Apr 17 15:24:41 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-274966fd-0736-4b94-9044-21d5db896776 tempest-VolumesExtendTest-93540250 None] cloning v_006f20ae-5306-44fe-afdf-e433f79284e8 to v_4b0ebe2e-141a-40f1-bcb5-83708d103f54 {{(pid=84696) _clone_object /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:456}} Apr 17 15:24:41 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-274966fd-0736-4b94-9044-21d5db896776 tempest-VolumesExtendTest-93540250 None] create snapshot v_4b0ebe2e-141a-40f1-bcb5-83708d103f54 {{(pid=84696) create_snapshot /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:635}} Apr 17 15:24:41 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-274966fd-0736-4b94-9044-21d5db896776 tempest-VolumesExtendTest-93540250 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:24:41 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-274966fd-0736-4b94-9044-21d5db896776 tempest-VolumesExtendTest-93540250 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:24:41 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-54dde4b2-8343-4ef5-8f0f-d617a0a46a99 tempest-VolumesGetTest-1961901732 None] Geting {'code': 200, 'error': None, 'data': {'results': 0, 'entries': []}} from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_fa8e71d4-54a7-4e0d-9a26-50cc7946ce11/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:24:41 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-54dde4b2-8343-4ef5-8f0f-d617a0a46a99 tempest-VolumesGetTest-1961901732 None] get volume v_fa8e71d4-54a7-4e0d-9a26-50cc7946ce11 info {{(pid=84696) get_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:239}} Apr 17 15:24:41 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-54dde4b2-8343-4ef5-8f0f-d617a0a46a99 tempest-VolumesGetTest-1961901732 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_fa8e71d4-54a7-4e0d-9a26-50cc7946ce11 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:24:41 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-54dde4b2-8343-4ef5-8f0f-d617a0a46a99 tempest-VolumesGetTest-1961901732 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_fa8e71d4-54a7-4e0d-9a26-50cc7946ce11 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:24:41 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-54dde4b2-8343-4ef5-8f0f-d617a0a46a99 tempest-VolumesGetTest-1961901732 None] Geting {'code': 200, 'error': None, 'data': {'origin': 'Pool-0/v_9a8586a6-f2f5-4bee-a56f-a06b01120c39@v_fa8e71d4-54a7-4e0d-9a26-50cc7946ce11', 'relatime': None, 'acltype': None, 'vscan': None, 'full_name': 'Pool-0/v_fa8e71d4-54a7-4e0d-9a26-50cc7946ce11', 'userrefs': None, 'primarycache': 'all', 'logbias': 'latency', 'creation': '1681746464', 'sync': 'always', 'is_clone': True, 'dedup': 'off', 'sharenfs': None, 'receive_resume_token': None, 'volsize': '1073741824', 'referenced': '57344', 'sharesmb': None, 'createtxg': '187930', 'reservation': '0', 'scontext': None, 'mountpoint': None, 'casesensitivity': None, 'guid': '1118096299032255136', 'usedbyrefreservation': '0', 'dnodesize': None, 'written': '0', 'logicalused': '0', 'compressratio': '1.00', 'rootcontext': 'none', 'default_scsi_id': '405b5facf109dfda', 'type': 'volume', 'compression': 'lz4', 'snapdir': None, 'overlay': None, 'encryption': 'off', 'xattr': None, 'volmode': 'default', 'copies': '1', 'snapshot_limit': '18446744073709551615', 'aclinherit': None, 'defcontext': 'none', 'readonly': 'off', 'version': None, 'recordsize': None, 'filesystem_limit': None, 'mounted': None, 'mlslabel': 'none', 'secondarycache': 'all', 'refreservation': '0', 'available': '113909792768', 'san:volume_id': '405b5facf109dfda8d99602cc7e7a97c113a2ad5', 'encryptionroot': None, 'exec': None, 'refquota': None, 'refcompressratio': '1.00', 'quota': None, 'utf8only': None, 'keylocation': 'none', 'snapdev': 'hidden', 'snapshot_count': '18446744073709551615', 'fscontext': 'none', 'clones': None, 'canmount': None, 'keystatus': None, 'atime': None, 'usedbysnapshots': '0', 'normalization': None, 'usedbychildren': '0', 'volblocksize': '131072', 'usedbydataset': '0', 'objsetid': '123401', 'name': 'v_fa8e71d4-54a7-4e0d-9a26-50cc7946ce11', 'defer_destroy': None, 'pbkdf2iters': '0', 'checksum': 'on', 'redundant_metadata': 'all', 'filesystem_count': None, 'devices': None, 'keyformat': 'none', 'setuid': None, 'used': '0', 'logicalreferenced': '28672', 'context': 'none', 'zoned': None, 'nbmand': None}} from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_fa8e71d4-54a7-4e0d-9a26-50cc7946ce11 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:24:41 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-54dde4b2-8343-4ef5-8f0f-d617a0a46a99 tempest-VolumesGetTest-1961901732 None] delete snapshot v_fa8e71d4-54a7-4e0d-9a26-50cc7946ce11 of volume v_9a8586a6-f2f5-4bee-a56f-a06b01120c39 {{(pid=84696) delete_snapshot /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:751}} Apr 17 15:24:41 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-54dde4b2-8343-4ef5-8f0f-d617a0a46a99 tempest-VolumesGetTest-1961901732 None] Sending pool request DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_9a8586a6-f2f5-4bee-a56f-a06b01120c39/snapshots/v_fa8e71d4-54a7-4e0d-9a26-50cc7946ce11 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:24:41 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-54dde4b2-8343-4ef5-8f0f-d617a0a46a99 tempest-VolumesGetTest-1961901732 None] Sending DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_9a8586a6-f2f5-4bee-a56f-a06b01120c39/snapshots/v_fa8e71d4-54a7-4e0d-9a26-50cc7946ce11 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:24:41 user cinder-volume[84696]: INFO cinder.volume.manager [None req-5317e26e-7867-4611-b396-2346612ed3c4 tempest-GroupSnapshotsTest-1926061586 None] GroupSnapshot f6413371-b635-4f3c-940c-0c0795e59bef: creating. Apr 17 15:24:42 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-5317e26e-7867-4611-b396-2346612ed3c4 tempest-GroupSnapshotsTest-1926061586 None] Group snapshot f6413371-b635-4f3c-940c-0c0795e59bef: creating. {{(pid=84696) create_group_snapshot /opt/stack/cinder/cinder/volume/manager.py:4129}} Apr 17 15:24:42 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-5317e26e-7867-4611-b396-2346612ed3c4 tempest-GroupSnapshotsTest-1926061586 None] create snapshot c9995e45-166a-4b06-8340-f012f00d640c for volume 7a71a6fd-8167-40ab-b057-5f82e0f55947 {{(pid=84696) create_snapshot /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:625}} Apr 17 15:24:42 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-5317e26e-7867-4611-b396-2346612ed3c4 tempest-GroupSnapshotsTest-1926061586 None] cloning v_7a71a6fd-8167-40ab-b057-5f82e0f55947 to s_c9995e45-166a-4b06-8340-f012f00d640c {{(pid=84696) _clone_object /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:456}} Apr 17 15:24:42 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-5317e26e-7867-4611-b396-2346612ed3c4 tempest-GroupSnapshotsTest-1926061586 None] create snapshot s_c9995e45-166a-4b06-8340-f012f00d640c {{(pid=84696) create_snapshot /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:635}} Apr 17 15:24:42 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5317e26e-7867-4611-b396-2346612ed3c4 tempest-GroupSnapshotsTest-1926061586 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_7a71a6fd-8167-40ab-b057-5f82e0f55947/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:24:42 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5317e26e-7867-4611-b396-2346612ed3c4 tempest-GroupSnapshotsTest-1926061586 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_7a71a6fd-8167-40ab-b057-5f82e0f55947/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:24:43 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-274966fd-0736-4b94-9044-21d5db896776 tempest-VolumesExtendTest-93540250 None] Geting {'code': 200, 'error': None, 'data': {'name': 'v_4b0ebe2e-141a-40f1-bcb5-83708d103f54'}} from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:24:43 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-274966fd-0736-4b94-9044-21d5db896776 tempest-VolumesExtendTest-93540250 None] create volume v_4b0ebe2e-141a-40f1-bcb5-83708d103f54 from snapshot v_4b0ebe2e-141a-40f1-bcb5-83708d103f54 {{(pid=84696) create_volume_from_snapshot /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:674}} Apr 17 15:24:43 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-274966fd-0736-4b94-9044-21d5db896776 tempest-VolumesExtendTest-93540250 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/clone {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:24:43 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-274966fd-0736-4b94-9044-21d5db896776 tempest-VolumesExtendTest-93540250 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/clone {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:24:43 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5317e26e-7867-4611-b396-2346612ed3c4 tempest-GroupSnapshotsTest-1926061586 None] Geting {'code': 200, 'error': None, 'data': {'name': 's_c9995e45-166a-4b06-8340-f012f00d640c'}} from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_7a71a6fd-8167-40ab-b057-5f82e0f55947/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:24:43 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-5317e26e-7867-4611-b396-2346612ed3c4 tempest-GroupSnapshotsTest-1926061586 None] create volume s_c9995e45-166a-4b06-8340-f012f00d640c from snapshot s_c9995e45-166a-4b06-8340-f012f00d640c {{(pid=84696) create_volume_from_snapshot /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:674}} Apr 17 15:24:43 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5317e26e-7867-4611-b396-2346612ed3c4 tempest-GroupSnapshotsTest-1926061586 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_7a71a6fd-8167-40ab-b057-5f82e0f55947/clone {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:24:43 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5317e26e-7867-4611-b396-2346612ed3c4 tempest-GroupSnapshotsTest-1926061586 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_7a71a6fd-8167-40ab-b057-5f82e0f55947/clone {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:24:44 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-54dde4b2-8343-4ef5-8f0f-d617a0a46a99 tempest-VolumesGetTest-1961901732 None] Geting {'code': 204} from DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_9a8586a6-f2f5-4bee-a56f-a06b01120c39/snapshots/v_fa8e71d4-54a7-4e0d-9a26-50cc7946ce11 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:24:44 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-54dde4b2-8343-4ef5-8f0f-d617a0a46a99 tempest-VolumesGetTest-1961901732 None] snapshot v_fa8e71d4-54a7-4e0d-9a26-50cc7946ce11 deleted {{(pid=84696) delete_snapshot /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:772}} Apr 17 15:24:44 user cinder-volume[84696]: DEBUG cinder.quota [None req-54dde4b2-8343-4ef5-8f0f-d617a0a46a99 tempest-VolumesGetTest-1961901732 None] Created reservations ['19da3165-d4e5-4241-adcb-6f92769d0e75', '7bc31620-17c8-4891-8870-644affa847ae', '0848d300-4f81-4343-b584-541b4a6e5c72', '868a2638-c933-4983-81a2-677c8023c1b2'] {{(pid=84696) reserve /opt/stack/cinder/cinder/quota.py:755}} Apr 17 15:24:44 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-274966fd-0736-4b94-9044-21d5db896776 tempest-VolumesExtendTest-93540250 None] Geting {'code': 200, 'error': None, 'data': {'origin': 'Pool-0/v_006f20ae-5306-44fe-afdf-e433f79284e8@v_4b0ebe2e-141a-40f1-bcb5-83708d103f54', 'is_clone': True, 'full_name': 'Pool-0/v_4b0ebe2e-141a-40f1-bcb5-83708d103f54', 'name': 'v_4b0ebe2e-141a-40f1-bcb5-83708d103f54'}} from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/clone {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:24:44 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-274966fd-0736-4b94-9044-21d5db896776 tempest-VolumesExtendTest-93540250 None] get volume v_4b0ebe2e-141a-40f1-bcb5-83708d103f54 info {{(pid=84696) get_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:239}} Apr 17 15:24:44 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-274966fd-0736-4b94-9044-21d5db896776 tempest-VolumesExtendTest-93540250 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_4b0ebe2e-141a-40f1-bcb5-83708d103f54 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:24:44 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-274966fd-0736-4b94-9044-21d5db896776 tempest-VolumesExtendTest-93540250 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_4b0ebe2e-141a-40f1-bcb5-83708d103f54 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:24:44 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-274966fd-0736-4b94-9044-21d5db896776 tempest-VolumesExtendTest-93540250 None] Geting {'code': 200, 'error': None, 'data': {'origin': 'Pool-0/v_006f20ae-5306-44fe-afdf-e433f79284e8@v_4b0ebe2e-141a-40f1-bcb5-83708d103f54', 'relatime': None, 'acltype': None, 'vscan': None, 'full_name': 'Pool-0/v_4b0ebe2e-141a-40f1-bcb5-83708d103f54', 'userrefs': None, 'primarycache': 'all', 'logbias': 'latency', 'creation': '1681746477', 'sync': 'always', 'is_clone': True, 'dedup': 'off', 'sharenfs': None, 'receive_resume_token': None, 'volsize': '1073741824', 'referenced': '16285696', 'sharesmb': None, 'createtxg': '187958', 'reservation': '0', 'scontext': None, 'mountpoint': None, 'casesensitivity': None, 'guid': '13728919998564651037', 'usedbyrefreservation': '0', 'dnodesize': None, 'written': '0', 'logicalused': '0', 'compressratio': '1.00', 'rootcontext': 'none', 'default_scsi_id': 'c4eb5e6968b5de3c', 'type': 'volume', 'compression': 'lz4', 'snapdir': None, 'overlay': None, 'encryption': 'off', 'xattr': None, 'volmode': 'default', 'copies': '1', 'snapshot_limit': '18446744073709551615', 'aclinherit': None, 'defcontext': 'none', 'readonly': 'off', 'version': None, 'recordsize': None, 'filesystem_limit': None, 'mounted': None, 'mlslabel': 'none', 'secondarycache': 'all', 'refreservation': '0', 'available': '113909743616', 'used': '0', 'encryptionroot': None, 'exec': None, 'refquota': None, 'refcompressratio': '1.17', 'quota': None, 'utf8only': None, 'keylocation': 'none', 'snapdev': 'hidden', 'snapshot_count': '18446744073709551615', 'fscontext': 'none', 'clones': None, 'canmount': None, 'keystatus': None, 'atime': None, 'usedbysnapshots': '0', 'normalization': None, 'usedbychildren': '0', 'volblocksize': '131072', 'usedbydataset': '0', 'objsetid': '122538', 'name': 'v_4b0ebe2e-141a-40f1-bcb5-83708d103f54', 'defer_destroy': None, 'pbkdf2iters': '0', 'checksum': 'on', 'redundant_metadata': 'all', 'filesystem_count': None, 'devices': None, 'keyformat': 'none', 'setuid': None, 'logicalreferenced': '19042304', 'context': 'none', 'zoned': None, 'nbmand': None}} from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_4b0ebe2e-141a-40f1-bcb5-83708d103f54 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:24:44 user cinder-volume[84696]: DEBUG cinder.volume.volume_utils [None req-274966fd-0736-4b94-9044-21d5db896776 tempest-VolumesExtendTest-93540250 None] Marking volume 4b0ebe2e-141a-40f1-bcb5-83708d103f54 as bootable. {{(pid=84696) enable_bootable_flag /opt/stack/cinder/cinder/volume/volume_utils.py:1140}} Apr 17 15:24:44 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-274966fd-0736-4b94-9044-21d5db896776 tempest-VolumesExtendTest-93540250 None] Copying metadata from image b3f0dec1-7758-44a8-82e0-76ada8837428 to 4b0ebe2e-141a-40f1-bcb5-83708d103f54. {{(pid=84696) _handle_bootable_volume_glance_meta /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:452}} Apr 17 15:24:44 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-274966fd-0736-4b94-9044-21d5db896776 tempest-VolumesExtendTest-93540250 None] Creating volume glance metadata for volume 4b0ebe2e-141a-40f1-bcb5-83708d103f54 backed by image b3f0dec1-7758-44a8-82e0-76ada8837428 with: {'hw_rng_model': 'virtio', 'owner_specified.openstack.md5': '', 'owner_specified.openstack.object': 'images/cirros-0.5.2-x86_64-disk', 'owner_specified.openstack.sha256': '', 'image_id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'image_name': 'cirros-0.5.2-x86_64-disk', 'checksum': 'b874c39491a2377b8490f5f1e89761a4', 'container_format': 'bare', 'disk_format': 'qcow2', 'min_disk': 0, 'min_ram': 0, 'size': 16300544}. {{(pid=84696) _capture_volume_image_metadata /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:683}} Apr 17 15:24:45 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-54dde4b2-8343-4ef5-8f0f-d617a0a46a99 tempest-VolumesGetTest-1961901732 None] Updating volume stats {{(pid=84696) _update_volume_stats /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:724}} Apr 17 15:24:45 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-54dde4b2-8343-4ef5-8f0f-d617a0a46a99 tempest-VolumesGetTest-1961901732 None] Get pool Pool-0 fsprops {{(pid=84696) get_pool_stats /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:894}} Apr 17 15:24:45 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-54dde4b2-8343-4ef5-8f0f-d617a0a46a99 tempest-VolumesGetTest-1961901732 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:24:45 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-54dde4b2-8343-4ef5-8f0f-d617a0a46a99 tempest-VolumesGetTest-1961901732 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:24:45 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-274966fd-0736-4b94-9044-21d5db896776 tempest-VolumesExtendTest-93540250 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (29ba5341-ef6b-4c3b-8b09-179ebb0a2af9) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'volume_name': 'volume-4b0ebe2e-141a-40f1-bcb5-83708d103f54', 'volume_size': 1, 'image_id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'image_location': (None, None), 'image_meta': {'name': 'cirros-0.5.2-x86_64-disk', 'disk_format': 'qcow2', 'container_format': 'bare', 'visibility': 'public', 'size': 16300544, 'virtual_size': 117440512, 'status': 'active', 'checksum': 'b874c39491a2377b8490f5f1e89761a4', 'protected': False, 'min_ram': 0, 'min_disk': 0, 'owner': 'a43c211b4ed542538799349489b1d884', 'os_hidden': False, 'os_hash_algo': 'sha512', 'os_hash_value': '6b813aa46bb90b4da216a4d19376593fa3f4fc7e617f03a92b7fe11e9a3981cbe8f0959dbebe36225e5f53dc4492341a4863cac4ed1ee0909f3fc78ef9c3e869', 'id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'created_at': datetime.datetime(2023, 4, 17, 15, 19, 53, tzinfo=datetime.timezone.utc), 'updated_at': datetime.datetime(2023, 4, 17, 15, 19, 55, tzinfo=datetime.timezone.utc), 'tags': [], 'file': '/v2/images/b3f0dec1-7758-44a8-82e0-76ada8837428/file', 'properties': {'hw_rng_model': 'virtio', 'owner_specified.openstack.md5': '', 'owner_specified.openstack.object': 'images/cirros-0.5.2-x86_64-disk', 'owner_specified.openstack.sha256': ''}}, 'image_service': }' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:24:45 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-274966fd-0736-4b94-9044-21d5db896776 tempest-VolumesExtendTest-93540250 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeOnFinishTask;volume:create, create.end' (0f9caa11-f822-425e-9075-1118516bf568) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:24:45 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=54,cinder:UPDATE=32,cinder:INSERT=14 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:24:45 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-274966fd-0736-4b94-9044-21d5db896776 tempest-VolumesExtendTest-93540250 None] Volume volume-4b0ebe2e-141a-40f1-bcb5-83708d103f54 (4b0ebe2e-141a-40f1-bcb5-83708d103f54): created successfully Apr 17 15:24:45 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-274966fd-0736-4b94-9044-21d5db896776 tempest-VolumesExtendTest-93540250 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeOnFinishTask;volume:create, create.end' (0f9caa11-f822-425e-9075-1118516bf568) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:24:45 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-274966fd-0736-4b94-9044-21d5db896776 tempest-VolumesExtendTest-93540250 None] Flow 'volume_create_manager' (5d41bdb9-721d-4cc7-b18e-75868a6ee8f5) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:24:45 user cinder-volume[84696]: INFO cinder.volume.manager [None req-274966fd-0736-4b94-9044-21d5db896776 tempest-VolumesExtendTest-93540250 None] Created volume successfully. Apr 17 15:24:45 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5317e26e-7867-4611-b396-2346612ed3c4 tempest-GroupSnapshotsTest-1926061586 None] Geting {'code': 200, 'error': None, 'data': {'origin': 'Pool-0/v_7a71a6fd-8167-40ab-b057-5f82e0f55947@s_c9995e45-166a-4b06-8340-f012f00d640c', 'is_clone': True, 'full_name': 'Pool-0/s_c9995e45-166a-4b06-8340-f012f00d640c', 'name': 's_c9995e45-166a-4b06-8340-f012f00d640c'}} from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_7a71a6fd-8167-40ab-b057-5f82e0f55947/clone {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:24:45 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5317e26e-7867-4611-b396-2346612ed3c4 tempest-GroupSnapshotsTest-1926061586 None] Sending pool request PUT to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/s_c9995e45-166a-4b06-8340-f012f00d640c/properties {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:24:45 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5317e26e-7867-4611-b396-2346612ed3c4 tempest-GroupSnapshotsTest-1926061586 None] Sending PUT to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/s_c9995e45-166a-4b06-8340-f012f00d640c/properties {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:24:45 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5317e26e-7867-4611-b396-2346612ed3c4 tempest-GroupSnapshotsTest-1926061586 None] Geting {'code': 201, 'error': None, 'data': None} from PUT to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/s_c9995e45-166a-4b06-8340-f012f00d640c/properties {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:24:45 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-5317e26e-7867-4611-b396-2346612ed3c4 tempest-GroupSnapshotsTest-1926061586 None] volume s_c9995e45-166a-4b06-8340-f012f00d640c properties updated {{(pid=84696) modify_property_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:301}} Apr 17 15:24:45 user cinder-volume[84696]: DEBUG cinder.coordination [None req-50e02017-ce73-4ed4-9f96-0a01c81adea4 tempest-VolumesGetTest-1961901732 None] Acquiring lock "cinder-9a8586a6-f2f5-4bee-a56f-a06b01120c39-delete_volume" by "delete_volume" {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:192}} Apr 17 15:24:45 user cinder-volume[84696]: DEBUG cinder.coordination [None req-50e02017-ce73-4ed4-9f96-0a01c81adea4 tempest-VolumesGetTest-1961901732 None] Lock "cinder-9a8586a6-f2f5-4bee-a56f-a06b01120c39-delete_volume" acquired by "delete_volume" :: waited 0.021s {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:196}} Apr 17 15:24:45 user cinder-volume[84696]: INFO cinder.volume.manager [None req-5317e26e-7867-4611-b396-2346612ed3c4 tempest-GroupSnapshotsTest-1926061586 None] group_snapshot f6413371-b635-4f3c-940c-0c0795e59bef: created successfully Apr 17 15:24:45 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-50e02017-ce73-4ed4-9f96-0a01c81adea4 tempest-VolumesGetTest-1961901732 None] remove_export for volume: 9a8586a6-f2f5-4bee-a56f-a06b01120c39. {{(pid=84696) remove_export /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:718}} Apr 17 15:24:45 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-50e02017-ce73-4ed4-9f96-0a01c81adea4 tempest-VolumesGetTest-1961901732 None] remove export {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:912}} Apr 17 15:24:45 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-50e02017-ce73-4ed4-9f96-0a01c81adea4 tempest-VolumesGetTest-1961901732 None] detach volume:Volume(_name_id=None,admin_metadata={},attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:24:25Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesGetTest-Volume-1667230460',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=9a8586a6-f2f5-4bee-a56f-a06b01120c39,launched_at=2023-04-17T15:24:28Z,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='a65d14b0a2894a6f90506d86feaefd15',provider_auth='CHAP fepHiPWc Gjbfk2vNdA6Tst',provider_geometry=None,provider_id=None,provider_location='172.16.0.220:3260,1 iqn.2010-10.org.openstack:9a8586a6-f2f5-4bee-a56f-a06b01120c39 0',replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:24:26Z,service_uuid='e628352e-c4cf-40eb-8389-cce40663026a',shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='deleting',terminated_at=2023-04-17T15:24:45Z,updated_at=2023-04-17T15:24:45Z,use_quota=True,user_id='6ffa3051570a4721a48a0171b3cc4d16',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c) from target:iqn.2010-10.org.openstack:9a8586a6-f2f5-4bee-a56f-a06b01120c39. {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:913}} Apr 17 15:24:45 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-50e02017-ce73-4ed4-9f96-0a01c81adea4 tempest-VolumesGetTest-1961901732 None] detach volume v_9a8586a6-f2f5-4bee-a56f-a06b01120c39 from target iqn.2010-10.org.openstack:9a8586a6-f2f5-4bee-a56f-a06b01120c39 {{(pid=84696) detach_target_vol /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:606}} Apr 17 15:24:45 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-50e02017-ce73-4ed4-9f96-0a01c81adea4 tempest-VolumesGetTest-1961901732 None] Sending pool request DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:9a8586a6-f2f5-4bee-a56f-a06b01120c39/luns/v_9a8586a6-f2f5-4bee-a56f-a06b01120c39 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:24:45 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-50e02017-ce73-4ed4-9f96-0a01c81adea4 tempest-VolumesGetTest-1961901732 None] Sending DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:9a8586a6-f2f5-4bee-a56f-a06b01120c39/luns/v_9a8586a6-f2f5-4bee-a56f-a06b01120c39 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:24:45 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-50e02017-ce73-4ed4-9f96-0a01c81adea4 tempest-VolumesGetTest-1961901732 None] Geting {'code': 404, 'error': {'class': 'opene.exceptions.ItemNotFoundError', 'message': 'Target iqn.2010-10.org.openstack:9a8586a6-f2f5-4bee-a56f-a06b01120c39 not exists.', 'url': 'http://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:9a8586a6-f2f5-4bee-a56f-a06b01120c39/luns/v_9a8586a6-f2f5-4bee-a56f-a06b01120c39'}, 'data': None} from DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:9a8586a6-f2f5-4bee-a56f-a06b01120c39/luns/v_9a8586a6-f2f5-4bee-a56f-a06b01120c39 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:24:45 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-50e02017-ce73-4ed4-9f96-0a01c81adea4 tempest-VolumesGetTest-1961901732 None] failed to remove resource iqn.2010-10.org.openstack:9a8586a6-f2f5-4bee-a56f-a06b01120c39 because of JDSS resource v_9a8586a6-f2f5-4bee-a56f-a06b01120c39 DNE. {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:920}} Apr 17 15:24:45 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-50e02017-ce73-4ed4-9f96-0a01c81adea4 tempest-VolumesGetTest-1961901732 None] delete target: iqn.2010-10.org.openstack:9a8586a6-f2f5-4bee-a56f-a06b01120c39 {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:930}} Apr 17 15:24:45 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-50e02017-ce73-4ed4-9f96-0a01c81adea4 tempest-VolumesGetTest-1961901732 None] delete target iqn.2010-10.org.openstack:9a8586a6-f2f5-4bee-a56f-a06b01120c39 {{(pid=84696) delete_target /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:442}} Apr 17 15:24:45 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-50e02017-ce73-4ed4-9f96-0a01c81adea4 tempest-VolumesGetTest-1961901732 None] Sending pool request DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:9a8586a6-f2f5-4bee-a56f-a06b01120c39 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:24:45 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-50e02017-ce73-4ed4-9f96-0a01c81adea4 tempest-VolumesGetTest-1961901732 None] Sending DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:9a8586a6-f2f5-4bee-a56f-a06b01120c39 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:24:45 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-50e02017-ce73-4ed4-9f96-0a01c81adea4 tempest-VolumesGetTest-1961901732 None] Geting {'code': 404, 'error': {'class': 'opene.exceptions.ItemNotFoundError', 'message': 'Target iqn.2010-10.org.openstack:9a8586a6-f2f5-4bee-a56f-a06b01120c39 not exists.', 'url': 'http://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:9a8586a6-f2f5-4bee-a56f-a06b01120c39'}, 'data': None} from DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:9a8586a6-f2f5-4bee-a56f-a06b01120c39 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:24:45 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-50e02017-ce73-4ed4-9f96-0a01c81adea4 tempest-VolumesGetTest-1961901732 None] failed to remove resource iqn.2010-10.org.openstack:9a8586a6-f2f5-4bee-a56f-a06b01120c39 because of JDSS resource iqn.2010-10.org.openstack:9a8586a6-f2f5-4bee-a56f-a06b01120c39 DNE. {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:935}} Apr 17 15:24:45 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-50e02017-ce73-4ed4-9f96-0a01c81adea4 tempest-VolumesGetTest-1961901732 None] deleating volume v_9a8586a6-f2f5-4bee-a56f-a06b01120c39 {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 15:24:45 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-50e02017-ce73-4ed4-9f96-0a01c81adea4 tempest-VolumesGetTest-1961901732 None] get snapshots for volume v_9a8586a6-f2f5-4bee-a56f-a06b01120c39 {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 15:24:45 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-50e02017-ce73-4ed4-9f96-0a01c81adea4 tempest-VolumesGetTest-1961901732 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_9a8586a6-f2f5-4bee-a56f-a06b01120c39/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:24:45 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-50e02017-ce73-4ed4-9f96-0a01c81adea4 tempest-VolumesGetTest-1961901732 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_9a8586a6-f2f5-4bee-a56f-a06b01120c39/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:24:45 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-43baaf2b-5b0a-4273-b977-215eaa56cd2d tempest-VolumesExtendTest-93540250 None] Extend volume 4b0ebe2e-141a-40f1-bcb5-83708d103f54 {{(pid=84696) extend_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:401}} Apr 17 15:24:45 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-43baaf2b-5b0a-4273-b977-215eaa56cd2d tempest-VolumesExtendTest-93540250 None] jdss extend volume v_4b0ebe2e-141a-40f1-bcb5-83708d103f54 to 2147483648 {{(pid=84696) extend_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:160}} Apr 17 15:24:45 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-43baaf2b-5b0a-4273-b977-215eaa56cd2d tempest-VolumesExtendTest-93540250 None] Sending pool request PUT to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_4b0ebe2e-141a-40f1-bcb5-83708d103f54 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:24:45 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-43baaf2b-5b0a-4273-b977-215eaa56cd2d tempest-VolumesExtendTest-93540250 None] Sending PUT to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_4b0ebe2e-141a-40f1-bcb5-83708d103f54 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:24:46 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-43baaf2b-5b0a-4273-b977-215eaa56cd2d tempest-VolumesExtendTest-93540250 None] Geting {'code': 201, 'error': None, 'data': None} from PUT to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_4b0ebe2e-141a-40f1-bcb5-83708d103f54 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:24:46 user cinder-volume[84696]: INFO cinder.volume.manager [None req-43baaf2b-5b0a-4273-b977-215eaa56cd2d tempest-VolumesExtendTest-93540250 None] Extend volume completed successfully. Apr 17 15:24:46 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-54dde4b2-8343-4ef5-8f0f-d617a0a46a99 tempest-VolumesGetTest-1961901732 None] Geting {'code': 200, 'error': None, 'data': {'available': '113909731328', 'status': 30, 'import_status': {'import_steps': {'san_setup': True, 'vip_setup': True, 'zfs_import': True, 'mount_system_volume': True, 'nas_setup': True}, 'import_successful': True}, 'scan': None, 'encryption': {'enabled': False}, 'deduplication_rate': '1.00', 'sysvol_upgrade': 'not_needed', 'vdevs': [{'name': 'wwn-0x6000c296b793bf526a107a9d823e947a', 'iostats': {'read': '0', 'write': '0', 'chksum': '0'}, 'disks': [{'origin': 'local', 'slot': '', 'led': 'off', 'name': 'sdn', 'iostats': {'read': '0', 'write': '0', 'chksum': '0'}, 'alias': '', 'health': 'ONLINE', 'sn': '6000c296b793bf526a107a9d823e947a', 'trim_data': {'status': 'trim_unsupported', 'progress': None, 'trim_time': None}, 'path': 'pci-0000:03:00.0-scsi-0:0:14:0', 'model': 'Virtual disk', 'id': 'wwn-0x6000c296b793bf526a107a9d823e947a', 'size': 128849018880}], 'health': 'ONLINE', 'vdev_replacings': [], 'vdev_spares': [], 'type': ''}], 'id': '14181317334440137330', 'health': 'ONLINE', 'iostats': {'read': '0', 'write': '0', 'chksum': '0'}, 'operation': 'none', 'size': '127775277056', 'autotrim': False, 'name': 'Pool-0'}} from GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:24:46 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-54dde4b2-8343-4ef5-8f0f-d617a0a46a99 tempest-VolumesGetTest-1961901732 None] Total capacity: 119, Free 106. {{(pid=84696) _update_volume_stats /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:757}} Apr 17 15:24:46 user cinder-volume[84696]: DEBUG cinder.manager [None req-54dde4b2-8343-4ef5-8f0f-d617a0a46a99 tempest-VolumesGetTest-1961901732 None] Notifying Schedulers of capabilities ... {{(pid=84696) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 17 15:24:46 user cinder-volume[84696]: INFO cinder.volume.manager [None req-54dde4b2-8343-4ef5-8f0f-d617a0a46a99 tempest-VolumesGetTest-1961901732 None] Deleted volume successfully. Apr 17 15:24:46 user cinder-volume[84696]: DEBUG cinder.coordination [None req-54dde4b2-8343-4ef5-8f0f-d617a0a46a99 tempest-VolumesGetTest-1961901732 None] Lock "cinder-fa8e71d4-54a7-4e0d-9a26-50cc7946ce11-delete_volume" released by "delete_volume" :: held 5.451s {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:207}} Apr 17 15:24:46 user cinder-volume[84696]: INFO cinder.volume.manager [None req-804a4aab-b03c-4885-b758-db492f88f09b tempest-GroupSnapshotsTest-1926061586 None] group_snapshot f6413371-b635-4f3c-940c-0c0795e59bef: deleting Apr 17 15:24:46 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-804a4aab-b03c-4885-b758-db492f88f09b tempest-GroupSnapshotsTest-1926061586 None] group_snapshot f6413371-b635-4f3c-940c-0c0795e59bef: deleting {{(pid=84696) delete_group_snapshot /opt/stack/cinder/cinder/volume/manager.py:4304}} Apr 17 15:24:46 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-50e02017-ce73-4ed4-9f96-0a01c81adea4 tempest-VolumesGetTest-1961901732 None] Geting {'code': 200, 'error': None, 'data': {'results': 0, 'entries': []}} from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_9a8586a6-f2f5-4bee-a56f-a06b01120c39/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:24:46 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-50e02017-ce73-4ed4-9f96-0a01c81adea4 tempest-VolumesGetTest-1961901732 None] get volume v_9a8586a6-f2f5-4bee-a56f-a06b01120c39 info {{(pid=84696) get_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:239}} Apr 17 15:24:46 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-50e02017-ce73-4ed4-9f96-0a01c81adea4 tempest-VolumesGetTest-1961901732 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_9a8586a6-f2f5-4bee-a56f-a06b01120c39 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:24:46 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-50e02017-ce73-4ed4-9f96-0a01c81adea4 tempest-VolumesGetTest-1961901732 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_9a8586a6-f2f5-4bee-a56f-a06b01120c39 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:24:46 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-804a4aab-b03c-4885-b758-db492f88f09b tempest-GroupSnapshotsTest-1926061586 None] deleating snapshot s_c9995e45-166a-4b06-8340-f012f00d640c. {{(pid=84696) delete_snapshot /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:649}} Apr 17 15:24:46 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-804a4aab-b03c-4885-b758-db492f88f09b tempest-GroupSnapshotsTest-1926061586 None] get snapshots for volume s_c9995e45-166a-4b06-8340-f012f00d640c {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 15:24:46 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-804a4aab-b03c-4885-b758-db492f88f09b tempest-GroupSnapshotsTest-1926061586 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/s_c9995e45-166a-4b06-8340-f012f00d640c/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:24:46 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-804a4aab-b03c-4885-b758-db492f88f09b tempest-GroupSnapshotsTest-1926061586 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/s_c9995e45-166a-4b06-8340-f012f00d640c/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:24:46 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-50e02017-ce73-4ed4-9f96-0a01c81adea4 tempest-VolumesGetTest-1961901732 None] Geting {'code': 200, 'error': None, 'data': {'origin': None, 'relatime': None, 'acltype': None, 'vscan': None, 'full_name': 'Pool-0/v_9a8586a6-f2f5-4bee-a56f-a06b01120c39', 'userrefs': None, 'primarycache': 'all', 'logbias': 'latency', 'creation': '1681746460', 'sync': 'always', 'is_clone': False, 'dedup': 'off', 'sharenfs': None, 'receive_resume_token': None, 'volsize': '1073741824', 'referenced': '57344', 'sharesmb': None, 'createtxg': '187914', 'reservation': '0', 'scontext': None, 'mountpoint': None, 'casesensitivity': None, 'guid': '14168385156902874606', 'usedbyrefreservation': '0', 'dnodesize': None, 'written': '57344', 'logicalused': '28672', 'compressratio': '1.00', 'rootcontext': 'none', 'default_scsi_id': 'a36d51abdbfddb00', 'type': 'volume', 'compression': 'lz4', 'snapdir': None, 'overlay': None, 'encryption': 'off', 'xattr': None, 'volmode': 'default', 'copies': '1', 'snapshot_limit': '18446744073709551615', 'aclinherit': None, 'defcontext': 'none', 'readonly': 'off', 'version': None, 'recordsize': None, 'filesystem_limit': None, 'mounted': None, 'mlslabel': 'none', 'secondarycache': 'all', 'refreservation': '0', 'available': '113909649408', 'san:volume_id': 'a36d51abdbfddb00f4cd9cd01d1c36f1c04cdfb1', 'encryptionroot': None, 'exec': None, 'refquota': None, 'refcompressratio': '1.00', 'quota': None, 'utf8only': None, 'keylocation': 'none', 'snapdev': 'hidden', 'snapshot_count': '18446744073709551615', 'fscontext': 'none', 'clones': None, 'canmount': None, 'keystatus': None, 'atime': None, 'usedbysnapshots': '0', 'normalization': None, 'usedbychildren': '0', 'volblocksize': '131072', 'usedbydataset': '57344', 'objsetid': '122775', 'name': 'v_9a8586a6-f2f5-4bee-a56f-a06b01120c39', 'defer_destroy': None, 'pbkdf2iters': '0', 'checksum': 'on', 'redundant_metadata': 'all', 'filesystem_count': None, 'devices': None, 'keyformat': 'none', 'setuid': None, 'used': '57344', 'logicalreferenced': '28672', 'context': 'none', 'zoned': None, 'nbmand': None}} from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_9a8586a6-f2f5-4bee-a56f-a06b01120c39 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:24:46 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-50e02017-ce73-4ed4-9f96-0a01c81adea4 tempest-VolumesGetTest-1961901732 None] delete volume:v_9a8586a6-f2f5-4bee-a56f-a06b01120c39 recursively children:{'force_umount': True} {{(pid=84696) delete_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:335}} Apr 17 15:24:46 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-50e02017-ce73-4ed4-9f96-0a01c81adea4 tempest-VolumesGetTest-1961901732 None] Sending pool request DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_9a8586a6-f2f5-4bee-a56f-a06b01120c39 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:24:46 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-50e02017-ce73-4ed4-9f96-0a01c81adea4 tempest-VolumesGetTest-1961901732 None] Sending DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_9a8586a6-f2f5-4bee-a56f-a06b01120c39 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:24:47 user cinder-volume[84696]: DEBUG cinder.coordination [None req-8502c4c2-1b69-4858-a3b9-7f3876102a3e tempest-VolumesExtendTest-93540250 None] Acquiring lock "cinder-4b0ebe2e-141a-40f1-bcb5-83708d103f54-delete_volume" by "delete_volume" {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:192}} Apr 17 15:24:47 user cinder-volume[84696]: DEBUG cinder.coordination [None req-8502c4c2-1b69-4858-a3b9-7f3876102a3e tempest-VolumesExtendTest-93540250 None] Lock "cinder-4b0ebe2e-141a-40f1-bcb5-83708d103f54-delete_volume" acquired by "delete_volume" :: waited 0.016s {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:196}} Apr 17 15:24:47 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-8502c4c2-1b69-4858-a3b9-7f3876102a3e tempest-VolumesExtendTest-93540250 None] remove_export for volume: 4b0ebe2e-141a-40f1-bcb5-83708d103f54. {{(pid=84696) remove_export /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:718}} Apr 17 15:24:47 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-8502c4c2-1b69-4858-a3b9-7f3876102a3e tempest-VolumesExtendTest-93540250 None] remove export {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:912}} Apr 17 15:24:47 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-8502c4c2-1b69-4858-a3b9-7f3876102a3e tempest-VolumesExtendTest-93540250 None] detach volume:Volume(_name_id=None,admin_metadata={},attach_status='detached',availability_zone='nova',bootable=True,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:24:41Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesExtendTest-Volume-1116151627',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=4b0ebe2e-141a-40f1-bcb5-83708d103f54,launched_at=2023-04-17T15:24:45Z,metadata={},migration_status=None,multiattach=False,previous_status='available',project_id='ef25ef72b5a64e2c8a60d3d139367fd1',provider_auth='CHAP PDWLpTOZ OclF4oghUrPqXn',provider_geometry=None,provider_id=None,provider_location='172.16.0.220:3260,1 iqn.2010-10.org.openstack:4b0ebe2e-141a-40f1-bcb5-83708d103f54 0',replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:24:41Z,service_uuid='e628352e-c4cf-40eb-8389-cce40663026a',shared_targets=True,size=2,snapshot_id=None,snapshots=,source_volid=None,status='deleting',terminated_at=2023-04-17T15:24:47Z,updated_at=2023-04-17T15:24:47Z,use_quota=True,user_id='d473e2b76b554904a4d625f35a78c038',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c) from target:iqn.2010-10.org.openstack:4b0ebe2e-141a-40f1-bcb5-83708d103f54. {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:913}} Apr 17 15:24:47 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-8502c4c2-1b69-4858-a3b9-7f3876102a3e tempest-VolumesExtendTest-93540250 None] detach volume v_4b0ebe2e-141a-40f1-bcb5-83708d103f54 from target iqn.2010-10.org.openstack:4b0ebe2e-141a-40f1-bcb5-83708d103f54 {{(pid=84696) detach_target_vol /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:606}} Apr 17 15:24:47 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-8502c4c2-1b69-4858-a3b9-7f3876102a3e tempest-VolumesExtendTest-93540250 None] Sending pool request DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:4b0ebe2e-141a-40f1-bcb5-83708d103f54/luns/v_4b0ebe2e-141a-40f1-bcb5-83708d103f54 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:24:47 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-8502c4c2-1b69-4858-a3b9-7f3876102a3e tempest-VolumesExtendTest-93540250 None] Sending DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:4b0ebe2e-141a-40f1-bcb5-83708d103f54/luns/v_4b0ebe2e-141a-40f1-bcb5-83708d103f54 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:24:47 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-8502c4c2-1b69-4858-a3b9-7f3876102a3e tempest-VolumesExtendTest-93540250 None] Geting {'code': 404, 'error': {'class': 'opene.exceptions.ItemNotFoundError', 'message': 'Target iqn.2010-10.org.openstack:4b0ebe2e-141a-40f1-bcb5-83708d103f54 not exists.', 'url': 'http://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:4b0ebe2e-141a-40f1-bcb5-83708d103f54/luns/v_4b0ebe2e-141a-40f1-bcb5-83708d103f54'}, 'data': None} from DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:4b0ebe2e-141a-40f1-bcb5-83708d103f54/luns/v_4b0ebe2e-141a-40f1-bcb5-83708d103f54 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:24:47 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-8502c4c2-1b69-4858-a3b9-7f3876102a3e tempest-VolumesExtendTest-93540250 None] failed to remove resource iqn.2010-10.org.openstack:4b0ebe2e-141a-40f1-bcb5-83708d103f54 because of JDSS resource v_4b0ebe2e-141a-40f1-bcb5-83708d103f54 DNE. {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:920}} Apr 17 15:24:47 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-8502c4c2-1b69-4858-a3b9-7f3876102a3e tempest-VolumesExtendTest-93540250 None] delete target: iqn.2010-10.org.openstack:4b0ebe2e-141a-40f1-bcb5-83708d103f54 {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:930}} Apr 17 15:24:47 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-8502c4c2-1b69-4858-a3b9-7f3876102a3e tempest-VolumesExtendTest-93540250 None] delete target iqn.2010-10.org.openstack:4b0ebe2e-141a-40f1-bcb5-83708d103f54 {{(pid=84696) delete_target /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:442}} Apr 17 15:24:47 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-8502c4c2-1b69-4858-a3b9-7f3876102a3e tempest-VolumesExtendTest-93540250 None] Sending pool request DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:4b0ebe2e-141a-40f1-bcb5-83708d103f54 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:24:47 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-8502c4c2-1b69-4858-a3b9-7f3876102a3e tempest-VolumesExtendTest-93540250 None] Sending DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:4b0ebe2e-141a-40f1-bcb5-83708d103f54 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:24:47 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-8502c4c2-1b69-4858-a3b9-7f3876102a3e tempest-VolumesExtendTest-93540250 None] Geting {'code': 404, 'error': {'class': 'opene.exceptions.ItemNotFoundError', 'message': 'Target iqn.2010-10.org.openstack:4b0ebe2e-141a-40f1-bcb5-83708d103f54 not exists.', 'url': 'http://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:4b0ebe2e-141a-40f1-bcb5-83708d103f54'}, 'data': None} from DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:4b0ebe2e-141a-40f1-bcb5-83708d103f54 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:24:47 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-8502c4c2-1b69-4858-a3b9-7f3876102a3e tempest-VolumesExtendTest-93540250 None] failed to remove resource iqn.2010-10.org.openstack:4b0ebe2e-141a-40f1-bcb5-83708d103f54 because of JDSS resource iqn.2010-10.org.openstack:4b0ebe2e-141a-40f1-bcb5-83708d103f54 DNE. {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:935}} Apr 17 15:24:47 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-8502c4c2-1b69-4858-a3b9-7f3876102a3e tempest-VolumesExtendTest-93540250 None] deleating volume v_4b0ebe2e-141a-40f1-bcb5-83708d103f54 {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 15:24:47 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-8502c4c2-1b69-4858-a3b9-7f3876102a3e tempest-VolumesExtendTest-93540250 None] get snapshots for volume v_4b0ebe2e-141a-40f1-bcb5-83708d103f54 {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 15:24:47 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-8502c4c2-1b69-4858-a3b9-7f3876102a3e tempest-VolumesExtendTest-93540250 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_4b0ebe2e-141a-40f1-bcb5-83708d103f54/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:24:47 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-8502c4c2-1b69-4858-a3b9-7f3876102a3e tempest-VolumesExtendTest-93540250 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_4b0ebe2e-141a-40f1-bcb5-83708d103f54/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:24:47 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-804a4aab-b03c-4885-b758-db492f88f09b tempest-GroupSnapshotsTest-1926061586 None] Geting {'code': 200, 'error': None, 'data': {'results': 0, 'entries': []}} from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/s_c9995e45-166a-4b06-8340-f012f00d640c/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:24:47 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-804a4aab-b03c-4885-b758-db492f88f09b tempest-GroupSnapshotsTest-1926061586 None] get volume s_c9995e45-166a-4b06-8340-f012f00d640c info {{(pid=84696) get_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:239}} Apr 17 15:24:47 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-804a4aab-b03c-4885-b758-db492f88f09b tempest-GroupSnapshotsTest-1926061586 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/s_c9995e45-166a-4b06-8340-f012f00d640c {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:24:47 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-804a4aab-b03c-4885-b758-db492f88f09b tempest-GroupSnapshotsTest-1926061586 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/s_c9995e45-166a-4b06-8340-f012f00d640c {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:24:47 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-804a4aab-b03c-4885-b758-db492f88f09b tempest-GroupSnapshotsTest-1926061586 None] Geting {'code': 200, 'error': None, 'data': {'origin': 'Pool-0/v_7a71a6fd-8167-40ab-b057-5f82e0f55947@s_c9995e45-166a-4b06-8340-f012f00d640c', 'relatime': None, 'acltype': None, 'vscan': None, 'full_name': 'Pool-0/s_c9995e45-166a-4b06-8340-f012f00d640c', 'userrefs': None, 'primarycache': 'all', 'logbias': 'latency', 'creation': '1681746477', 'sync': 'always', 'is_clone': True, 'dedup': 'off', 'sharenfs': None, 'receive_resume_token': None, 'volsize': '1073741824', 'referenced': '57344', 'sharesmb': None, 'createtxg': '187964', 'reservation': '0', 'scontext': None, 'mountpoint': None, 'casesensitivity': None, 'guid': '11926057807763245373', 'usedbyrefreservation': '0', 'dnodesize': None, 'written': '0', 'logicalused': '0', 'compressratio': '1.00', 'rootcontext': 'none', 'default_scsi_id': '1d4493637e225b11', 'type': 'volume', 'compression': 'lz4', 'snapdir': None, 'overlay': None, 'encryption': 'off', 'xattr': None, 'volmode': 'default', 'copies': '1', 'snapshot_limit': '18446744073709551615', 'aclinherit': None, 'defcontext': 'none', 'readonly': 'on', 'version': None, 'recordsize': None, 'filesystem_limit': None, 'mounted': None, 'mlslabel': 'none', 'secondarycache': 'all', 'refreservation': '0', 'available': '113909649408', 'used': '0', 'encryptionroot': None, 'exec': None, 'refquota': None, 'refcompressratio': '1.00', 'quota': None, 'utf8only': None, 'keylocation': 'none', 'snapdev': 'hidden', 'snapshot_count': '18446744073709551615', 'fscontext': 'none', 'clones': None, 'canmount': None, 'keystatus': None, 'atime': None, 'usedbysnapshots': '0', 'normalization': None, 'usedbychildren': '0', 'volblocksize': '131072', 'usedbydataset': '0', 'objsetid': '122330', 'name': 's_c9995e45-166a-4b06-8340-f012f00d640c', 'defer_destroy': None, 'pbkdf2iters': '0', 'checksum': 'on', 'redundant_metadata': 'all', 'filesystem_count': None, 'devices': None, 'keyformat': 'none', 'setuid': None, 'logicalreferenced': '28672', 'context': 'none', 'zoned': None, 'nbmand': None}} from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/s_c9995e45-166a-4b06-8340-f012f00d640c {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:24:47 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-804a4aab-b03c-4885-b758-db492f88f09b tempest-GroupSnapshotsTest-1926061586 None] delete snapshot s_c9995e45-166a-4b06-8340-f012f00d640c of volume v_7a71a6fd-8167-40ab-b057-5f82e0f55947 {{(pid=84696) delete_snapshot /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:751}} Apr 17 15:24:47 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-804a4aab-b03c-4885-b758-db492f88f09b tempest-GroupSnapshotsTest-1926061586 None] Sending pool request DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_7a71a6fd-8167-40ab-b057-5f82e0f55947/snapshots/s_c9995e45-166a-4b06-8340-f012f00d640c {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:24:47 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-804a4aab-b03c-4885-b758-db492f88f09b tempest-GroupSnapshotsTest-1926061586 None] Sending DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_7a71a6fd-8167-40ab-b057-5f82e0f55947/snapshots/s_c9995e45-166a-4b06-8340-f012f00d640c {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:24:47 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-8502c4c2-1b69-4858-a3b9-7f3876102a3e tempest-VolumesExtendTest-93540250 None] Geting {'code': 200, 'error': None, 'data': {'results': 0, 'entries': []}} from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_4b0ebe2e-141a-40f1-bcb5-83708d103f54/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:24:47 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-8502c4c2-1b69-4858-a3b9-7f3876102a3e tempest-VolumesExtendTest-93540250 None] get volume v_4b0ebe2e-141a-40f1-bcb5-83708d103f54 info {{(pid=84696) get_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:239}} Apr 17 15:24:47 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-8502c4c2-1b69-4858-a3b9-7f3876102a3e tempest-VolumesExtendTest-93540250 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_4b0ebe2e-141a-40f1-bcb5-83708d103f54 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:24:47 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-8502c4c2-1b69-4858-a3b9-7f3876102a3e tempest-VolumesExtendTest-93540250 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_4b0ebe2e-141a-40f1-bcb5-83708d103f54 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:24:48 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-8502c4c2-1b69-4858-a3b9-7f3876102a3e tempest-VolumesExtendTest-93540250 None] Geting {'code': 200, 'error': None, 'data': {'origin': 'Pool-0/v_006f20ae-5306-44fe-afdf-e433f79284e8@v_4b0ebe2e-141a-40f1-bcb5-83708d103f54', 'relatime': None, 'acltype': None, 'vscan': None, 'full_name': 'Pool-0/v_4b0ebe2e-141a-40f1-bcb5-83708d103f54', 'userrefs': None, 'primarycache': 'all', 'logbias': 'latency', 'creation': '1681746477', 'sync': 'always', 'is_clone': True, 'dedup': 'off', 'sharenfs': None, 'receive_resume_token': None, 'volsize': '2147483648', 'referenced': '16285696', 'sharesmb': None, 'createtxg': '187958', 'reservation': '0', 'scontext': None, 'mountpoint': None, 'casesensitivity': None, 'guid': '13728919998564651037', 'usedbyrefreservation': '0', 'dnodesize': None, 'written': '57344', 'logicalused': '28672', 'compressratio': '1.00', 'rootcontext': 'none', 'default_scsi_id': 'c4eb5e6968b5de3c', 'type': 'volume', 'compression': 'lz4', 'snapdir': None, 'overlay': None, 'encryption': 'off', 'xattr': None, 'volmode': 'default', 'copies': '1', 'snapshot_limit': '18446744073709551615', 'aclinherit': None, 'defcontext': 'none', 'readonly': 'off', 'version': None, 'recordsize': None, 'filesystem_limit': None, 'mounted': None, 'mlslabel': 'none', 'secondarycache': 'all', 'refreservation': '0', 'available': '113909637120', 'san:volume_id': 'c4eb5e6968b5de3c34b5264c9a3bdfb7d3564d27', 'encryptionroot': None, 'exec': None, 'refquota': None, 'refcompressratio': '1.17', 'quota': None, 'utf8only': None, 'keylocation': 'none', 'snapdev': 'hidden', 'snapshot_count': '18446744073709551615', 'fscontext': 'none', 'clones': None, 'canmount': None, 'keystatus': None, 'atime': None, 'usedbysnapshots': '0', 'normalization': None, 'usedbychildren': '0', 'volblocksize': '131072', 'usedbydataset': '57344', 'objsetid': '122538', 'name': 'v_4b0ebe2e-141a-40f1-bcb5-83708d103f54', 'defer_destroy': None, 'pbkdf2iters': '0', 'checksum': 'on', 'redundant_metadata': 'all', 'filesystem_count': None, 'devices': None, 'keyformat': 'none', 'setuid': None, 'used': '57344', 'logicalreferenced': '19042304', 'context': 'none', 'zoned': None, 'nbmand': None}} from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_4b0ebe2e-141a-40f1-bcb5-83708d103f54 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:24:48 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-8502c4c2-1b69-4858-a3b9-7f3876102a3e tempest-VolumesExtendTest-93540250 None] delete snapshot v_4b0ebe2e-141a-40f1-bcb5-83708d103f54 of volume v_006f20ae-5306-44fe-afdf-e433f79284e8 {{(pid=84696) delete_snapshot /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:751}} Apr 17 15:24:48 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-8502c4c2-1b69-4858-a3b9-7f3876102a3e tempest-VolumesExtendTest-93540250 None] Sending pool request DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots/v_4b0ebe2e-141a-40f1-bcb5-83708d103f54 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:24:48 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-8502c4c2-1b69-4858-a3b9-7f3876102a3e tempest-VolumesExtendTest-93540250 None] Sending DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots/v_4b0ebe2e-141a-40f1-bcb5-83708d103f54 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:24:49 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-50e02017-ce73-4ed4-9f96-0a01c81adea4 tempest-VolumesGetTest-1961901732 None] Geting {'code': 204} from DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_9a8586a6-f2f5-4bee-a56f-a06b01120c39 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:24:49 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-50e02017-ce73-4ed4-9f96-0a01c81adea4 tempest-VolumesGetTest-1961901732 None] volume v_9a8586a6-f2f5-4bee-a56f-a06b01120c39 deleted {{(pid=84696) delete_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:346}} Apr 17 15:24:49 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:INSERT=6,cinder:SELECT=68,cinder:UPDATE=24,cinder:DELETE=2 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:24:49 user cinder-volume[84696]: DEBUG cinder.quota [None req-50e02017-ce73-4ed4-9f96-0a01c81adea4 tempest-VolumesGetTest-1961901732 None] Created reservations ['7c3ecffb-d402-4873-b4ae-40ee1f20b299', '58bf8ae5-2e54-4086-9b73-97221f6bb8dc', '83d8fa80-efcb-4a29-8451-4abeb5a2d8db', '55e0e8fd-e85c-4075-9cf4-9a8c889b44c7'] {{(pid=84696) reserve /opt/stack/cinder/cinder/quota.py:755}} Apr 17 15:24:49 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-50e02017-ce73-4ed4-9f96-0a01c81adea4 tempest-VolumesGetTest-1961901732 None] Updating volume stats {{(pid=84696) _update_volume_stats /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:724}} Apr 17 15:24:49 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-50e02017-ce73-4ed4-9f96-0a01c81adea4 tempest-VolumesGetTest-1961901732 None] Get pool Pool-0 fsprops {{(pid=84696) get_pool_stats /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:894}} Apr 17 15:24:49 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-50e02017-ce73-4ed4-9f96-0a01c81adea4 tempest-VolumesGetTest-1961901732 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:24:49 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-50e02017-ce73-4ed4-9f96-0a01c81adea4 tempest-VolumesGetTest-1961901732 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:24:50 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-804a4aab-b03c-4885-b758-db492f88f09b tempest-GroupSnapshotsTest-1926061586 None] Geting {'code': 204} from DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_7a71a6fd-8167-40ab-b057-5f82e0f55947/snapshots/s_c9995e45-166a-4b06-8340-f012f00d640c {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:24:50 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-804a4aab-b03c-4885-b758-db492f88f09b tempest-GroupSnapshotsTest-1926061586 None] snapshot s_c9995e45-166a-4b06-8340-f012f00d640c deleted {{(pid=84696) delete_snapshot /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:772}} Apr 17 15:24:50 user cinder-volume[84696]: WARNING cinder.quota [None req-804a4aab-b03c-4885-b758-db492f88f09b tempest-GroupSnapshotsTest-1926061586 None] Deprecated: Default quota for resource: gigabytes_tempest-GroupSnapshotsTest-volume-type-1057865787 is set by the default quota flag: quota_gigabytes_tempest-GroupSnapshotsTest-volume-type-1057865787, it is now deprecated. Please use the default quota class for default quota. Apr 17 15:24:50 user cinder-volume[84696]: WARNING cinder.quota [None req-804a4aab-b03c-4885-b758-db492f88f09b tempest-GroupSnapshotsTest-1926061586 None] Deprecated: Default quota for resource: snapshots_tempest-GroupSnapshotsTest-volume-type-1057865787 is set by the default quota flag: quota_snapshots_tempest-GroupSnapshotsTest-volume-type-1057865787, it is now deprecated. Please use the default quota class for default quota. Apr 17 15:24:50 user cinder-volume[84696]: DEBUG cinder.quota [None req-804a4aab-b03c-4885-b758-db492f88f09b tempest-GroupSnapshotsTest-1926061586 None] Created reservations ['7b59149b-a1cc-454a-9222-5fee9630cb97', 'fce30c4d-fcb0-4262-b215-e1978e3c66ea', 'ed766a92-2026-475a-8965-b540a9aeb554', '437eec2c-4417-473e-a097-b9ac23baf3e6'] {{(pid=84696) reserve /opt/stack/cinder/cinder/quota.py:755}} Apr 17 15:24:50 user cinder-volume[84696]: INFO cinder.volume.manager [None req-804a4aab-b03c-4885-b758-db492f88f09b tempest-GroupSnapshotsTest-1926061586 None] group_snapshot f6413371-b635-4f3c-940c-0c0795e59bef: deleted successfully Apr 17 15:24:50 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-50e02017-ce73-4ed4-9f96-0a01c81adea4 tempest-VolumesGetTest-1961901732 None] Geting {'code': 200, 'error': None, 'data': {'available': '113909755904', 'status': 30, 'import_status': {'import_steps': {'san_setup': True, 'vip_setup': True, 'zfs_import': True, 'mount_system_volume': True, 'nas_setup': True}, 'import_successful': True}, 'scan': None, 'encryption': {'enabled': False}, 'deduplication_rate': '1.00', 'sysvol_upgrade': 'not_needed', 'vdevs': [{'name': 'wwn-0x6000c296b793bf526a107a9d823e947a', 'iostats': {'read': '0', 'write': '0', 'chksum': '0'}, 'disks': [{'origin': 'local', 'slot': '', 'led': 'off', 'name': 'sdn', 'iostats': {'read': '0', 'write': '0', 'chksum': '0'}, 'alias': '', 'health': 'ONLINE', 'sn': '6000c296b793bf526a107a9d823e947a', 'trim_data': {'status': 'trim_unsupported', 'progress': None, 'trim_time': None}, 'path': 'pci-0000:03:00.0-scsi-0:0:14:0', 'model': 'Virtual disk', 'id': 'wwn-0x6000c296b793bf526a107a9d823e947a', 'size': 128849018880}], 'health': 'ONLINE', 'vdev_replacings': [], 'vdev_spares': [], 'type': ''}], 'id': '14181317334440137330', 'health': 'ONLINE', 'iostats': {'read': '0', 'write': '0', 'chksum': '0'}, 'operation': 'none', 'size': '127775277056', 'autotrim': False, 'name': 'Pool-0'}} from GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:24:50 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-50e02017-ce73-4ed4-9f96-0a01c81adea4 tempest-VolumesGetTest-1961901732 None] Total capacity: 119, Free 106. {{(pid=84696) _update_volume_stats /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:757}} Apr 17 15:24:50 user cinder-volume[84696]: DEBUG cinder.manager [None req-50e02017-ce73-4ed4-9f96-0a01c81adea4 tempest-VolumesGetTest-1961901732 None] Notifying Schedulers of capabilities ... {{(pid=84696) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 17 15:24:50 user cinder-volume[84696]: INFO cinder.volume.manager [None req-50e02017-ce73-4ed4-9f96-0a01c81adea4 tempest-VolumesGetTest-1961901732 None] Deleted volume successfully. Apr 17 15:24:50 user cinder-volume[84696]: DEBUG cinder.coordination [None req-50e02017-ce73-4ed4-9f96-0a01c81adea4 tempest-VolumesGetTest-1961901732 None] Lock "cinder-9a8586a6-f2f5-4bee-a56f-a06b01120c39-delete_volume" released by "delete_volume" :: held 5.102s {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:207}} Apr 17 15:24:50 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-7303fa52-959d-4a80-9bb5-5cd76868f509 tempest-VolumesGetTest-1961901732 None] Volume reschedule parameters: True retry: {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 15:24:50 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-7303fa52-959d-4a80-9bb5-5cd76868f509 tempest-VolumesGetTest-1961901732 None] Flow 'volume_create_manager' (d96d305d-b889-4e00-adc7-7e12c885857e) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:24:50 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-7303fa52-959d-4a80-9bb5-5cd76868f509 tempest-VolumesGetTest-1961901732 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (90ad4870-a3ee-4a04-9022-19249b170227) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:24:50 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-7303fa52-959d-4a80-9bb5-5cd76868f509 tempest-VolumesGetTest-1961901732 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (90ad4870-a3ee-4a04-9022-19249b170227) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:24:50Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesGetTest-Volume-193773742',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=3982de76-ea42-41a0-b8e5-4b24b489009b,launched_at=None,metadata={Type='Test'},migration_status=None,multiattach=False,previous_status=None,project_id='a65d14b0a2894a6f90506d86feaefd15',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:24:51Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:24:51Z,use_quota=True,user_id='6ffa3051570a4721a48a0171b3cc4d16',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:24:50 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-7303fa52-959d-4a80-9bb5-5cd76868f509 tempest-VolumesGetTest-1961901732 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (ccce0f2c-b342-4f1d-b734-73d627b7cff1) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:24:50 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-7303fa52-959d-4a80-9bb5-5cd76868f509 tempest-VolumesGetTest-1961901732 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (ccce0f2c-b342-4f1d-b734-73d627b7cff1) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:24:50 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-7303fa52-959d-4a80-9bb5-5cd76868f509 tempest-VolumesGetTest-1961901732 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (9516e57c-d1cc-4851-8f4d-d20fc57743ca) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:24:50 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-8502c4c2-1b69-4858-a3b9-7f3876102a3e tempest-VolumesExtendTest-93540250 None] Geting {'code': 204} from DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots/v_4b0ebe2e-141a-40f1-bcb5-83708d103f54 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:24:50 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-8502c4c2-1b69-4858-a3b9-7f3876102a3e tempest-VolumesExtendTest-93540250 None] snapshot v_4b0ebe2e-141a-40f1-bcb5-83708d103f54 deleted {{(pid=84696) delete_snapshot /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:772}} Apr 17 15:24:50 user cinder-volume[84696]: DEBUG cinder.quota [None req-8502c4c2-1b69-4858-a3b9-7f3876102a3e tempest-VolumesExtendTest-93540250 None] Created reservations ['8930dc96-2163-48ac-8c5f-9d29648eb1c5', '2d39d458-ed7c-4848-95aa-7c8d347dc402', '58b23096-7ed5-45a4-b842-fcca1e6096bf', '80f6962e-fbd6-4a8a-895e-4bbeccaaada8'] {{(pid=84696) reserve /opt/stack/cinder/cinder/quota.py:755}} Apr 17 15:24:50 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-7303fa52-959d-4a80-9bb5-5cd76868f509 tempest-VolumesGetTest-1961901732 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (9516e57c-d1cc-4851-8f4d-d20fc57743ca) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'image', 'volume_id': '3982de76-ea42-41a0-b8e5-4b24b489009b', 'volume_name': 'volume-3982de76-ea42-41a0-b8e5-4b24b489009b', 'volume_size': 1, 'image_id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'image_location': (None, None), 'image_meta': {'name': 'cirros-0.5.2-x86_64-disk', 'disk_format': 'qcow2', 'container_format': 'bare', 'visibility': 'public', 'size': 16300544, 'virtual_size': 117440512, 'status': 'active', 'checksum': 'b874c39491a2377b8490f5f1e89761a4', 'protected': False, 'min_ram': 0, 'min_disk': 0, 'owner': 'a43c211b4ed542538799349489b1d884', 'os_hidden': False, 'os_hash_algo': 'sha512', 'os_hash_value': '6b813aa46bb90b4da216a4d19376593fa3f4fc7e617f03a92b7fe11e9a3981cbe8f0959dbebe36225e5f53dc4492341a4863cac4ed1ee0909f3fc78ef9c3e869', 'id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'created_at': datetime.datetime(2023, 4, 17, 15, 19, 53, tzinfo=datetime.timezone.utc), 'updated_at': datetime.datetime(2023, 4, 17, 15, 19, 55, tzinfo=datetime.timezone.utc), 'tags': [], 'file': '/v2/images/b3f0dec1-7758-44a8-82e0-76ada8837428/file', 'properties': {'hw_rng_model': 'virtio', 'owner_specified.openstack.md5': '', 'owner_specified.openstack.object': 'images/cirros-0.5.2-x86_64-disk', 'owner_specified.openstack.sha256': ''}}, 'image_service': }' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:24:50 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-7303fa52-959d-4a80-9bb5-5cd76868f509 tempest-VolumesGetTest-1961901732 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (1d6d1abc-e65f-4cd2-a90f-0c276857c2b1) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:24:51 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-7303fa52-959d-4a80-9bb5-5cd76868f509 tempest-VolumesGetTest-1961901732 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (1d6d1abc-e65f-4cd2-a90f-0c276857c2b1) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:24:51 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-7303fa52-959d-4a80-9bb5-5cd76868f509 tempest-VolumesGetTest-1961901732 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (15712217-4f5d-4dcc-b735-df2ed570f9ab) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:24:51 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-7303fa52-959d-4a80-9bb5-5cd76868f509 tempest-VolumesGetTest-1961901732 None] Volume 3982de76-ea42-41a0-b8e5-4b24b489009b: being created as image with specification: {'status': 'creating', 'volume_name': 'volume-3982de76-ea42-41a0-b8e5-4b24b489009b', 'volume_size': 1, 'image_id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'image_location': (None, None), 'image_meta': {'name': 'cirros-0.5.2-x86_64-disk', 'disk_format': 'qcow2', 'container_format': 'bare', 'visibility': 'public', 'size': 16300544, 'virtual_size': 117440512, 'status': 'active', 'checksum': 'b874c39491a2377b8490f5f1e89761a4', 'protected': False, 'min_ram': 0, 'min_disk': 0, 'owner': 'a43c211b4ed542538799349489b1d884', 'os_hidden': False, 'os_hash_algo': 'sha512', 'os_hash_value': '6b813aa46bb90b4da216a4d19376593fa3f4fc7e617f03a92b7fe11e9a3981cbe8f0959dbebe36225e5f53dc4492341a4863cac4ed1ee0909f3fc78ef9c3e869', 'id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'created_at': datetime.datetime(2023, 4, 17, 15, 19, 53, tzinfo=datetime.timezone.utc), 'updated_at': datetime.datetime(2023, 4, 17, 15, 19, 55, tzinfo=datetime.timezone.utc), 'tags': [], 'file': '/v2/images/b3f0dec1-7758-44a8-82e0-76ada8837428/file', 'properties': {'hw_rng_model': 'virtio', 'owner_specified.openstack.md5': '', 'owner_specified.openstack.object': 'images/cirros-0.5.2-x86_64-disk', 'owner_specified.openstack.sha256': ''}}, 'image_service': } Apr 17 15:24:51 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-7303fa52-959d-4a80-9bb5-5cd76868f509 tempest-VolumesGetTest-1961901732 None] Cloning 3982de76-ea42-41a0-b8e5-4b24b489009b from image b3f0dec1-7758-44a8-82e0-76ada8837428 at location (None, None). {{(pid=84696) _create_from_image /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1071}} Apr 17 15:24:51 user cinder-volume[84696]: DEBUG cinder.coordination [None req-7303fa52-959d-4a80-9bb5-5cd76868f509 tempest-VolumesGetTest-1961901732 None] Acquiring lock "cinder-b3f0dec1-7758-44a8-82e0-76ada8837428" by "_prepare_image_cache_entry" {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:192}} Apr 17 15:24:51 user cinder-volume[84696]: DEBUG cinder.coordination [None req-7303fa52-959d-4a80-9bb5-5cd76868f509 tempest-VolumesGetTest-1961901732 None] Lock "cinder-b3f0dec1-7758-44a8-82e0-76ada8837428" acquired by "_prepare_image_cache_entry" :: waited 0.025s {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:196}} Apr 17 15:24:51 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-8502c4c2-1b69-4858-a3b9-7f3876102a3e tempest-VolumesExtendTest-93540250 None] Updating volume stats {{(pid=84696) _update_volume_stats /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:724}} Apr 17 15:24:51 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-8502c4c2-1b69-4858-a3b9-7f3876102a3e tempest-VolumesExtendTest-93540250 None] Get pool Pool-0 fsprops {{(pid=84696) get_pool_stats /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:894}} Apr 17 15:24:51 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-8502c4c2-1b69-4858-a3b9-7f3876102a3e tempest-VolumesExtendTest-93540250 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:24:51 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-8502c4c2-1b69-4858-a3b9-7f3876102a3e tempest-VolumesExtendTest-93540250 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:24:51 user cinder-volume[84696]: DEBUG cinder.image.cache [None req-7303fa52-959d-4a80-9bb5-5cd76868f509 tempest-VolumesGetTest-1961901732 None] Found image-volume cache entry: {'id': 1, 'image_id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'volume_id': '006f20ae-5306-44fe-afdf-e433f79284e8', 'host': 'user@joviandss-2#joviandss-2', 'size': 1, 'image_updated_at': datetime.datetime(2023, 4, 17, 15, 19, 55), 'last_used': datetime.datetime(2023, 4, 17, 15, 24, 51, 55535)}. {{(pid=84696) get_entry /opt/stack/cinder/cinder/image/cache.py:76}} Apr 17 15:24:51 user cinder-volume[84696]: DEBUG cinder.image.cache [None req-7303fa52-959d-4a80-9bb5-5cd76868f509 tempest-VolumesGetTest-1961901732 None] Image-volume cache entry image_update_at = 2023-04-17 15:19:55+00:00, requested image updated_at = 2023-04-17 15:19:55+00:00. {{(pid=84696) _should_update_entry /opt/stack/cinder/cinder/image/cache.py:259}} Apr 17 15:24:51 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-7303fa52-959d-4a80-9bb5-5cd76868f509 tempest-VolumesGetTest-1961901732 None] Found cache entry for image = b3f0dec1-7758-44a8-82e0-76ada8837428 on host user@joviandss-2#joviandss-2. {{(pid=84696) _prepare_image_cache_entry /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:886}} Apr 17 15:24:51 user cinder-volume[84696]: DEBUG cinder.coordination [None req-7303fa52-959d-4a80-9bb5-5cd76868f509 tempest-VolumesGetTest-1961901732 None] Lock "cinder-b3f0dec1-7758-44a8-82e0-76ada8837428" released by "_prepare_image_cache_entry" :: held 0.032s {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:207}} Apr 17 15:24:51 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-7303fa52-959d-4a80-9bb5-5cd76868f509 tempest-VolumesGetTest-1961901732 None] Attempting to retrieve cache entry for image = b3f0dec1-7758-44a8-82e0-76ada8837428 on host user@joviandss-2#joviandss-2. {{(pid=84696) _create_from_image_cache /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:827}} Apr 17 15:24:51 user cinder-volume[84696]: DEBUG cinder.image.cache [None req-7303fa52-959d-4a80-9bb5-5cd76868f509 tempest-VolumesGetTest-1961901732 None] Found image-volume cache entry: {'id': 1, 'image_id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'volume_id': '006f20ae-5306-44fe-afdf-e433f79284e8', 'host': 'user@joviandss-2#joviandss-2', 'size': 1, 'image_updated_at': datetime.datetime(2023, 4, 17, 15, 19, 55), 'last_used': datetime.datetime(2023, 4, 17, 15, 24, 51, 81970)}. {{(pid=84696) get_entry /opt/stack/cinder/cinder/image/cache.py:76}} Apr 17 15:24:51 user cinder-volume[84696]: DEBUG cinder.image.cache [None req-7303fa52-959d-4a80-9bb5-5cd76868f509 tempest-VolumesGetTest-1961901732 None] Image-volume cache entry image_update_at = 2023-04-17 15:19:55+00:00, requested image updated_at = 2023-04-17 15:19:55+00:00. {{(pid=84696) _should_update_entry /opt/stack/cinder/cinder/image/cache.py:259}} Apr 17 15:24:51 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-7303fa52-959d-4a80-9bb5-5cd76868f509 tempest-VolumesGetTest-1961901732 None] Creating from source image-volume 006f20ae-5306-44fe-afdf-e433f79284e8 {{(pid=84696) _create_from_image_cache /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:841}} Apr 17 15:24:51 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-7303fa52-959d-4a80-9bb5-5cd76868f509 tempest-VolumesGetTest-1961901732 None] cloned volume 006f20ae-5306-44fe-afdf-e433f79284e8 to 3982de76-ea42-41a0-b8e5-4b24b489009b {{(pid=84696) create_cloned_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:521}} Apr 17 15:24:51 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-7303fa52-959d-4a80-9bb5-5cd76868f509 tempest-VolumesGetTest-1961901732 None] cloning v_006f20ae-5306-44fe-afdf-e433f79284e8 to v_3982de76-ea42-41a0-b8e5-4b24b489009b {{(pid=84696) _clone_object /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:456}} Apr 17 15:24:51 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-7303fa52-959d-4a80-9bb5-5cd76868f509 tempest-VolumesGetTest-1961901732 None] create snapshot v_3982de76-ea42-41a0-b8e5-4b24b489009b {{(pid=84696) create_snapshot /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:635}} Apr 17 15:24:51 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-7303fa52-959d-4a80-9bb5-5cd76868f509 tempest-VolumesGetTest-1961901732 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:24:51 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-7303fa52-959d-4a80-9bb5-5cd76868f509 tempest-VolumesGetTest-1961901732 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:24:51 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:UPDATE=34,cinder:SELECT=54,cinder:DELETE=3,cinder:INSERT=9 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:24:51 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-b126ed32-230c-4823-8d14-03ab093a03a3 tempest-GroupSnapshotsTest-1926061586 None] remove_export for volume: 7a71a6fd-8167-40ab-b057-5f82e0f55947. {{(pid=84696) remove_export /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:718}} Apr 17 15:24:51 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-b126ed32-230c-4823-8d14-03ab093a03a3 tempest-GroupSnapshotsTest-1926061586 None] remove export {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:912}} Apr 17 15:24:51 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-b126ed32-230c-4823-8d14-03ab093a03a3 tempest-GroupSnapshotsTest-1926061586 None] detach volume:Volume(_name_id=None,admin_metadata={},attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:24:39Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-volume-1159247536',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=83f168f7-009f-4ecb-8cda-bee4f351f951,host='user@joviandss-2#joviandss-2',id=7a71a6fd-8167-40ab-b057-5f82e0f55947,launched_at=2023-04-17T15:24:41Z,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='6cae12f3c2614d6fb32f21db60fec8d3',provider_auth='CHAP FxwsChIb GFcmgjpVJMhLaB',provider_geometry=None,provider_id=None,provider_location='172.16.0.220:3260,1 iqn.2010-10.org.openstack:7a71a6fd-8167-40ab-b057-5f82e0f55947 0',replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:24:39Z,service_uuid='e628352e-c4cf-40eb-8389-cce40663026a',shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='deleting',terminated_at=None,updated_at=2023-04-17T15:24:51Z,use_quota=True,user_id='560c850c324949d7a214e7a59dd65f8d',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(2fbbd4b4-c823-4693-8ac9-0db5fcafba0b),volume_type_id=2fbbd4b4-c823-4693-8ac9-0db5fcafba0b) from target:iqn.2010-10.org.openstack:7a71a6fd-8167-40ab-b057-5f82e0f55947. {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:913}} Apr 17 15:24:51 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-b126ed32-230c-4823-8d14-03ab093a03a3 tempest-GroupSnapshotsTest-1926061586 None] detach volume v_7a71a6fd-8167-40ab-b057-5f82e0f55947 from target iqn.2010-10.org.openstack:7a71a6fd-8167-40ab-b057-5f82e0f55947 {{(pid=84696) detach_target_vol /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:606}} Apr 17 15:24:51 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-b126ed32-230c-4823-8d14-03ab093a03a3 tempest-GroupSnapshotsTest-1926061586 None] Sending pool request DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:7a71a6fd-8167-40ab-b057-5f82e0f55947/luns/v_7a71a6fd-8167-40ab-b057-5f82e0f55947 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:24:51 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-b126ed32-230c-4823-8d14-03ab093a03a3 tempest-GroupSnapshotsTest-1926061586 None] Sending DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:7a71a6fd-8167-40ab-b057-5f82e0f55947/luns/v_7a71a6fd-8167-40ab-b057-5f82e0f55947 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:24:51 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-b126ed32-230c-4823-8d14-03ab093a03a3 tempest-GroupSnapshotsTest-1926061586 None] Geting {'code': 404, 'error': {'class': 'opene.exceptions.ItemNotFoundError', 'message': 'Target iqn.2010-10.org.openstack:7a71a6fd-8167-40ab-b057-5f82e0f55947 not exists.', 'url': 'http://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:7a71a6fd-8167-40ab-b057-5f82e0f55947/luns/v_7a71a6fd-8167-40ab-b057-5f82e0f55947'}, 'data': None} from DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:7a71a6fd-8167-40ab-b057-5f82e0f55947/luns/v_7a71a6fd-8167-40ab-b057-5f82e0f55947 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:24:51 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-b126ed32-230c-4823-8d14-03ab093a03a3 tempest-GroupSnapshotsTest-1926061586 None] failed to remove resource iqn.2010-10.org.openstack:7a71a6fd-8167-40ab-b057-5f82e0f55947 because of JDSS resource v_7a71a6fd-8167-40ab-b057-5f82e0f55947 DNE. {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:920}} Apr 17 15:24:51 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-b126ed32-230c-4823-8d14-03ab093a03a3 tempest-GroupSnapshotsTest-1926061586 None] delete target: iqn.2010-10.org.openstack:7a71a6fd-8167-40ab-b057-5f82e0f55947 {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:930}} Apr 17 15:24:51 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-b126ed32-230c-4823-8d14-03ab093a03a3 tempest-GroupSnapshotsTest-1926061586 None] delete target iqn.2010-10.org.openstack:7a71a6fd-8167-40ab-b057-5f82e0f55947 {{(pid=84696) delete_target /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:442}} Apr 17 15:24:51 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-b126ed32-230c-4823-8d14-03ab093a03a3 tempest-GroupSnapshotsTest-1926061586 None] Sending pool request DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:7a71a6fd-8167-40ab-b057-5f82e0f55947 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:24:51 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-b126ed32-230c-4823-8d14-03ab093a03a3 tempest-GroupSnapshotsTest-1926061586 None] Sending DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:7a71a6fd-8167-40ab-b057-5f82e0f55947 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:24:51 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-b126ed32-230c-4823-8d14-03ab093a03a3 tempest-GroupSnapshotsTest-1926061586 None] Geting {'code': 404, 'error': {'class': 'opene.exceptions.ItemNotFoundError', 'message': 'Target iqn.2010-10.org.openstack:7a71a6fd-8167-40ab-b057-5f82e0f55947 not exists.', 'url': 'http://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:7a71a6fd-8167-40ab-b057-5f82e0f55947'}, 'data': None} from DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:7a71a6fd-8167-40ab-b057-5f82e0f55947 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:24:51 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-b126ed32-230c-4823-8d14-03ab093a03a3 tempest-GroupSnapshotsTest-1926061586 None] failed to remove resource iqn.2010-10.org.openstack:7a71a6fd-8167-40ab-b057-5f82e0f55947 because of JDSS resource iqn.2010-10.org.openstack:7a71a6fd-8167-40ab-b057-5f82e0f55947 DNE. {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:935}} Apr 17 15:24:51 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-b126ed32-230c-4823-8d14-03ab093a03a3 tempest-GroupSnapshotsTest-1926061586 None] deleating volume v_7a71a6fd-8167-40ab-b057-5f82e0f55947 {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 15:24:51 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-b126ed32-230c-4823-8d14-03ab093a03a3 tempest-GroupSnapshotsTest-1926061586 None] get snapshots for volume v_7a71a6fd-8167-40ab-b057-5f82e0f55947 {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 15:24:51 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-b126ed32-230c-4823-8d14-03ab093a03a3 tempest-GroupSnapshotsTest-1926061586 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_7a71a6fd-8167-40ab-b057-5f82e0f55947/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:24:51 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-b126ed32-230c-4823-8d14-03ab093a03a3 tempest-GroupSnapshotsTest-1926061586 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_7a71a6fd-8167-40ab-b057-5f82e0f55947/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:24:51 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-57c4caf5-947d-4d53-8d87-afb00c52f60a tempest-VolumesExtendTest-93540250 None] Volume reschedule parameters: True retry: {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 15:24:51 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-57c4caf5-947d-4d53-8d87-afb00c52f60a tempest-VolumesExtendTest-93540250 None] Flow 'volume_create_manager' (5c71d51e-2fd0-49bd-a294-77374596bb7f) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:24:51 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-57c4caf5-947d-4d53-8d87-afb00c52f60a tempest-VolumesExtendTest-93540250 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (936f5f5c-31c7-4ea6-a4bf-5ae5ab8529b9) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:24:51 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-57c4caf5-947d-4d53-8d87-afb00c52f60a tempest-VolumesExtendTest-93540250 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (936f5f5c-31c7-4ea6-a4bf-5ae5ab8529b9) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:24:52Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesExtendTest-Volume-2061662394',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=31d34366-a0ee-4a52-822b-79de201b5a12,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='ef25ef72b5a64e2c8a60d3d139367fd1',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:24:52Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:24:52Z,use_quota=True,user_id='d473e2b76b554904a4d625f35a78c038',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:24:51 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-57c4caf5-947d-4d53-8d87-afb00c52f60a tempest-VolumesExtendTest-93540250 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (acd1f8eb-cb25-43bb-8d4a-6802dba2f9d2) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:24:51 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-57c4caf5-947d-4d53-8d87-afb00c52f60a tempest-VolumesExtendTest-93540250 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (acd1f8eb-cb25-43bb-8d4a-6802dba2f9d2) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:24:51 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-57c4caf5-947d-4d53-8d87-afb00c52f60a tempest-VolumesExtendTest-93540250 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (b844bed6-d6fa-4ad6-a3e5-6ef70cb959f1) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:24:51 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-57c4caf5-947d-4d53-8d87-afb00c52f60a tempest-VolumesExtendTest-93540250 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (b844bed6-d6fa-4ad6-a3e5-6ef70cb959f1) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': '31d34366-a0ee-4a52-822b-79de201b5a12', 'volume_name': 'volume-31d34366-a0ee-4a52-822b-79de201b5a12', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:24:51 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-57c4caf5-947d-4d53-8d87-afb00c52f60a tempest-VolumesExtendTest-93540250 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (ddfa3889-2395-46a7-8e7b-7ea7ea658bf6) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:24:51 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-57c4caf5-947d-4d53-8d87-afb00c52f60a tempest-VolumesExtendTest-93540250 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (ddfa3889-2395-46a7-8e7b-7ea7ea658bf6) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:24:51 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-57c4caf5-947d-4d53-8d87-afb00c52f60a tempest-VolumesExtendTest-93540250 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (74f5ba97-afaf-45c7-8972-3062721d224b) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:24:51 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-57c4caf5-947d-4d53-8d87-afb00c52f60a tempest-VolumesExtendTest-93540250 None] Volume 31d34366-a0ee-4a52-822b-79de201b5a12: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-31d34366-a0ee-4a52-822b-79de201b5a12', 'volume_size': 1} Apr 17 15:24:51 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-57c4caf5-947d-4d53-8d87-afb00c52f60a tempest-VolumesExtendTest-93540250 None] creating volume v_31d34366-a0ee-4a52-822b-79de201b5a12. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 15:24:51 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-57c4caf5-947d-4d53-8d87-afb00c52f60a tempest-VolumesExtendTest-93540250 None] create volume {'name': 'v_31d34366-a0ee-4a52-822b-79de201b5a12', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 15:24:51 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-57c4caf5-947d-4d53-8d87-afb00c52f60a tempest-VolumesExtendTest-93540250 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:24:51 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-57c4caf5-947d-4d53-8d87-afb00c52f60a tempest-VolumesExtendTest-93540250 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:24:52 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-8502c4c2-1b69-4858-a3b9-7f3876102a3e tempest-VolumesExtendTest-93540250 None] Geting {'code': 200, 'error': None, 'data': {'available': '113909874688', 'status': 30, 'import_status': {'import_steps': {'san_setup': True, 'vip_setup': True, 'zfs_import': True, 'mount_system_volume': True, 'nas_setup': True}, 'import_successful': True}, 'scan': None, 'encryption': {'enabled': False}, 'deduplication_rate': '1.00', 'sysvol_upgrade': 'not_needed', 'vdevs': [{'name': 'wwn-0x6000c296b793bf526a107a9d823e947a', 'iostats': {'read': '0', 'write': '0', 'chksum': '0'}, 'disks': [{'origin': 'local', 'slot': '', 'led': 'off', 'name': 'sdn', 'iostats': {'read': '0', 'write': '0', 'chksum': '0'}, 'alias': '', 'health': 'ONLINE', 'sn': '6000c296b793bf526a107a9d823e947a', 'trim_data': {'status': 'trim_unsupported', 'progress': None, 'trim_time': None}, 'path': 'pci-0000:03:00.0-scsi-0:0:14:0', 'model': 'Virtual disk', 'id': 'wwn-0x6000c296b793bf526a107a9d823e947a', 'size': 128849018880}], 'health': 'ONLINE', 'vdev_replacings': [], 'vdev_spares': [], 'type': ''}], 'id': '14181317334440137330', 'health': 'ONLINE', 'iostats': {'read': '0', 'write': '0', 'chksum': '0'}, 'operation': 'none', 'size': '127775277056', 'autotrim': False, 'name': 'Pool-0'}} from GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:24:52 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-8502c4c2-1b69-4858-a3b9-7f3876102a3e tempest-VolumesExtendTest-93540250 None] Total capacity: 119, Free 106. {{(pid=84696) _update_volume_stats /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:757}} Apr 17 15:24:52 user cinder-volume[84696]: DEBUG cinder.manager [None req-8502c4c2-1b69-4858-a3b9-7f3876102a3e tempest-VolumesExtendTest-93540250 None] Notifying Schedulers of capabilities ... {{(pid=84696) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 17 15:24:52 user cinder-volume[84696]: INFO cinder.volume.manager [None req-8502c4c2-1b69-4858-a3b9-7f3876102a3e tempest-VolumesExtendTest-93540250 None] Deleted volume successfully. Apr 17 15:24:52 user cinder-volume[84696]: DEBUG cinder.coordination [None req-8502c4c2-1b69-4858-a3b9-7f3876102a3e tempest-VolumesExtendTest-93540250 None] Lock "cinder-4b0ebe2e-141a-40f1-bcb5-83708d103f54-delete_volume" released by "delete_volume" :: held 5.056s {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:207}} Apr 17 15:24:52 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-b126ed32-230c-4823-8d14-03ab093a03a3 tempest-GroupSnapshotsTest-1926061586 None] Geting {'code': 200, 'error': None, 'data': {'results': 0, 'entries': []}} from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_7a71a6fd-8167-40ab-b057-5f82e0f55947/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:24:52 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-b126ed32-230c-4823-8d14-03ab093a03a3 tempest-GroupSnapshotsTest-1926061586 None] get volume v_7a71a6fd-8167-40ab-b057-5f82e0f55947 info {{(pid=84696) get_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:239}} Apr 17 15:24:52 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-b126ed32-230c-4823-8d14-03ab093a03a3 tempest-GroupSnapshotsTest-1926061586 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_7a71a6fd-8167-40ab-b057-5f82e0f55947 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:24:52 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-b126ed32-230c-4823-8d14-03ab093a03a3 tempest-GroupSnapshotsTest-1926061586 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_7a71a6fd-8167-40ab-b057-5f82e0f55947 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:24:52 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-b126ed32-230c-4823-8d14-03ab093a03a3 tempest-GroupSnapshotsTest-1926061586 None] Geting {'code': 200, 'error': None, 'data': {'origin': None, 'relatime': None, 'acltype': None, 'vscan': None, 'full_name': 'Pool-0/v_7a71a6fd-8167-40ab-b057-5f82e0f55947', 'userrefs': None, 'primarycache': 'all', 'logbias': 'latency', 'creation': '1681746473', 'sync': 'always', 'is_clone': False, 'dedup': 'off', 'sharenfs': None, 'receive_resume_token': None, 'volsize': '1073741824', 'referenced': '57344', 'sharesmb': None, 'createtxg': '187950', 'reservation': '0', 'scontext': None, 'mountpoint': None, 'casesensitivity': None, 'guid': '14235400779179527817', 'usedbyrefreservation': '0', 'dnodesize': None, 'written': '57344', 'logicalused': '28672', 'compressratio': '1.00', 'rootcontext': 'none', 'default_scsi_id': '741804c169718516', 'type': 'volume', 'compression': 'lz4', 'snapdir': None, 'overlay': None, 'encryption': 'off', 'xattr': None, 'volmode': 'default', 'copies': '1', 'snapshot_limit': '18446744073709551615', 'aclinherit': None, 'defcontext': 'none', 'readonly': 'off', 'version': None, 'recordsize': None, 'filesystem_limit': None, 'mounted': None, 'mlslabel': 'none', 'secondarycache': 'all', 'refreservation': '0', 'available': '113909874688', 'san:volume_id': '741804c169718516f6012870cc34ff0f9db44f0b', 'encryptionroot': None, 'exec': None, 'refquota': None, 'refcompressratio': '1.00', 'quota': None, 'utf8only': None, 'keylocation': 'none', 'snapdev': 'hidden', 'snapshot_count': '18446744073709551615', 'fscontext': 'none', 'clones': None, 'canmount': None, 'keystatus': None, 'atime': None, 'usedbysnapshots': '0', 'normalization': None, 'usedbychildren': '0', 'volblocksize': '131072', 'usedbydataset': '57344', 'objsetid': '121931', 'name': 'v_7a71a6fd-8167-40ab-b057-5f82e0f55947', 'defer_destroy': None, 'pbkdf2iters': '0', 'checksum': 'on', 'redundant_metadata': 'all', 'filesystem_count': None, 'devices': None, 'keyformat': 'none', 'setuid': None, 'used': '57344', 'logicalreferenced': '28672', 'context': 'none', 'zoned': None, 'nbmand': None}} from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_7a71a6fd-8167-40ab-b057-5f82e0f55947 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:24:52 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-b126ed32-230c-4823-8d14-03ab093a03a3 tempest-GroupSnapshotsTest-1926061586 None] delete volume:v_7a71a6fd-8167-40ab-b057-5f82e0f55947 recursively children:{'force_umount': True} {{(pid=84696) delete_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:335}} Apr 17 15:24:52 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-b126ed32-230c-4823-8d14-03ab093a03a3 tempest-GroupSnapshotsTest-1926061586 None] Sending pool request DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_7a71a6fd-8167-40ab-b057-5f82e0f55947 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:24:52 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-b126ed32-230c-4823-8d14-03ab093a03a3 tempest-GroupSnapshotsTest-1926061586 None] Sending DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_7a71a6fd-8167-40ab-b057-5f82e0f55947 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:24:53 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-7303fa52-959d-4a80-9bb5-5cd76868f509 tempest-VolumesGetTest-1961901732 None] Geting {'code': 200, 'error': None, 'data': {'name': 'v_3982de76-ea42-41a0-b8e5-4b24b489009b'}} from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:24:53 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-7303fa52-959d-4a80-9bb5-5cd76868f509 tempest-VolumesGetTest-1961901732 None] create volume v_3982de76-ea42-41a0-b8e5-4b24b489009b from snapshot v_3982de76-ea42-41a0-b8e5-4b24b489009b {{(pid=84696) create_volume_from_snapshot /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:674}} Apr 17 15:24:53 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-7303fa52-959d-4a80-9bb5-5cd76868f509 tempest-VolumesGetTest-1961901732 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/clone {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:24:53 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-7303fa52-959d-4a80-9bb5-5cd76868f509 tempest-VolumesGetTest-1961901732 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/clone {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:24:53 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-57c4caf5-947d-4d53-8d87-afb00c52f60a tempest-VolumesExtendTest-93540250 None] Geting {'code': 201, 'error': None, 'data': {'full_name': 'Pool-0/v_31d34366-a0ee-4a52-822b-79de201b5a12', 'is_clone': False, 'name': 'v_31d34366-a0ee-4a52-822b-79de201b5a12'}} from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:24:53 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-57c4caf5-947d-4d53-8d87-afb00c52f60a tempest-VolumesExtendTest-93540250 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (74f5ba97-afaf-45c7-8972-3062721d224b) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'volume_name': 'volume-31d34366-a0ee-4a52-822b-79de201b5a12', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:24:53 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-57c4caf5-947d-4d53-8d87-afb00c52f60a tempest-VolumesExtendTest-93540250 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeOnFinishTask;volume:create, create.end' (8fd09087-ba6d-4f06-8ccb-baa920b8ded3) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:24:53 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-57c4caf5-947d-4d53-8d87-afb00c52f60a tempest-VolumesExtendTest-93540250 None] Volume volume-31d34366-a0ee-4a52-822b-79de201b5a12 (31d34366-a0ee-4a52-822b-79de201b5a12): created successfully Apr 17 15:24:53 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-57c4caf5-947d-4d53-8d87-afb00c52f60a tempest-VolumesExtendTest-93540250 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeOnFinishTask;volume:create, create.end' (8fd09087-ba6d-4f06-8ccb-baa920b8ded3) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:24:53 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-57c4caf5-947d-4d53-8d87-afb00c52f60a tempest-VolumesExtendTest-93540250 None] Flow 'volume_create_manager' (5c71d51e-2fd0-49bd-a294-77374596bb7f) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:24:53 user cinder-volume[84696]: INFO cinder.volume.manager [None req-57c4caf5-947d-4d53-8d87-afb00c52f60a tempest-VolumesExtendTest-93540250 None] Created volume successfully. Apr 17 15:24:54 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-99f3e477-abde-4bf2-8835-cd1893dacfd6 tempest-VolumesExtendTest-93540250 None] create snapshot 42bfd73e-bbbf-4fc6-b4cd-de5cc180f53b for volume 31d34366-a0ee-4a52-822b-79de201b5a12 {{(pid=84696) create_snapshot /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:625}} Apr 17 15:24:54 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-99f3e477-abde-4bf2-8835-cd1893dacfd6 tempest-VolumesExtendTest-93540250 None] cloning v_31d34366-a0ee-4a52-822b-79de201b5a12 to s_42bfd73e-bbbf-4fc6-b4cd-de5cc180f53b {{(pid=84696) _clone_object /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:456}} Apr 17 15:24:54 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-99f3e477-abde-4bf2-8835-cd1893dacfd6 tempest-VolumesExtendTest-93540250 None] create snapshot s_42bfd73e-bbbf-4fc6-b4cd-de5cc180f53b {{(pid=84696) create_snapshot /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:635}} Apr 17 15:24:54 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-99f3e477-abde-4bf2-8835-cd1893dacfd6 tempest-VolumesExtendTest-93540250 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_31d34366-a0ee-4a52-822b-79de201b5a12/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:24:54 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-99f3e477-abde-4bf2-8835-cd1893dacfd6 tempest-VolumesExtendTest-93540250 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_31d34366-a0ee-4a52-822b-79de201b5a12/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:24:54 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-7303fa52-959d-4a80-9bb5-5cd76868f509 tempest-VolumesGetTest-1961901732 None] Geting {'code': 200, 'error': None, 'data': {'origin': 'Pool-0/v_006f20ae-5306-44fe-afdf-e433f79284e8@v_3982de76-ea42-41a0-b8e5-4b24b489009b', 'is_clone': True, 'full_name': 'Pool-0/v_3982de76-ea42-41a0-b8e5-4b24b489009b', 'name': 'v_3982de76-ea42-41a0-b8e5-4b24b489009b'}} from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/clone {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:24:54 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-7303fa52-959d-4a80-9bb5-5cd76868f509 tempest-VolumesGetTest-1961901732 None] get volume v_3982de76-ea42-41a0-b8e5-4b24b489009b info {{(pid=84696) get_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:239}} Apr 17 15:24:54 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-7303fa52-959d-4a80-9bb5-5cd76868f509 tempest-VolumesGetTest-1961901732 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_3982de76-ea42-41a0-b8e5-4b24b489009b {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:24:54 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-7303fa52-959d-4a80-9bb5-5cd76868f509 tempest-VolumesGetTest-1961901732 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_3982de76-ea42-41a0-b8e5-4b24b489009b {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:24:54 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-7303fa52-959d-4a80-9bb5-5cd76868f509 tempest-VolumesGetTest-1961901732 None] Geting {'code': 200, 'error': None, 'data': {'origin': 'Pool-0/v_006f20ae-5306-44fe-afdf-e433f79284e8@v_3982de76-ea42-41a0-b8e5-4b24b489009b', 'relatime': None, 'acltype': None, 'vscan': None, 'full_name': 'Pool-0/v_3982de76-ea42-41a0-b8e5-4b24b489009b', 'userrefs': None, 'primarycache': 'all', 'logbias': 'latency', 'creation': '1681746486', 'sync': 'always', 'is_clone': True, 'dedup': 'off', 'sharenfs': None, 'receive_resume_token': None, 'volsize': '1073741824', 'referenced': '16285696', 'sharesmb': None, 'createtxg': '187985', 'reservation': '0', 'scontext': None, 'mountpoint': None, 'casesensitivity': None, 'guid': '4548803350227175746', 'usedbyrefreservation': '0', 'dnodesize': None, 'written': '0', 'logicalused': '0', 'compressratio': '1.00', 'rootcontext': 'none', 'default_scsi_id': '9906613680b42815', 'type': 'volume', 'compression': 'lz4', 'snapdir': None, 'overlay': None, 'encryption': 'off', 'xattr': None, 'volmode': 'default', 'copies': '1', 'snapshot_limit': '18446744073709551615', 'aclinherit': None, 'defcontext': 'none', 'readonly': 'off', 'version': None, 'recordsize': None, 'filesystem_limit': None, 'mounted': None, 'mlslabel': 'none', 'secondarycache': 'all', 'refreservation': '0', 'available': '113909862400', 'used': '0', 'encryptionroot': None, 'exec': None, 'refquota': None, 'refcompressratio': '1.17', 'quota': None, 'utf8only': None, 'keylocation': 'none', 'snapdev': 'hidden', 'snapshot_count': '18446744073709551615', 'fscontext': 'none', 'clones': None, 'canmount': None, 'keystatus': None, 'atime': None, 'usedbysnapshots': '0', 'normalization': None, 'usedbychildren': '0', 'volblocksize': '131072', 'usedbydataset': '0', 'objsetid': '121287', 'name': 'v_3982de76-ea42-41a0-b8e5-4b24b489009b', 'defer_destroy': None, 'pbkdf2iters': '0', 'checksum': 'on', 'redundant_metadata': 'all', 'filesystem_count': None, 'devices': None, 'keyformat': 'none', 'setuid': None, 'logicalreferenced': '19042304', 'context': 'none', 'zoned': None, 'nbmand': None}} from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_3982de76-ea42-41a0-b8e5-4b24b489009b {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:24:54 user cinder-volume[84696]: DEBUG cinder.volume.volume_utils [None req-7303fa52-959d-4a80-9bb5-5cd76868f509 tempest-VolumesGetTest-1961901732 None] Marking volume 3982de76-ea42-41a0-b8e5-4b24b489009b as bootable. {{(pid=84696) enable_bootable_flag /opt/stack/cinder/cinder/volume/volume_utils.py:1140}} Apr 17 15:24:54 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-7303fa52-959d-4a80-9bb5-5cd76868f509 tempest-VolumesGetTest-1961901732 None] Copying metadata from image b3f0dec1-7758-44a8-82e0-76ada8837428 to 3982de76-ea42-41a0-b8e5-4b24b489009b. {{(pid=84696) _handle_bootable_volume_glance_meta /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:452}} Apr 17 15:24:54 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-7303fa52-959d-4a80-9bb5-5cd76868f509 tempest-VolumesGetTest-1961901732 None] Creating volume glance metadata for volume 3982de76-ea42-41a0-b8e5-4b24b489009b backed by image b3f0dec1-7758-44a8-82e0-76ada8837428 with: {'hw_rng_model': 'virtio', 'owner_specified.openstack.md5': '', 'owner_specified.openstack.object': 'images/cirros-0.5.2-x86_64-disk', 'owner_specified.openstack.sha256': '', 'image_id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'image_name': 'cirros-0.5.2-x86_64-disk', 'checksum': 'b874c39491a2377b8490f5f1e89761a4', 'container_format': 'bare', 'disk_format': 'qcow2', 'min_disk': 0, 'min_ram': 0, 'size': 16300544}. {{(pid=84696) _capture_volume_image_metadata /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:683}} Apr 17 15:24:54 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-7303fa52-959d-4a80-9bb5-5cd76868f509 tempest-VolumesGetTest-1961901732 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (15712217-4f5d-4dcc-b735-df2ed570f9ab) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'volume_name': 'volume-3982de76-ea42-41a0-b8e5-4b24b489009b', 'volume_size': 1, 'image_id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'image_location': (None, None), 'image_meta': {'name': 'cirros-0.5.2-x86_64-disk', 'disk_format': 'qcow2', 'container_format': 'bare', 'visibility': 'public', 'size': 16300544, 'virtual_size': 117440512, 'status': 'active', 'checksum': 'b874c39491a2377b8490f5f1e89761a4', 'protected': False, 'min_ram': 0, 'min_disk': 0, 'owner': 'a43c211b4ed542538799349489b1d884', 'os_hidden': False, 'os_hash_algo': 'sha512', 'os_hash_value': '6b813aa46bb90b4da216a4d19376593fa3f4fc7e617f03a92b7fe11e9a3981cbe8f0959dbebe36225e5f53dc4492341a4863cac4ed1ee0909f3fc78ef9c3e869', 'id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'created_at': datetime.datetime(2023, 4, 17, 15, 19, 53, tzinfo=datetime.timezone.utc), 'updated_at': datetime.datetime(2023, 4, 17, 15, 19, 55, tzinfo=datetime.timezone.utc), 'tags': [], 'file': '/v2/images/b3f0dec1-7758-44a8-82e0-76ada8837428/file', 'properties': {'hw_rng_model': 'virtio', 'owner_specified.openstack.md5': '', 'owner_specified.openstack.object': 'images/cirros-0.5.2-x86_64-disk', 'owner_specified.openstack.sha256': ''}}, 'image_service': }' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:24:54 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-7303fa52-959d-4a80-9bb5-5cd76868f509 tempest-VolumesGetTest-1961901732 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeOnFinishTask;volume:create, create.end' (986d76d0-8ea1-4d49-a4ce-66af7893d13d) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:24:54 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-7303fa52-959d-4a80-9bb5-5cd76868f509 tempest-VolumesGetTest-1961901732 None] Volume volume-3982de76-ea42-41a0-b8e5-4b24b489009b (3982de76-ea42-41a0-b8e5-4b24b489009b): created successfully Apr 17 15:24:54 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-7303fa52-959d-4a80-9bb5-5cd76868f509 tempest-VolumesGetTest-1961901732 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeOnFinishTask;volume:create, create.end' (986d76d0-8ea1-4d49-a4ce-66af7893d13d) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:24:54 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-7303fa52-959d-4a80-9bb5-5cd76868f509 tempest-VolumesGetTest-1961901732 None] Flow 'volume_create_manager' (d96d305d-b889-4e00-adc7-7e12c885857e) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:24:54 user cinder-volume[84696]: INFO cinder.volume.manager [None req-7303fa52-959d-4a80-9bb5-5cd76868f509 tempest-VolumesGetTest-1961901732 None] Created volume successfully. Apr 17 15:24:54 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-b126ed32-230c-4823-8d14-03ab093a03a3 tempest-GroupSnapshotsTest-1926061586 None] Geting {'code': 204} from DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_7a71a6fd-8167-40ab-b057-5f82e0f55947 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:24:54 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-b126ed32-230c-4823-8d14-03ab093a03a3 tempest-GroupSnapshotsTest-1926061586 None] volume v_7a71a6fd-8167-40ab-b057-5f82e0f55947 deleted {{(pid=84696) delete_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:346}} Apr 17 15:24:54 user cinder-volume[84696]: DEBUG cinder.quota [None req-b126ed32-230c-4823-8d14-03ab093a03a3 tempest-GroupSnapshotsTest-1926061586 None] Created reservations ['63ee0b6f-4971-452d-afd3-4edfe7833a5e'] {{(pid=84696) reserve /opt/stack/cinder/cinder/quota.py:755}} Apr 17 15:24:54 user cinder-volume[84696]: WARNING cinder.quota [None req-b126ed32-230c-4823-8d14-03ab093a03a3 tempest-GroupSnapshotsTest-1926061586 None] Deprecated: Default quota for resource: volumes_tempest-GroupSnapshotsTest-volume-type-1057865787 is set by the default quota flag: quota_volumes_tempest-GroupSnapshotsTest-volume-type-1057865787, it is now deprecated. Please use the default quota class for default quota. Apr 17 15:24:54 user cinder-volume[84696]: DEBUG cinder.quota [None req-b126ed32-230c-4823-8d14-03ab093a03a3 tempest-GroupSnapshotsTest-1926061586 None] Created reservations ['0a8904ae-f0b9-4ec4-815e-c1cc6e0dafce', '5766da10-43ae-4352-ad9d-4b2a4ddb7500', 'fee02f7b-7cbd-4739-a604-b4917c69feb0', 'fcef1eed-5155-4379-a992-feea0eda8648'] {{(pid=84696) reserve /opt/stack/cinder/cinder/quota.py:755}} Apr 17 15:24:54 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=58,cinder:UPDATE=31,cinder:INSERT=11 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:24:54 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-b126ed32-230c-4823-8d14-03ab093a03a3 tempest-GroupSnapshotsTest-1926061586 None] Updating volume stats {{(pid=84696) _update_volume_stats /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:724}} Apr 17 15:24:54 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-b126ed32-230c-4823-8d14-03ab093a03a3 tempest-GroupSnapshotsTest-1926061586 None] Get pool Pool-0 fsprops {{(pid=84696) get_pool_stats /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:894}} Apr 17 15:24:54 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-b126ed32-230c-4823-8d14-03ab093a03a3 tempest-GroupSnapshotsTest-1926061586 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:24:54 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-b126ed32-230c-4823-8d14-03ab093a03a3 tempest-GroupSnapshotsTest-1926061586 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:24:55 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-5fb13a1c-30cf-4428-98db-437f48b42dd6 tempest-VolumesGetTest-1961901732 None] Volume reschedule parameters: True retry: {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 15:24:55 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-5fb13a1c-30cf-4428-98db-437f48b42dd6 tempest-VolumesGetTest-1961901732 None] Flow 'volume_create_manager' (4bf66f1d-22bb-437a-b307-2624fe0070b0) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:24:55 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-5fb13a1c-30cf-4428-98db-437f48b42dd6 tempest-VolumesGetTest-1961901732 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (643c62c1-3184-437b-99e9-36d2f3846ef5) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:24:55 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-99f3e477-abde-4bf2-8835-cd1893dacfd6 tempest-VolumesExtendTest-93540250 None] Geting {'code': 200, 'error': None, 'data': {'name': 's_42bfd73e-bbbf-4fc6-b4cd-de5cc180f53b'}} from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_31d34366-a0ee-4a52-822b-79de201b5a12/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:24:55 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-99f3e477-abde-4bf2-8835-cd1893dacfd6 tempest-VolumesExtendTest-93540250 None] create volume s_42bfd73e-bbbf-4fc6-b4cd-de5cc180f53b from snapshot s_42bfd73e-bbbf-4fc6-b4cd-de5cc180f53b {{(pid=84696) create_volume_from_snapshot /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:674}} Apr 17 15:24:55 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-99f3e477-abde-4bf2-8835-cd1893dacfd6 tempest-VolumesExtendTest-93540250 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_31d34366-a0ee-4a52-822b-79de201b5a12/clone {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:24:55 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-99f3e477-abde-4bf2-8835-cd1893dacfd6 tempest-VolumesExtendTest-93540250 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_31d34366-a0ee-4a52-822b-79de201b5a12/clone {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:24:55 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-5fb13a1c-30cf-4428-98db-437f48b42dd6 tempest-VolumesGetTest-1961901732 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (643c62c1-3184-437b-99e9-36d2f3846ef5) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:24:55Z,deleted=False,deleted_at=None,display_description='tempest-@#$%^* description-1018437819',display_name=None,ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=6e83844e-db12-47e9-b1b9-acd2e0f7d31a,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='a65d14b0a2894a6f90506d86feaefd15',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:24:56Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:24:56Z,use_quota=True,user_id='6ffa3051570a4721a48a0171b3cc4d16',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:24:55 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-5fb13a1c-30cf-4428-98db-437f48b42dd6 tempest-VolumesGetTest-1961901732 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (ee5e3f47-aa31-4b2b-bd50-c63e54219069) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:24:55 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-5fb13a1c-30cf-4428-98db-437f48b42dd6 tempest-VolumesGetTest-1961901732 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (ee5e3f47-aa31-4b2b-bd50-c63e54219069) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:24:55 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-5fb13a1c-30cf-4428-98db-437f48b42dd6 tempest-VolumesGetTest-1961901732 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (1bc2ccfc-8de6-469d-b185-68bbcd3df381) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:24:55 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-5fb13a1c-30cf-4428-98db-437f48b42dd6 tempest-VolumesGetTest-1961901732 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (1bc2ccfc-8de6-469d-b185-68bbcd3df381) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': '6e83844e-db12-47e9-b1b9-acd2e0f7d31a', 'volume_name': 'volume-6e83844e-db12-47e9-b1b9-acd2e0f7d31a', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:24:55 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-5fb13a1c-30cf-4428-98db-437f48b42dd6 tempest-VolumesGetTest-1961901732 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (feab1854-8c9c-4d88-a57d-a120801716fc) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:24:55 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-5fb13a1c-30cf-4428-98db-437f48b42dd6 tempest-VolumesGetTest-1961901732 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (feab1854-8c9c-4d88-a57d-a120801716fc) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:24:55 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-5fb13a1c-30cf-4428-98db-437f48b42dd6 tempest-VolumesGetTest-1961901732 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (1f74d704-daf6-4b20-81ce-d213f54707fa) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:24:55 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-5fb13a1c-30cf-4428-98db-437f48b42dd6 tempest-VolumesGetTest-1961901732 None] Volume 6e83844e-db12-47e9-b1b9-acd2e0f7d31a: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-6e83844e-db12-47e9-b1b9-acd2e0f7d31a', 'volume_size': 1} Apr 17 15:24:55 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-5fb13a1c-30cf-4428-98db-437f48b42dd6 tempest-VolumesGetTest-1961901732 None] creating volume v_6e83844e-db12-47e9-b1b9-acd2e0f7d31a. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 15:24:55 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-5fb13a1c-30cf-4428-98db-437f48b42dd6 tempest-VolumesGetTest-1961901732 None] create volume {'name': 'v_6e83844e-db12-47e9-b1b9-acd2e0f7d31a', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 15:24:55 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5fb13a1c-30cf-4428-98db-437f48b42dd6 tempest-VolumesGetTest-1961901732 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:24:55 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5fb13a1c-30cf-4428-98db-437f48b42dd6 tempest-VolumesGetTest-1961901732 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:24:55 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-b126ed32-230c-4823-8d14-03ab093a03a3 tempest-GroupSnapshotsTest-1926061586 None] Geting {'code': 200, 'error': None, 'data': {'available': '113909850112', 'status': 30, 'import_status': {'import_steps': {'san_setup': True, 'vip_setup': True, 'zfs_import': True, 'mount_system_volume': True, 'nas_setup': True}, 'import_successful': True}, 'scan': None, 'encryption': {'enabled': False}, 'deduplication_rate': '1.00', 'sysvol_upgrade': 'not_needed', 'vdevs': [{'name': 'wwn-0x6000c296b793bf526a107a9d823e947a', 'iostats': {'read': '0', 'write': '0', 'chksum': '0'}, 'disks': [{'origin': 'local', 'slot': '', 'led': 'off', 'name': 'sdn', 'iostats': {'read': '0', 'write': '0', 'chksum': '0'}, 'alias': '', 'health': 'ONLINE', 'sn': '6000c296b793bf526a107a9d823e947a', 'trim_data': {'status': 'trim_unsupported', 'progress': None, 'trim_time': None}, 'path': 'pci-0000:03:00.0-scsi-0:0:14:0', 'model': 'Virtual disk', 'id': 'wwn-0x6000c296b793bf526a107a9d823e947a', 'size': 128849018880}], 'health': 'ONLINE', 'vdev_replacings': [], 'vdev_spares': [], 'type': ''}], 'id': '14181317334440137330', 'health': 'ONLINE', 'iostats': {'read': '0', 'write': '0', 'chksum': '0'}, 'operation': 'none', 'size': '127775277056', 'autotrim': False, 'name': 'Pool-0'}} from GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:24:55 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-b126ed32-230c-4823-8d14-03ab093a03a3 tempest-GroupSnapshotsTest-1926061586 None] Total capacity: 119, Free 106. {{(pid=84696) _update_volume_stats /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:757}} Apr 17 15:24:55 user cinder-volume[84696]: DEBUG cinder.manager [None req-b126ed32-230c-4823-8d14-03ab093a03a3 tempest-GroupSnapshotsTest-1926061586 None] Notifying Schedulers of capabilities ... {{(pid=84696) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 17 15:24:55 user cinder-volume[84696]: INFO cinder.volume.manager [None req-b126ed32-230c-4823-8d14-03ab093a03a3 tempest-GroupSnapshotsTest-1926061586 None] Delete group completed successfully. Apr 17 15:24:56 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-120c9c51-a7c0-483f-b2de-e5ce22a20a39 req-6d09bc69-27ce-4ab0-b688-f3045639f36b tempest-TaggedBootDevicesTest_v242-1317742003 None] create export for volume: c74b7878-4936-4015-8b63-142923947310. {{(pid=84696) create_export /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:699}} Apr 17 15:24:56 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-120c9c51-a7c0-483f-b2de-e5ce22a20a39 req-6d09bc69-27ce-4ab0-b688-f3045639f36b tempest-TaggedBootDevicesTest_v242-1317742003 None] ensure volume c74b7878-4936-4015-8b63-142923947310 assigned to a proper target {{(pid=84696) _ensure_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:866}} Apr 17 15:24:56 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [req-120c9c51-a7c0-483f-b2de-e5ce22a20a39 req-6d09bc69-27ce-4ab0-b688-f3045639f36b tempest-TaggedBootDevicesTest_v242-1317742003 None] check if targe iqn.2010-10.org.openstack:c74b7878-4936-4015-8b63-142923947310 exists {{(pid=84696) is_target /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:376}} Apr 17 15:24:56 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-120c9c51-a7c0-483f-b2de-e5ce22a20a39 req-6d09bc69-27ce-4ab0-b688-f3045639f36b tempest-TaggedBootDevicesTest_v242-1317742003 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:c74b7878-4936-4015-8b63-142923947310 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:24:56 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-120c9c51-a7c0-483f-b2de-e5ce22a20a39 req-6d09bc69-27ce-4ab0-b688-f3045639f36b tempest-TaggedBootDevicesTest_v242-1317742003 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:c74b7878-4936-4015-8b63-142923947310 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:24:56 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-120c9c51-a7c0-483f-b2de-e5ce22a20a39 req-6d09bc69-27ce-4ab0-b688-f3045639f36b tempest-TaggedBootDevicesTest_v242-1317742003 None] Geting {'code': 404, 'error': {'class': 'opene.exceptions.ItemNotFoundError', 'message': 'Target iqn.2010-10.org.openstack:c74b7878-4936-4015-8b63-142923947310 not exists.', 'url': 'http://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:c74b7878-4936-4015-8b63-142923947310'}, 'data': None} from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:c74b7878-4936-4015-8b63-142923947310 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:24:56 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-120c9c51-a7c0-483f-b2de-e5ce22a20a39 req-6d09bc69-27ce-4ab0-b688-f3045639f36b tempest-TaggedBootDevicesTest_v242-1317742003 None] create target and attach volume c74b7878-4936-4015-8b63-142923947310 to it {{(pid=84696) _create_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:837}} Apr 17 15:24:56 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [req-120c9c51-a7c0-483f-b2de-e5ce22a20a39 req-6d09bc69-27ce-4ab0-b688-f3045639f36b tempest-TaggedBootDevicesTest_v242-1317742003 None] create target iqn.2010-10.org.openstack:c74b7878-4936-4015-8b63-142923947310 {{(pid=84696) create_target /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:410}} Apr 17 15:24:56 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-120c9c51-a7c0-483f-b2de-e5ce22a20a39 req-6d09bc69-27ce-4ab0-b688-f3045639f36b tempest-TaggedBootDevicesTest_v242-1317742003 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:24:56 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-120c9c51-a7c0-483f-b2de-e5ce22a20a39 req-6d09bc69-27ce-4ab0-b688-f3045639f36b tempest-TaggedBootDevicesTest_v242-1317742003 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:24:56 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-99f3e477-abde-4bf2-8835-cd1893dacfd6 tempest-VolumesExtendTest-93540250 None] Geting {'code': 200, 'error': None, 'data': {'origin': 'Pool-0/v_31d34366-a0ee-4a52-822b-79de201b5a12@s_42bfd73e-bbbf-4fc6-b4cd-de5cc180f53b', 'is_clone': True, 'full_name': 'Pool-0/s_42bfd73e-bbbf-4fc6-b4cd-de5cc180f53b', 'name': 's_42bfd73e-bbbf-4fc6-b4cd-de5cc180f53b'}} from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_31d34366-a0ee-4a52-822b-79de201b5a12/clone {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:24:56 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-99f3e477-abde-4bf2-8835-cd1893dacfd6 tempest-VolumesExtendTest-93540250 None] Sending pool request PUT to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/s_42bfd73e-bbbf-4fc6-b4cd-de5cc180f53b/properties {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:24:56 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-99f3e477-abde-4bf2-8835-cd1893dacfd6 tempest-VolumesExtendTest-93540250 None] Sending PUT to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/s_42bfd73e-bbbf-4fc6-b4cd-de5cc180f53b/properties {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:24:56 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-99f3e477-abde-4bf2-8835-cd1893dacfd6 tempest-VolumesExtendTest-93540250 None] Geting {'code': 201, 'error': None, 'data': None} from PUT to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/s_42bfd73e-bbbf-4fc6-b4cd-de5cc180f53b/properties {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:24:56 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-99f3e477-abde-4bf2-8835-cd1893dacfd6 tempest-VolumesExtendTest-93540250 None] volume s_42bfd73e-bbbf-4fc6-b4cd-de5cc180f53b properties updated {{(pid=84696) modify_property_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:301}} Apr 17 15:24:57 user cinder-volume[84696]: INFO cinder.volume.manager [None req-99f3e477-abde-4bf2-8835-cd1893dacfd6 tempest-VolumesExtendTest-93540250 None] Create snapshot completed successfully Apr 17 15:24:57 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5fb13a1c-30cf-4428-98db-437f48b42dd6 tempest-VolumesGetTest-1961901732 None] Geting {'code': 201, 'error': None, 'data': {'full_name': 'Pool-0/v_6e83844e-db12-47e9-b1b9-acd2e0f7d31a', 'is_clone': False, 'name': 'v_6e83844e-db12-47e9-b1b9-acd2e0f7d31a'}} from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:24:57 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-5fb13a1c-30cf-4428-98db-437f48b42dd6 tempest-VolumesGetTest-1961901732 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (1f74d704-daf6-4b20-81ce-d213f54707fa) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'volume_name': 'volume-6e83844e-db12-47e9-b1b9-acd2e0f7d31a', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:24:57 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-5fb13a1c-30cf-4428-98db-437f48b42dd6 tempest-VolumesGetTest-1961901732 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeOnFinishTask;volume:create, create.end' (8ad5692d-f4f7-4aee-b104-b874f3bc2d49) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:24:57 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-5fb13a1c-30cf-4428-98db-437f48b42dd6 tempest-VolumesGetTest-1961901732 None] Volume volume-6e83844e-db12-47e9-b1b9-acd2e0f7d31a (6e83844e-db12-47e9-b1b9-acd2e0f7d31a): created successfully Apr 17 15:24:57 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-5fb13a1c-30cf-4428-98db-437f48b42dd6 tempest-VolumesGetTest-1961901732 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeOnFinishTask;volume:create, create.end' (8ad5692d-f4f7-4aee-b104-b874f3bc2d49) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:24:57 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-5fb13a1c-30cf-4428-98db-437f48b42dd6 tempest-VolumesGetTest-1961901732 None] Flow 'volume_create_manager' (4bf66f1d-22bb-437a-b307-2624fe0070b0) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:24:57 user cinder-volume[84696]: INFO cinder.volume.manager [None req-5fb13a1c-30cf-4428-98db-437f48b42dd6 tempest-VolumesGetTest-1961901732 None] Created volume successfully. Apr 17 15:24:57 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-30662506-e322-4033-9ea6-c910941899bb tempest-VolumesExtendTest-93540250 None] Extend volume 31d34366-a0ee-4a52-822b-79de201b5a12 {{(pid=84696) extend_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:401}} Apr 17 15:24:57 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-30662506-e322-4033-9ea6-c910941899bb tempest-VolumesExtendTest-93540250 None] jdss extend volume v_31d34366-a0ee-4a52-822b-79de201b5a12 to 2147483648 {{(pid=84696) extend_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:160}} Apr 17 15:24:57 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-30662506-e322-4033-9ea6-c910941899bb tempest-VolumesExtendTest-93540250 None] Sending pool request PUT to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_31d34366-a0ee-4a52-822b-79de201b5a12 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:24:57 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-30662506-e322-4033-9ea6-c910941899bb tempest-VolumesExtendTest-93540250 None] Sending PUT to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_31d34366-a0ee-4a52-822b-79de201b5a12 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:24:57 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-30662506-e322-4033-9ea6-c910941899bb tempest-VolumesExtendTest-93540250 None] Geting {'code': 201, 'error': None, 'data': None} from PUT to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_31d34366-a0ee-4a52-822b-79de201b5a12 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:24:57 user cinder-volume[84696]: INFO cinder.volume.manager [None req-30662506-e322-4033-9ea6-c910941899bb tempest-VolumesExtendTest-93540250 None] Extend volume completed successfully. Apr 17 15:24:57 user cinder-volume[84696]: DEBUG cinder.coordination [None req-5ddb6e37-cf4a-4825-831b-16072286044b tempest-VolumesGetTest-1961901732 None] Acquiring lock "cinder-6e83844e-db12-47e9-b1b9-acd2e0f7d31a-delete_volume" by "delete_volume" {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:192}} Apr 17 15:24:57 user cinder-volume[84696]: DEBUG cinder.coordination [None req-5ddb6e37-cf4a-4825-831b-16072286044b tempest-VolumesGetTest-1961901732 None] Lock "cinder-6e83844e-db12-47e9-b1b9-acd2e0f7d31a-delete_volume" acquired by "delete_volume" :: waited 0.012s {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:196}} Apr 17 15:24:57 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-5ddb6e37-cf4a-4825-831b-16072286044b tempest-VolumesGetTest-1961901732 None] remove_export for volume: 6e83844e-db12-47e9-b1b9-acd2e0f7d31a. {{(pid=84696) remove_export /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:718}} Apr 17 15:24:57 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-5ddb6e37-cf4a-4825-831b-16072286044b tempest-VolumesGetTest-1961901732 None] remove export {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:912}} Apr 17 15:24:57 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-5ddb6e37-cf4a-4825-831b-16072286044b tempest-VolumesGetTest-1961901732 None] detach volume:Volume(_name_id=None,admin_metadata={},attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:24:55Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesGetTest-Volume-193773742',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=6e83844e-db12-47e9-b1b9-acd2e0f7d31a,launched_at=2023-04-17T15:24:57Z,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='a65d14b0a2894a6f90506d86feaefd15',provider_auth='CHAP jwCZRUGL CQmqajRpvidWt8',provider_geometry=None,provider_id=None,provider_location='172.16.0.220:3260,1 iqn.2010-10.org.openstack:6e83844e-db12-47e9-b1b9-acd2e0f7d31a 0',replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:24:56Z,service_uuid='e628352e-c4cf-40eb-8389-cce40663026a',shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='deleting',terminated_at=2023-04-17T15:24:58Z,updated_at=2023-04-17T15:24:58Z,use_quota=True,user_id='6ffa3051570a4721a48a0171b3cc4d16',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c) from target:iqn.2010-10.org.openstack:6e83844e-db12-47e9-b1b9-acd2e0f7d31a. {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:913}} Apr 17 15:24:57 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-5ddb6e37-cf4a-4825-831b-16072286044b tempest-VolumesGetTest-1961901732 None] detach volume v_6e83844e-db12-47e9-b1b9-acd2e0f7d31a from target iqn.2010-10.org.openstack:6e83844e-db12-47e9-b1b9-acd2e0f7d31a {{(pid=84696) detach_target_vol /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:606}} Apr 17 15:24:57 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5ddb6e37-cf4a-4825-831b-16072286044b tempest-VolumesGetTest-1961901732 None] Sending pool request DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:6e83844e-db12-47e9-b1b9-acd2e0f7d31a/luns/v_6e83844e-db12-47e9-b1b9-acd2e0f7d31a {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:24:57 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5ddb6e37-cf4a-4825-831b-16072286044b tempest-VolumesGetTest-1961901732 None] Sending DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:6e83844e-db12-47e9-b1b9-acd2e0f7d31a/luns/v_6e83844e-db12-47e9-b1b9-acd2e0f7d31a {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:24:57 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5ddb6e37-cf4a-4825-831b-16072286044b tempest-VolumesGetTest-1961901732 None] Geting {'code': 404, 'error': {'class': 'opene.exceptions.ItemNotFoundError', 'message': 'Target iqn.2010-10.org.openstack:6e83844e-db12-47e9-b1b9-acd2e0f7d31a not exists.', 'url': 'http://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:6e83844e-db12-47e9-b1b9-acd2e0f7d31a/luns/v_6e83844e-db12-47e9-b1b9-acd2e0f7d31a'}, 'data': None} from DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:6e83844e-db12-47e9-b1b9-acd2e0f7d31a/luns/v_6e83844e-db12-47e9-b1b9-acd2e0f7d31a {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:24:57 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-5ddb6e37-cf4a-4825-831b-16072286044b tempest-VolumesGetTest-1961901732 None] failed to remove resource iqn.2010-10.org.openstack:6e83844e-db12-47e9-b1b9-acd2e0f7d31a because of JDSS resource v_6e83844e-db12-47e9-b1b9-acd2e0f7d31a DNE. {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:920}} Apr 17 15:24:57 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-5ddb6e37-cf4a-4825-831b-16072286044b tempest-VolumesGetTest-1961901732 None] delete target: iqn.2010-10.org.openstack:6e83844e-db12-47e9-b1b9-acd2e0f7d31a {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:930}} Apr 17 15:24:57 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-5ddb6e37-cf4a-4825-831b-16072286044b tempest-VolumesGetTest-1961901732 None] delete target iqn.2010-10.org.openstack:6e83844e-db12-47e9-b1b9-acd2e0f7d31a {{(pid=84696) delete_target /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:442}} Apr 17 15:24:57 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5ddb6e37-cf4a-4825-831b-16072286044b tempest-VolumesGetTest-1961901732 None] Sending pool request DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:6e83844e-db12-47e9-b1b9-acd2e0f7d31a {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:24:57 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5ddb6e37-cf4a-4825-831b-16072286044b tempest-VolumesGetTest-1961901732 None] Sending DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:6e83844e-db12-47e9-b1b9-acd2e0f7d31a {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:24:57 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5ddb6e37-cf4a-4825-831b-16072286044b tempest-VolumesGetTest-1961901732 None] Geting {'code': 404, 'error': {'class': 'opene.exceptions.ItemNotFoundError', 'message': 'Target iqn.2010-10.org.openstack:6e83844e-db12-47e9-b1b9-acd2e0f7d31a not exists.', 'url': 'http://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:6e83844e-db12-47e9-b1b9-acd2e0f7d31a'}, 'data': None} from DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:6e83844e-db12-47e9-b1b9-acd2e0f7d31a {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:24:57 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-5ddb6e37-cf4a-4825-831b-16072286044b tempest-VolumesGetTest-1961901732 None] failed to remove resource iqn.2010-10.org.openstack:6e83844e-db12-47e9-b1b9-acd2e0f7d31a because of JDSS resource iqn.2010-10.org.openstack:6e83844e-db12-47e9-b1b9-acd2e0f7d31a DNE. {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:935}} Apr 17 15:24:57 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-5ddb6e37-cf4a-4825-831b-16072286044b tempest-VolumesGetTest-1961901732 None] deleating volume v_6e83844e-db12-47e9-b1b9-acd2e0f7d31a {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 15:24:57 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-5ddb6e37-cf4a-4825-831b-16072286044b tempest-VolumesGetTest-1961901732 None] get snapshots for volume v_6e83844e-db12-47e9-b1b9-acd2e0f7d31a {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 15:24:57 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5ddb6e37-cf4a-4825-831b-16072286044b tempest-VolumesGetTest-1961901732 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_6e83844e-db12-47e9-b1b9-acd2e0f7d31a/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:24:57 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5ddb6e37-cf4a-4825-831b-16072286044b tempest-VolumesGetTest-1961901732 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_6e83844e-db12-47e9-b1b9-acd2e0f7d31a/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:24:58 user cinder-volume[84696]: DEBUG cinder.coordination [None req-f0e17204-bc80-43a6-be0b-c1c052993cce tempest-VolumesExtendTest-93540250 None] Acquiring lock "cinder-42bfd73e-bbbf-4fc6-b4cd-de5cc180f53b-delete_snapshot" by "delete_snapshot" {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:192}} Apr 17 15:24:58 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5ddb6e37-cf4a-4825-831b-16072286044b tempest-VolumesGetTest-1961901732 None] Geting {'code': 200, 'error': None, 'data': {'results': 0, 'entries': []}} from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_6e83844e-db12-47e9-b1b9-acd2e0f7d31a/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:24:58 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-5ddb6e37-cf4a-4825-831b-16072286044b tempest-VolumesGetTest-1961901732 None] get volume v_6e83844e-db12-47e9-b1b9-acd2e0f7d31a info {{(pid=84696) get_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:239}} Apr 17 15:24:58 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5ddb6e37-cf4a-4825-831b-16072286044b tempest-VolumesGetTest-1961901732 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_6e83844e-db12-47e9-b1b9-acd2e0f7d31a {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:24:58 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5ddb6e37-cf4a-4825-831b-16072286044b tempest-VolumesGetTest-1961901732 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_6e83844e-db12-47e9-b1b9-acd2e0f7d31a {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:24:58 user cinder-volume[84696]: DEBUG cinder.coordination [None req-f0e17204-bc80-43a6-be0b-c1c052993cce tempest-VolumesExtendTest-93540250 None] Lock "cinder-42bfd73e-bbbf-4fc6-b4cd-de5cc180f53b-delete_snapshot" acquired by "delete_snapshot" :: waited 0.032s {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:196}} Apr 17 15:24:58 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-f0e17204-bc80-43a6-be0b-c1c052993cce tempest-VolumesExtendTest-93540250 None] deleating snapshot s_42bfd73e-bbbf-4fc6-b4cd-de5cc180f53b. {{(pid=84696) delete_snapshot /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:649}} Apr 17 15:24:58 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-f0e17204-bc80-43a6-be0b-c1c052993cce tempest-VolumesExtendTest-93540250 None] get snapshots for volume s_42bfd73e-bbbf-4fc6-b4cd-de5cc180f53b {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 15:24:58 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-f0e17204-bc80-43a6-be0b-c1c052993cce tempest-VolumesExtendTest-93540250 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/s_42bfd73e-bbbf-4fc6-b4cd-de5cc180f53b/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:24:58 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-f0e17204-bc80-43a6-be0b-c1c052993cce tempest-VolumesExtendTest-93540250 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/s_42bfd73e-bbbf-4fc6-b4cd-de5cc180f53b/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:24:58 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5ddb6e37-cf4a-4825-831b-16072286044b tempest-VolumesGetTest-1961901732 None] Geting {'code': 200, 'error': None, 'data': {'origin': None, 'relatime': None, 'acltype': None, 'vscan': None, 'full_name': 'Pool-0/v_6e83844e-db12-47e9-b1b9-acd2e0f7d31a', 'userrefs': None, 'primarycache': 'all', 'logbias': 'latency', 'creation': '1681746490', 'sync': 'always', 'is_clone': False, 'dedup': 'off', 'sharenfs': None, 'receive_resume_token': None, 'volsize': '1073741824', 'referenced': '57344', 'sharesmb': None, 'createtxg': '187996', 'reservation': '0', 'scontext': None, 'mountpoint': None, 'casesensitivity': None, 'guid': '11632385578241281089', 'usedbyrefreservation': '0', 'dnodesize': None, 'written': '57344', 'logicalused': '28672', 'compressratio': '1.00', 'rootcontext': 'none', 'default_scsi_id': 'd5801167bbf4ddd5', 'type': 'volume', 'compression': 'lz4', 'snapdir': None, 'overlay': None, 'encryption': 'off', 'xattr': None, 'volmode': 'default', 'copies': '1', 'snapshot_limit': '18446744073709551615', 'aclinherit': None, 'defcontext': 'none', 'readonly': 'off', 'version': None, 'recordsize': None, 'filesystem_limit': None, 'mounted': None, 'mlslabel': 'none', 'secondarycache': 'all', 'refreservation': '0', 'available': '113909673984', 'san:volume_id': 'd5801167bbf4ddd524c7b7bf69912d236e63d4c0', 'encryptionroot': None, 'exec': None, 'refquota': None, 'refcompressratio': '1.00', 'quota': None, 'utf8only': None, 'keylocation': 'none', 'snapdev': 'hidden', 'snapshot_count': '18446744073709551615', 'fscontext': 'none', 'clones': None, 'canmount': None, 'keystatus': None, 'atime': None, 'usedbysnapshots': '0', 'normalization': None, 'usedbychildren': '0', 'volblocksize': '131072', 'usedbydataset': '57344', 'objsetid': '121306', 'name': 'v_6e83844e-db12-47e9-b1b9-acd2e0f7d31a', 'defer_destroy': None, 'pbkdf2iters': '0', 'checksum': 'on', 'redundant_metadata': 'all', 'filesystem_count': None, 'devices': None, 'keyformat': 'none', 'setuid': None, 'used': '57344', 'logicalreferenced': '28672', 'context': 'none', 'zoned': None, 'nbmand': None}} from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_6e83844e-db12-47e9-b1b9-acd2e0f7d31a {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:24:58 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-5ddb6e37-cf4a-4825-831b-16072286044b tempest-VolumesGetTest-1961901732 None] delete volume:v_6e83844e-db12-47e9-b1b9-acd2e0f7d31a recursively children:{'force_umount': True} {{(pid=84696) delete_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:335}} Apr 17 15:24:58 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5ddb6e37-cf4a-4825-831b-16072286044b tempest-VolumesGetTest-1961901732 None] Sending pool request DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_6e83844e-db12-47e9-b1b9-acd2e0f7d31a {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:24:58 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5ddb6e37-cf4a-4825-831b-16072286044b tempest-VolumesGetTest-1961901732 None] Sending DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_6e83844e-db12-47e9-b1b9-acd2e0f7d31a {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:24:59 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-f0e17204-bc80-43a6-be0b-c1c052993cce tempest-VolumesExtendTest-93540250 None] Geting {'code': 200, 'error': None, 'data': {'results': 0, 'entries': []}} from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/s_42bfd73e-bbbf-4fc6-b4cd-de5cc180f53b/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:24:59 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-f0e17204-bc80-43a6-be0b-c1c052993cce tempest-VolumesExtendTest-93540250 None] get volume s_42bfd73e-bbbf-4fc6-b4cd-de5cc180f53b info {{(pid=84696) get_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:239}} Apr 17 15:24:59 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-f0e17204-bc80-43a6-be0b-c1c052993cce tempest-VolumesExtendTest-93540250 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/s_42bfd73e-bbbf-4fc6-b4cd-de5cc180f53b {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:24:59 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-f0e17204-bc80-43a6-be0b-c1c052993cce tempest-VolumesExtendTest-93540250 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/s_42bfd73e-bbbf-4fc6-b4cd-de5cc180f53b {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:24:59 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-f0e17204-bc80-43a6-be0b-c1c052993cce tempest-VolumesExtendTest-93540250 None] Geting {'code': 200, 'error': None, 'data': {'origin': 'Pool-0/v_31d34366-a0ee-4a52-822b-79de201b5a12@s_42bfd73e-bbbf-4fc6-b4cd-de5cc180f53b', 'relatime': None, 'acltype': None, 'vscan': None, 'full_name': 'Pool-0/s_42bfd73e-bbbf-4fc6-b4cd-de5cc180f53b', 'userrefs': None, 'primarycache': 'all', 'logbias': 'latency', 'creation': '1681746489', 'sync': 'always', 'is_clone': True, 'dedup': 'off', 'sharenfs': None, 'receive_resume_token': None, 'volsize': '1073741824', 'referenced': '57344', 'sharesmb': None, 'createtxg': '187993', 'reservation': '0', 'scontext': None, 'mountpoint': None, 'casesensitivity': None, 'guid': '16882120684687976463', 'usedbyrefreservation': '0', 'dnodesize': None, 'written': '0', 'logicalused': '0', 'compressratio': '1.00', 'rootcontext': 'none', 'default_scsi_id': '36c789935e4d9ad2', 'type': 'volume', 'compression': 'lz4', 'snapdir': None, 'overlay': None, 'encryption': 'off', 'xattr': None, 'volmode': 'default', 'copies': '1', 'snapshot_limit': '18446744073709551615', 'aclinherit': None, 'defcontext': 'none', 'readonly': 'on', 'version': None, 'recordsize': None, 'filesystem_limit': None, 'mounted': None, 'mlslabel': 'none', 'secondarycache': 'all', 'refreservation': '0', 'available': '113909673984', 'used': '0', 'encryptionroot': None, 'exec': None, 'refquota': None, 'refcompressratio': '1.00', 'quota': None, 'utf8only': None, 'keylocation': 'none', 'snapdev': 'hidden', 'snapshot_count': '18446744073709551615', 'fscontext': 'none', 'clones': None, 'canmount': None, 'keystatus': None, 'atime': None, 'usedbysnapshots': '0', 'normalization': None, 'usedbychildren': '0', 'volblocksize': '131072', 'usedbydataset': '0', 'objsetid': '123276', 'name': 's_42bfd73e-bbbf-4fc6-b4cd-de5cc180f53b', 'defer_destroy': None, 'pbkdf2iters': '0', 'checksum': 'on', 'redundant_metadata': 'all', 'filesystem_count': None, 'devices': None, 'keyformat': 'none', 'setuid': None, 'logicalreferenced': '28672', 'context': 'none', 'zoned': None, 'nbmand': None}} from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/s_42bfd73e-bbbf-4fc6-b4cd-de5cc180f53b {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:24:59 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-f0e17204-bc80-43a6-be0b-c1c052993cce tempest-VolumesExtendTest-93540250 None] delete snapshot s_42bfd73e-bbbf-4fc6-b4cd-de5cc180f53b of volume v_31d34366-a0ee-4a52-822b-79de201b5a12 {{(pid=84696) delete_snapshot /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:751}} Apr 17 15:24:59 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-f0e17204-bc80-43a6-be0b-c1c052993cce tempest-VolumesExtendTest-93540250 None] Sending pool request DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_31d34366-a0ee-4a52-822b-79de201b5a12/snapshots/s_42bfd73e-bbbf-4fc6-b4cd-de5cc180f53b {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:24:59 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-f0e17204-bc80-43a6-be0b-c1c052993cce tempest-VolumesExtendTest-93540250 None] Sending DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_31d34366-a0ee-4a52-822b-79de201b5a12/snapshots/s_42bfd73e-bbbf-4fc6-b4cd-de5cc180f53b {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:24:59 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-120c9c51-a7c0-483f-b2de-e5ce22a20a39 req-6d09bc69-27ce-4ab0-b688-f3045639f36b tempest-TaggedBootDevicesTest_v242-1317742003 None] Geting {'code': 201, 'error': None, 'data': {'active': True, 'allow_ip': [], 'conflicted': False, 'deny_ip': [], 'incoming_users_active': True, 'name': 'iqn.2010-10.org.openstack:c74b7878-4936-4015-8b63-142923947310', 'outgoing_user': None}} from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:24:59 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [req-120c9c51-a7c0-483f-b2de-e5ce22a20a39 req-6d09bc69-27ce-4ab0-b688-f3045639f36b tempest-TaggedBootDevicesTest_v242-1317742003 None] atach volume v_c74b7878-4936-4015-8b63-142923947310 to target iqn.2010-10.org.openstack:c74b7878-4936-4015-8b63-142923947310 {{(pid=84696) attach_target_vol /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:576}} Apr 17 15:24:59 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-120c9c51-a7c0-483f-b2de-e5ce22a20a39 req-6d09bc69-27ce-4ab0-b688-f3045639f36b tempest-TaggedBootDevicesTest_v242-1317742003 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:c74b7878-4936-4015-8b63-142923947310/luns {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:24:59 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-120c9c51-a7c0-483f-b2de-e5ce22a20a39 req-6d09bc69-27ce-4ab0-b688-f3045639f36b tempest-TaggedBootDevicesTest_v242-1317742003 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:c74b7878-4936-4015-8b63-142923947310/luns {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:24:59 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-e901a82d-2ea1-4b10-a528-c1880ebdb240 tempest-VolumeTypesTest-1824483253 None] Volume reschedule parameters: True retry: {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 15:24:59 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e901a82d-2ea1-4b10-a528-c1880ebdb240 tempest-VolumeTypesTest-1824483253 None] Flow 'volume_create_manager' (f751a11d-b614-4fd7-a7e3-b2057694e5ab) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:24:59 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e901a82d-2ea1-4b10-a528-c1880ebdb240 tempest-VolumeTypesTest-1824483253 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (8d3085d6-39dc-454c-bb00-b4d3e7d0b0ef) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:24:59 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e901a82d-2ea1-4b10-a528-c1880ebdb240 tempest-VolumeTypesTest-1824483253 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (8d3085d6-39dc-454c-bb00-b4d3e7d0b0ef) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:25:00Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumeTypesTest-volume-2002497870',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=29534406-54ef-4f00-9568-69ae248eae21,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='1741e85eaa4b420cabe83f57ee957bb7',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:25:00Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:25:00Z,use_quota=True,user_id='685e853373e84057b2eb234a0815d817',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(5f3a1ff8-565b-4399-ab5f-1d26e9b981fe),volume_type_id=5f3a1ff8-565b-4399-ab5f-1d26e9b981fe)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:24:59 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e901a82d-2ea1-4b10-a528-c1880ebdb240 tempest-VolumeTypesTest-1824483253 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (d5384936-7114-490c-a37d-28893f7381c2) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:24:59 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e901a82d-2ea1-4b10-a528-c1880ebdb240 tempest-VolumeTypesTest-1824483253 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (d5384936-7114-490c-a37d-28893f7381c2) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:24:59 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e901a82d-2ea1-4b10-a528-c1880ebdb240 tempest-VolumeTypesTest-1824483253 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (c7d7148e-4570-48b1-92d2-71e0850f43bf) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:24:59 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e901a82d-2ea1-4b10-a528-c1880ebdb240 tempest-VolumeTypesTest-1824483253 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (c7d7148e-4570-48b1-92d2-71e0850f43bf) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': '29534406-54ef-4f00-9568-69ae248eae21', 'volume_name': 'volume-29534406-54ef-4f00-9568-69ae248eae21', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:24:59 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e901a82d-2ea1-4b10-a528-c1880ebdb240 tempest-VolumeTypesTest-1824483253 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (123726d5-a56d-491b-b197-612d57e113dc) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:24:59 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e901a82d-2ea1-4b10-a528-c1880ebdb240 tempest-VolumeTypesTest-1824483253 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (123726d5-a56d-491b-b197-612d57e113dc) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:24:59 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e901a82d-2ea1-4b10-a528-c1880ebdb240 tempest-VolumeTypesTest-1824483253 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (59d93612-dc48-4e87-b44a-274ce009ae60) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:24:59 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-e901a82d-2ea1-4b10-a528-c1880ebdb240 tempest-VolumeTypesTest-1824483253 None] Volume 29534406-54ef-4f00-9568-69ae248eae21: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-29534406-54ef-4f00-9568-69ae248eae21', 'volume_size': 1} Apr 17 15:24:59 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-e901a82d-2ea1-4b10-a528-c1880ebdb240 tempest-VolumeTypesTest-1824483253 None] creating volume v_29534406-54ef-4f00-9568-69ae248eae21. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 15:24:59 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-e901a82d-2ea1-4b10-a528-c1880ebdb240 tempest-VolumeTypesTest-1824483253 None] create volume {'name': 'v_29534406-54ef-4f00-9568-69ae248eae21', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 15:24:59 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-e901a82d-2ea1-4b10-a528-c1880ebdb240 tempest-VolumeTypesTest-1824483253 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:24:59 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-e901a82d-2ea1-4b10-a528-c1880ebdb240 tempest-VolumeTypesTest-1824483253 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:25:01 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5ddb6e37-cf4a-4825-831b-16072286044b tempest-VolumesGetTest-1961901732 None] Geting {'code': 204} from DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_6e83844e-db12-47e9-b1b9-acd2e0f7d31a {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:25:01 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-5ddb6e37-cf4a-4825-831b-16072286044b tempest-VolumesGetTest-1961901732 None] volume v_6e83844e-db12-47e9-b1b9-acd2e0f7d31a deleted {{(pid=84696) delete_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:346}} Apr 17 15:25:01 user cinder-volume[84696]: DEBUG cinder.quota [None req-5ddb6e37-cf4a-4825-831b-16072286044b tempest-VolumesGetTest-1961901732 None] Created reservations ['9084bf9a-0ee2-4040-a0f7-a910cf4b4e6e', 'fb3e7e98-2b7d-42b2-b742-94338f724353', '4d587117-294e-4a7e-81e5-a4f632e89201', '07ef99a1-9b4a-4edf-9688-c619bbfa6afa'] {{(pid=84696) reserve /opt/stack/cinder/cinder/quota.py:755}} Apr 17 15:25:01 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=60,cinder:DELETE=3,cinder:UPDATE=26,cinder:INSERT=11 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:25:01 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-5ddb6e37-cf4a-4825-831b-16072286044b tempest-VolumesGetTest-1961901732 None] Updating volume stats {{(pid=84696) _update_volume_stats /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:724}} Apr 17 15:25:01 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-5ddb6e37-cf4a-4825-831b-16072286044b tempest-VolumesGetTest-1961901732 None] Get pool Pool-0 fsprops {{(pid=84696) get_pool_stats /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:894}} Apr 17 15:25:01 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5ddb6e37-cf4a-4825-831b-16072286044b tempest-VolumesGetTest-1961901732 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:25:01 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5ddb6e37-cf4a-4825-831b-16072286044b tempest-VolumesGetTest-1961901732 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:25:01 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-120c9c51-a7c0-483f-b2de-e5ce22a20a39 req-6d09bc69-27ce-4ab0-b688-f3045639f36b tempest-TaggedBootDevicesTest_v242-1317742003 None] Geting {'code': 201, 'error': None, 'data': {'block_size': 512, 'device_handler': 'vdisk_blockio', 'eui': '6132396565663337', 'lun': 0, 'mode': 'wt', 'name': 'v_c74b7878-4936-4015-8b63-142923947310', 'prod_id': 'Storage', 'scsi_id': 'a29eef3759db89b1'}} from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:c74b7878-4936-4015-8b63-142923947310/luns {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:25:01 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [req-120c9c51-a7c0-483f-b2de-e5ce22a20a39 req-6d09bc69-27ce-4ab0-b688-f3045639f36b tempest-TaggedBootDevicesTest_v242-1317742003 None] add credentails to target iqn.2010-10.org.openstack:c74b7878-4936-4015-8b63-142923947310 {{(pid=84696) create_target_user /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:474}} Apr 17 15:25:01 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-120c9c51-a7c0-483f-b2de-e5ce22a20a39 req-6d09bc69-27ce-4ab0-b688-f3045639f36b tempest-TaggedBootDevicesTest_v242-1317742003 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:c74b7878-4936-4015-8b63-142923947310/incoming-users {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:25:01 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-120c9c51-a7c0-483f-b2de-e5ce22a20a39 req-6d09bc69-27ce-4ab0-b688-f3045639f36b tempest-TaggedBootDevicesTest_v242-1317742003 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:c74b7878-4936-4015-8b63-142923947310/incoming-users {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:25:01 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-f0e17204-bc80-43a6-be0b-c1c052993cce tempest-VolumesExtendTest-93540250 None] Geting {'code': 204} from DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_31d34366-a0ee-4a52-822b-79de201b5a12/snapshots/s_42bfd73e-bbbf-4fc6-b4cd-de5cc180f53b {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:25:01 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-f0e17204-bc80-43a6-be0b-c1c052993cce tempest-VolumesExtendTest-93540250 None] snapshot s_42bfd73e-bbbf-4fc6-b4cd-de5cc180f53b deleted {{(pid=84696) delete_snapshot /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:772}} Apr 17 15:25:01 user cinder-volume[84696]: WARNING cinder.quota [None req-f0e17204-bc80-43a6-be0b-c1c052993cce tempest-VolumesExtendTest-93540250 None] Deprecated: Default quota for resource: snapshots_joviandss-2 is set by the default quota flag: quota_snapshots_joviandss-2, it is now deprecated. Please use the default quota class for default quota. Apr 17 15:25:01 user cinder-volume[84696]: DEBUG cinder.quota [None req-f0e17204-bc80-43a6-be0b-c1c052993cce tempest-VolumesExtendTest-93540250 None] Created reservations ['a7b74bde-65b3-4c28-a977-cea91acd0ca2', 'ccc52beb-ea7a-4b78-a780-27ac8ad2d434', 'bdce7fa0-8b1d-4daa-ac98-dc2863db6319', 'd076bd22-5bc3-4d59-8b0a-85ba0328f60f'] {{(pid=84696) reserve /opt/stack/cinder/cinder/quota.py:755}} Apr 17 15:25:01 user cinder-volume[84696]: INFO cinder.volume.manager [None req-f0e17204-bc80-43a6-be0b-c1c052993cce tempest-VolumesExtendTest-93540250 None] Delete snapshot completed successfully. Apr 17 15:25:01 user cinder-volume[84696]: DEBUG cinder.coordination [None req-f0e17204-bc80-43a6-be0b-c1c052993cce tempest-VolumesExtendTest-93540250 None] Lock "cinder-42bfd73e-bbbf-4fc6-b4cd-de5cc180f53b-delete_snapshot" released by "delete_snapshot" :: held 3.089s {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:207}} Apr 17 15:25:01 user cinder-volume[84696]: DEBUG cinder.coordination [None req-25c80733-a960-453f-9159-ec587e0787bf tempest-VolumesExtendTest-93540250 None] Acquiring lock "cinder-31d34366-a0ee-4a52-822b-79de201b5a12-delete_volume" by "delete_volume" {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:192}} Apr 17 15:25:01 user cinder-volume[84696]: DEBUG cinder.coordination [None req-25c80733-a960-453f-9159-ec587e0787bf tempest-VolumesExtendTest-93540250 None] Lock "cinder-31d34366-a0ee-4a52-822b-79de201b5a12-delete_volume" acquired by "delete_volume" :: waited 0.010s {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:196}} Apr 17 15:25:01 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-25c80733-a960-453f-9159-ec587e0787bf tempest-VolumesExtendTest-93540250 None] remove_export for volume: 31d34366-a0ee-4a52-822b-79de201b5a12. {{(pid=84696) remove_export /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:718}} Apr 17 15:25:01 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-25c80733-a960-453f-9159-ec587e0787bf tempest-VolumesExtendTest-93540250 None] remove export {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:912}} Apr 17 15:25:01 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-25c80733-a960-453f-9159-ec587e0787bf tempest-VolumesExtendTest-93540250 None] detach volume:Volume(_name_id=None,admin_metadata={},attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:24:52Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesExtendTest-Volume-2061662394',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=31d34366-a0ee-4a52-822b-79de201b5a12,launched_at=2023-04-17T15:24:53Z,metadata={},migration_status=None,multiattach=False,previous_status='available',project_id='ef25ef72b5a64e2c8a60d3d139367fd1',provider_auth='CHAP eXmGVTnJ tCXy17OMxLeHPq',provider_geometry=None,provider_id=None,provider_location='172.16.0.220:3260,1 iqn.2010-10.org.openstack:31d34366-a0ee-4a52-822b-79de201b5a12 0',replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:24:52Z,service_uuid='e628352e-c4cf-40eb-8389-cce40663026a',shared_targets=True,size=2,snapshot_id=None,snapshots=,source_volid=None,status='deleting',terminated_at=2023-04-17T15:25:02Z,updated_at=2023-04-17T15:25:02Z,use_quota=True,user_id='d473e2b76b554904a4d625f35a78c038',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c) from target:iqn.2010-10.org.openstack:31d34366-a0ee-4a52-822b-79de201b5a12. {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:913}} Apr 17 15:25:01 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-25c80733-a960-453f-9159-ec587e0787bf tempest-VolumesExtendTest-93540250 None] detach volume v_31d34366-a0ee-4a52-822b-79de201b5a12 from target iqn.2010-10.org.openstack:31d34366-a0ee-4a52-822b-79de201b5a12 {{(pid=84696) detach_target_vol /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:606}} Apr 17 15:25:01 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-25c80733-a960-453f-9159-ec587e0787bf tempest-VolumesExtendTest-93540250 None] Sending pool request DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:31d34366-a0ee-4a52-822b-79de201b5a12/luns/v_31d34366-a0ee-4a52-822b-79de201b5a12 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:25:01 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-25c80733-a960-453f-9159-ec587e0787bf tempest-VolumesExtendTest-93540250 None] Sending DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:31d34366-a0ee-4a52-822b-79de201b5a12/luns/v_31d34366-a0ee-4a52-822b-79de201b5a12 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:25:02 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-25c80733-a960-453f-9159-ec587e0787bf tempest-VolumesExtendTest-93540250 None] Geting {'code': 404, 'error': {'class': 'opene.exceptions.ItemNotFoundError', 'message': 'Target iqn.2010-10.org.openstack:31d34366-a0ee-4a52-822b-79de201b5a12 not exists.', 'url': 'http://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:31d34366-a0ee-4a52-822b-79de201b5a12/luns/v_31d34366-a0ee-4a52-822b-79de201b5a12'}, 'data': None} from DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:31d34366-a0ee-4a52-822b-79de201b5a12/luns/v_31d34366-a0ee-4a52-822b-79de201b5a12 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:25:02 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-25c80733-a960-453f-9159-ec587e0787bf tempest-VolumesExtendTest-93540250 None] failed to remove resource iqn.2010-10.org.openstack:31d34366-a0ee-4a52-822b-79de201b5a12 because of JDSS resource v_31d34366-a0ee-4a52-822b-79de201b5a12 DNE. {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:920}} Apr 17 15:25:02 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-25c80733-a960-453f-9159-ec587e0787bf tempest-VolumesExtendTest-93540250 None] delete target: iqn.2010-10.org.openstack:31d34366-a0ee-4a52-822b-79de201b5a12 {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:930}} Apr 17 15:25:02 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-25c80733-a960-453f-9159-ec587e0787bf tempest-VolumesExtendTest-93540250 None] delete target iqn.2010-10.org.openstack:31d34366-a0ee-4a52-822b-79de201b5a12 {{(pid=84696) delete_target /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:442}} Apr 17 15:25:02 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-25c80733-a960-453f-9159-ec587e0787bf tempest-VolumesExtendTest-93540250 None] Sending pool request DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:31d34366-a0ee-4a52-822b-79de201b5a12 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:25:02 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-25c80733-a960-453f-9159-ec587e0787bf tempest-VolumesExtendTest-93540250 None] Sending DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:31d34366-a0ee-4a52-822b-79de201b5a12 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:25:02 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-25c80733-a960-453f-9159-ec587e0787bf tempest-VolumesExtendTest-93540250 None] Geting {'code': 404, 'error': {'class': 'opene.exceptions.ItemNotFoundError', 'message': 'Target iqn.2010-10.org.openstack:31d34366-a0ee-4a52-822b-79de201b5a12 not exists.', 'url': 'http://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:31d34366-a0ee-4a52-822b-79de201b5a12'}, 'data': None} from DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:31d34366-a0ee-4a52-822b-79de201b5a12 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:25:02 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-25c80733-a960-453f-9159-ec587e0787bf tempest-VolumesExtendTest-93540250 None] failed to remove resource iqn.2010-10.org.openstack:31d34366-a0ee-4a52-822b-79de201b5a12 because of JDSS resource iqn.2010-10.org.openstack:31d34366-a0ee-4a52-822b-79de201b5a12 DNE. {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:935}} Apr 17 15:25:02 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-25c80733-a960-453f-9159-ec587e0787bf tempest-VolumesExtendTest-93540250 None] deleating volume v_31d34366-a0ee-4a52-822b-79de201b5a12 {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 15:25:02 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-25c80733-a960-453f-9159-ec587e0787bf tempest-VolumesExtendTest-93540250 None] get snapshots for volume v_31d34366-a0ee-4a52-822b-79de201b5a12 {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 15:25:02 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-25c80733-a960-453f-9159-ec587e0787bf tempest-VolumesExtendTest-93540250 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_31d34366-a0ee-4a52-822b-79de201b5a12/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:25:02 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-25c80733-a960-453f-9159-ec587e0787bf tempest-VolumesExtendTest-93540250 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_31d34366-a0ee-4a52-822b-79de201b5a12/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:25:02 user cinder-volume[84696]: DEBUG cinder.coordination [None req-14897f06-95d2-4866-b6a0-c80278ef170a tempest-VolumesGetTest-1961901732 None] Acquiring lock "cinder-3982de76-ea42-41a0-b8e5-4b24b489009b-delete_volume" by "delete_volume" {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:192}} Apr 17 15:25:02 user cinder-volume[84696]: DEBUG cinder.coordination [None req-14897f06-95d2-4866-b6a0-c80278ef170a tempest-VolumesGetTest-1961901732 None] Lock "cinder-3982de76-ea42-41a0-b8e5-4b24b489009b-delete_volume" acquired by "delete_volume" :: waited 0.020s {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:196}} Apr 17 15:25:02 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5ddb6e37-cf4a-4825-831b-16072286044b tempest-VolumesGetTest-1961901732 None] Geting {'code': 200, 'error': None, 'data': {'available': '113909825536', 'status': 30, 'import_status': {'import_steps': {'san_setup': True, 'vip_setup': True, 'zfs_import': True, 'mount_system_volume': True, 'nas_setup': True}, 'import_successful': True}, 'scan': None, 'encryption': {'enabled': False}, 'deduplication_rate': '1.00', 'sysvol_upgrade': 'not_needed', 'vdevs': [{'name': 'wwn-0x6000c296b793bf526a107a9d823e947a', 'iostats': {'read': '0', 'write': '0', 'chksum': '0'}, 'disks': [{'origin': 'local', 'slot': '', 'led': 'off', 'name': 'sdn', 'iostats': {'read': '0', 'write': '0', 'chksum': '0'}, 'alias': '', 'health': 'ONLINE', 'sn': '6000c296b793bf526a107a9d823e947a', 'trim_data': {'status': 'trim_unsupported', 'progress': None, 'trim_time': None}, 'path': 'pci-0000:03:00.0-scsi-0:0:14:0', 'model': 'Virtual disk', 'id': 'wwn-0x6000c296b793bf526a107a9d823e947a', 'size': 128849018880}], 'health': 'ONLINE', 'vdev_replacings': [], 'vdev_spares': [], 'type': ''}], 'id': '14181317334440137330', 'health': 'ONLINE', 'iostats': {'read': '0', 'write': '0', 'chksum': '0'}, 'operation': 'none', 'size': '127775277056', 'autotrim': False, 'name': 'Pool-0'}} from GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:25:02 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-5ddb6e37-cf4a-4825-831b-16072286044b tempest-VolumesGetTest-1961901732 None] Total capacity: 119, Free 106. {{(pid=84696) _update_volume_stats /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:757}} Apr 17 15:25:02 user cinder-volume[84696]: DEBUG cinder.manager [None req-5ddb6e37-cf4a-4825-831b-16072286044b tempest-VolumesGetTest-1961901732 None] Notifying Schedulers of capabilities ... {{(pid=84696) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 17 15:25:02 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-14897f06-95d2-4866-b6a0-c80278ef170a tempest-VolumesGetTest-1961901732 None] remove_export for volume: 3982de76-ea42-41a0-b8e5-4b24b489009b. {{(pid=84696) remove_export /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:718}} Apr 17 15:25:02 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-14897f06-95d2-4866-b6a0-c80278ef170a tempest-VolumesGetTest-1961901732 None] remove export {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:912}} Apr 17 15:25:02 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-14897f06-95d2-4866-b6a0-c80278ef170a tempest-VolumesGetTest-1961901732 None] detach volume:Volume(_name_id=None,admin_metadata={},attach_status='detached',availability_zone='nova',bootable=True,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:24:50Z,deleted=False,deleted_at=None,display_description='This is the new description of volume',display_name='tempest-VolumesGetTest-new-Volume-885589958',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=3982de76-ea42-41a0-b8e5-4b24b489009b,launched_at=2023-04-17T15:24:54Z,metadata={Type='Test'},migration_status=None,multiattach=False,previous_status=None,project_id='a65d14b0a2894a6f90506d86feaefd15',provider_auth='CHAP nXQCotUf zMX3N4j1mvPiRT',provider_geometry=None,provider_id=None,provider_location='172.16.0.220:3260,1 iqn.2010-10.org.openstack:3982de76-ea42-41a0-b8e5-4b24b489009b 0',replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:24:51Z,service_uuid='e628352e-c4cf-40eb-8389-cce40663026a',shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='deleting',terminated_at=2023-04-17T15:25:02Z,updated_at=2023-04-17T15:25:02Z,use_quota=True,user_id='6ffa3051570a4721a48a0171b3cc4d16',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c) from target:iqn.2010-10.org.openstack:3982de76-ea42-41a0-b8e5-4b24b489009b. {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:913}} Apr 17 15:25:02 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-14897f06-95d2-4866-b6a0-c80278ef170a tempest-VolumesGetTest-1961901732 None] detach volume v_3982de76-ea42-41a0-b8e5-4b24b489009b from target iqn.2010-10.org.openstack:3982de76-ea42-41a0-b8e5-4b24b489009b {{(pid=84696) detach_target_vol /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:606}} Apr 17 15:25:02 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-14897f06-95d2-4866-b6a0-c80278ef170a tempest-VolumesGetTest-1961901732 None] Sending pool request DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:3982de76-ea42-41a0-b8e5-4b24b489009b/luns/v_3982de76-ea42-41a0-b8e5-4b24b489009b {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:25:02 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-14897f06-95d2-4866-b6a0-c80278ef170a tempest-VolumesGetTest-1961901732 None] Sending DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:3982de76-ea42-41a0-b8e5-4b24b489009b/luns/v_3982de76-ea42-41a0-b8e5-4b24b489009b {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:25:02 user cinder-volume[84696]: INFO cinder.volume.manager [None req-5ddb6e37-cf4a-4825-831b-16072286044b tempest-VolumesGetTest-1961901732 None] Deleted volume successfully. Apr 17 15:25:02 user cinder-volume[84696]: DEBUG cinder.coordination [None req-5ddb6e37-cf4a-4825-831b-16072286044b tempest-VolumesGetTest-1961901732 None] Lock "cinder-6e83844e-db12-47e9-b1b9-acd2e0f7d31a-delete_volume" released by "delete_volume" :: held 4.463s {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:207}} Apr 17 15:25:02 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-14897f06-95d2-4866-b6a0-c80278ef170a tempest-VolumesGetTest-1961901732 None] Geting {'code': 404, 'error': {'class': 'opene.exceptions.ItemNotFoundError', 'message': 'Target iqn.2010-10.org.openstack:3982de76-ea42-41a0-b8e5-4b24b489009b not exists.', 'url': 'http://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:3982de76-ea42-41a0-b8e5-4b24b489009b/luns/v_3982de76-ea42-41a0-b8e5-4b24b489009b'}, 'data': None} from DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:3982de76-ea42-41a0-b8e5-4b24b489009b/luns/v_3982de76-ea42-41a0-b8e5-4b24b489009b {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:25:02 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-14897f06-95d2-4866-b6a0-c80278ef170a tempest-VolumesGetTest-1961901732 None] failed to remove resource iqn.2010-10.org.openstack:3982de76-ea42-41a0-b8e5-4b24b489009b because of JDSS resource v_3982de76-ea42-41a0-b8e5-4b24b489009b DNE. {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:920}} Apr 17 15:25:02 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-14897f06-95d2-4866-b6a0-c80278ef170a tempest-VolumesGetTest-1961901732 None] delete target: iqn.2010-10.org.openstack:3982de76-ea42-41a0-b8e5-4b24b489009b {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:930}} Apr 17 15:25:02 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-14897f06-95d2-4866-b6a0-c80278ef170a tempest-VolumesGetTest-1961901732 None] delete target iqn.2010-10.org.openstack:3982de76-ea42-41a0-b8e5-4b24b489009b {{(pid=84696) delete_target /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:442}} Apr 17 15:25:02 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-14897f06-95d2-4866-b6a0-c80278ef170a tempest-VolumesGetTest-1961901732 None] Sending pool request DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:3982de76-ea42-41a0-b8e5-4b24b489009b {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:25:02 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-14897f06-95d2-4866-b6a0-c80278ef170a tempest-VolumesGetTest-1961901732 None] Sending DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:3982de76-ea42-41a0-b8e5-4b24b489009b {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:25:02 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-14897f06-95d2-4866-b6a0-c80278ef170a tempest-VolumesGetTest-1961901732 None] Geting {'code': 404, 'error': {'class': 'opene.exceptions.ItemNotFoundError', 'message': 'Target iqn.2010-10.org.openstack:3982de76-ea42-41a0-b8e5-4b24b489009b not exists.', 'url': 'http://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:3982de76-ea42-41a0-b8e5-4b24b489009b'}, 'data': None} from DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:3982de76-ea42-41a0-b8e5-4b24b489009b {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:25:02 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-14897f06-95d2-4866-b6a0-c80278ef170a tempest-VolumesGetTest-1961901732 None] failed to remove resource iqn.2010-10.org.openstack:3982de76-ea42-41a0-b8e5-4b24b489009b because of JDSS resource iqn.2010-10.org.openstack:3982de76-ea42-41a0-b8e5-4b24b489009b DNE. {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:935}} Apr 17 15:25:02 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-14897f06-95d2-4866-b6a0-c80278ef170a tempest-VolumesGetTest-1961901732 None] deleating volume v_3982de76-ea42-41a0-b8e5-4b24b489009b {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 15:25:02 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-14897f06-95d2-4866-b6a0-c80278ef170a tempest-VolumesGetTest-1961901732 None] get snapshots for volume v_3982de76-ea42-41a0-b8e5-4b24b489009b {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 15:25:02 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-14897f06-95d2-4866-b6a0-c80278ef170a tempest-VolumesGetTest-1961901732 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_3982de76-ea42-41a0-b8e5-4b24b489009b/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:25:02 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-14897f06-95d2-4866-b6a0-c80278ef170a tempest-VolumesGetTest-1961901732 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_3982de76-ea42-41a0-b8e5-4b24b489009b/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:25:02 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-e901a82d-2ea1-4b10-a528-c1880ebdb240 tempest-VolumeTypesTest-1824483253 None] Geting {'code': 201, 'error': None, 'data': {'full_name': 'Pool-0/v_29534406-54ef-4f00-9568-69ae248eae21', 'is_clone': False, 'name': 'v_29534406-54ef-4f00-9568-69ae248eae21'}} from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:25:02 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e901a82d-2ea1-4b10-a528-c1880ebdb240 tempest-VolumeTypesTest-1824483253 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (59d93612-dc48-4e87-b44a-274ce009ae60) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'volume_name': 'volume-29534406-54ef-4f00-9568-69ae248eae21', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:25:02 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e901a82d-2ea1-4b10-a528-c1880ebdb240 tempest-VolumeTypesTest-1824483253 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeOnFinishTask;volume:create, create.end' (c7d473be-5f18-4f58-82be-e5733a03a4f6) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:25:02 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-e901a82d-2ea1-4b10-a528-c1880ebdb240 tempest-VolumeTypesTest-1824483253 None] Volume volume-29534406-54ef-4f00-9568-69ae248eae21 (29534406-54ef-4f00-9568-69ae248eae21): created successfully Apr 17 15:25:02 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e901a82d-2ea1-4b10-a528-c1880ebdb240 tempest-VolumeTypesTest-1824483253 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeOnFinishTask;volume:create, create.end' (c7d473be-5f18-4f58-82be-e5733a03a4f6) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:25:02 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e901a82d-2ea1-4b10-a528-c1880ebdb240 tempest-VolumeTypesTest-1824483253 None] Flow 'volume_create_manager' (f751a11d-b614-4fd7-a7e3-b2057694e5ab) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:25:02 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-25c80733-a960-453f-9159-ec587e0787bf tempest-VolumesExtendTest-93540250 None] Geting {'code': 200, 'error': None, 'data': {'results': 0, 'entries': []}} from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_31d34366-a0ee-4a52-822b-79de201b5a12/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:25:02 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-25c80733-a960-453f-9159-ec587e0787bf tempest-VolumesExtendTest-93540250 None] get volume v_31d34366-a0ee-4a52-822b-79de201b5a12 info {{(pid=84696) get_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:239}} Apr 17 15:25:02 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-25c80733-a960-453f-9159-ec587e0787bf tempest-VolumesExtendTest-93540250 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_31d34366-a0ee-4a52-822b-79de201b5a12 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:25:02 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-25c80733-a960-453f-9159-ec587e0787bf tempest-VolumesExtendTest-93540250 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_31d34366-a0ee-4a52-822b-79de201b5a12 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:25:02 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-25c80733-a960-453f-9159-ec587e0787bf tempest-VolumesExtendTest-93540250 None] Geting {'code': 200, 'error': None, 'data': {'origin': None, 'relatime': None, 'acltype': None, 'vscan': None, 'full_name': 'Pool-0/v_31d34366-a0ee-4a52-822b-79de201b5a12', 'userrefs': None, 'primarycache': 'all', 'logbias': 'latency', 'creation': '1681746486', 'sync': 'always', 'is_clone': False, 'dedup': 'off', 'sharenfs': None, 'receive_resume_token': None, 'volsize': '2147483648', 'referenced': '57344', 'sharesmb': None, 'createtxg': '187982', 'reservation': '0', 'scontext': None, 'mountpoint': None, 'casesensitivity': None, 'guid': '940531860148082616', 'usedbyrefreservation': '0', 'dnodesize': None, 'written': '57344', 'logicalused': '28672', 'compressratio': '1.00', 'rootcontext': 'none', 'default_scsi_id': 'b74f05b14a04a80a', 'type': 'volume', 'compression': 'lz4', 'snapdir': None, 'overlay': None, 'encryption': 'off', 'xattr': None, 'volmode': 'default', 'copies': '1', 'snapshot_limit': '18446744073709551615', 'aclinherit': None, 'defcontext': 'none', 'readonly': 'off', 'version': None, 'recordsize': None, 'filesystem_limit': None, 'mounted': None, 'mlslabel': 'none', 'secondarycache': 'all', 'refreservation': '0', 'available': '113909743616', 'san:volume_id': 'b74f05b14a04a80a3ebf60b775cd3fc28491c0df', 'encryptionroot': None, 'exec': None, 'refquota': None, 'refcompressratio': '1.00', 'quota': None, 'utf8only': None, 'keylocation': 'none', 'snapdev': 'hidden', 'snapshot_count': '18446744073709551615', 'fscontext': 'none', 'clones': None, 'canmount': None, 'keystatus': None, 'atime': None, 'usedbysnapshots': '0', 'normalization': None, 'usedbychildren': '0', 'volblocksize': '131072', 'usedbydataset': '57344', 'objsetid': '121279', 'name': 'v_31d34366-a0ee-4a52-822b-79de201b5a12', 'defer_destroy': None, 'pbkdf2iters': '0', 'checksum': 'on', 'redundant_metadata': 'all', 'filesystem_count': None, 'devices': None, 'keyformat': 'none', 'setuid': None, 'used': '57344', 'logicalreferenced': '28672', 'context': 'none', 'zoned': None, 'nbmand': None}} from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_31d34366-a0ee-4a52-822b-79de201b5a12 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:25:02 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-25c80733-a960-453f-9159-ec587e0787bf tempest-VolumesExtendTest-93540250 None] delete volume:v_31d34366-a0ee-4a52-822b-79de201b5a12 recursively children:{'force_umount': True} {{(pid=84696) delete_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:335}} Apr 17 15:25:02 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-25c80733-a960-453f-9159-ec587e0787bf tempest-VolumesExtendTest-93540250 None] Sending pool request DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_31d34366-a0ee-4a52-822b-79de201b5a12 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:25:02 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-25c80733-a960-453f-9159-ec587e0787bf tempest-VolumesExtendTest-93540250 None] Sending DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_31d34366-a0ee-4a52-822b-79de201b5a12 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:25:02 user cinder-volume[84696]: INFO cinder.volume.manager [None req-e901a82d-2ea1-4b10-a528-c1880ebdb240 tempest-VolumeTypesTest-1824483253 None] Created volume successfully. Apr 17 15:25:03 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-120c9c51-a7c0-483f-b2de-e5ce22a20a39 req-6d09bc69-27ce-4ab0-b688-f3045639f36b tempest-TaggedBootDevicesTest_v242-1317742003 None] Geting {'code': 201, 'error': None, 'data': {'name': 'PUvXoCNO'}} from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:c74b7878-4936-4015-8b63-142923947310/incoming-users {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:25:03 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-120c9c51-a7c0-483f-b2de-e5ce22a20a39 req-6d09bc69-27ce-4ab0-b688-f3045639f36b tempest-TaggedBootDevicesTest_v242-1317742003 None] initialize_connection for c74b7878-4936-4015-8b63-142923947310 10.0.0.210. {{(pid=84696) initialize_connection /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:1018}} Apr 17 15:25:03 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-120c9c51-a7c0-483f-b2de-e5ce22a20a39 req-6d09bc69-27ce-4ab0-b688-f3045639f36b tempest-TaggedBootDevicesTest_v242-1317742003 None] Connection info returned from driver {'target_iqn': 'iqn.2010-10.org.openstack:c74b7878-4936-4015-8b63-142923947310', 'target_portal': '172.16.0.220:3260', 'target_discovered': False, 'auth_method': 'CHAP', 'auth_username': 'PUvXoCNO', 'auth_password': '***', 'target_lun': 0, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False, 'driver_volume_type': 'iscsi', 'attachment_id': 'e51e7201-04a3-43d8-b3a0-c7467cb02c04'} {{(pid=84696) _connection_create /opt/stack/cinder/cinder/volume/manager.py:4875}} Apr 17 15:25:03 user cinder-volume[84696]: INFO cinder.volume.manager [req-120c9c51-a7c0-483f-b2de-e5ce22a20a39 req-6d09bc69-27ce-4ab0-b688-f3045639f36b tempest-TaggedBootDevicesTest_v242-1317742003 None] attachment_update completed successfully. Apr 17 15:25:03 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-14897f06-95d2-4866-b6a0-c80278ef170a tempest-VolumesGetTest-1961901732 None] Geting {'code': 200, 'error': None, 'data': {'results': 0, 'entries': []}} from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_3982de76-ea42-41a0-b8e5-4b24b489009b/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:25:03 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-14897f06-95d2-4866-b6a0-c80278ef170a tempest-VolumesGetTest-1961901732 None] get volume v_3982de76-ea42-41a0-b8e5-4b24b489009b info {{(pid=84696) get_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:239}} Apr 17 15:25:03 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-14897f06-95d2-4866-b6a0-c80278ef170a tempest-VolumesGetTest-1961901732 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_3982de76-ea42-41a0-b8e5-4b24b489009b {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:25:03 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-14897f06-95d2-4866-b6a0-c80278ef170a tempest-VolumesGetTest-1961901732 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_3982de76-ea42-41a0-b8e5-4b24b489009b {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:25:03 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-14897f06-95d2-4866-b6a0-c80278ef170a tempest-VolumesGetTest-1961901732 None] Geting {'code': 200, 'error': None, 'data': {'origin': 'Pool-0/v_006f20ae-5306-44fe-afdf-e433f79284e8@v_3982de76-ea42-41a0-b8e5-4b24b489009b', 'relatime': None, 'acltype': None, 'vscan': None, 'full_name': 'Pool-0/v_3982de76-ea42-41a0-b8e5-4b24b489009b', 'userrefs': None, 'primarycache': 'all', 'logbias': 'latency', 'creation': '1681746486', 'sync': 'always', 'is_clone': True, 'dedup': 'off', 'sharenfs': None, 'receive_resume_token': None, 'volsize': '1073741824', 'referenced': '16285696', 'sharesmb': None, 'createtxg': '187985', 'reservation': '0', 'scontext': None, 'mountpoint': None, 'casesensitivity': None, 'guid': '4548803350227175746', 'usedbyrefreservation': '0', 'dnodesize': None, 'written': '0', 'logicalused': '0', 'compressratio': '1.00', 'rootcontext': 'none', 'default_scsi_id': '9906613680b42815', 'type': 'volume', 'compression': 'lz4', 'snapdir': None, 'overlay': None, 'encryption': 'off', 'xattr': None, 'volmode': 'default', 'copies': '1', 'snapshot_limit': '18446744073709551615', 'aclinherit': None, 'defcontext': 'none', 'readonly': 'off', 'version': None, 'recordsize': None, 'filesystem_limit': None, 'mounted': None, 'mlslabel': 'none', 'secondarycache': 'all', 'refreservation': '0', 'available': '113909743616', 'san:volume_id': '9906613680b42815e915f267d691e644279e95bd', 'encryptionroot': None, 'exec': None, 'refquota': None, 'refcompressratio': '1.17', 'quota': None, 'utf8only': None, 'keylocation': 'none', 'snapdev': 'hidden', 'snapshot_count': '18446744073709551615', 'fscontext': 'none', 'clones': None, 'canmount': None, 'keystatus': None, 'atime': None, 'usedbysnapshots': '0', 'normalization': None, 'usedbychildren': '0', 'volblocksize': '131072', 'usedbydataset': '0', 'objsetid': '121287', 'name': 'v_3982de76-ea42-41a0-b8e5-4b24b489009b', 'defer_destroy': None, 'pbkdf2iters': '0', 'checksum': 'on', 'redundant_metadata': 'all', 'filesystem_count': None, 'devices': None, 'keyformat': 'none', 'setuid': None, 'used': '0', 'logicalreferenced': '19042304', 'context': 'none', 'zoned': None, 'nbmand': None}} from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_3982de76-ea42-41a0-b8e5-4b24b489009b {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:25:03 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-14897f06-95d2-4866-b6a0-c80278ef170a tempest-VolumesGetTest-1961901732 None] delete snapshot v_3982de76-ea42-41a0-b8e5-4b24b489009b of volume v_006f20ae-5306-44fe-afdf-e433f79284e8 {{(pid=84696) delete_snapshot /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:751}} Apr 17 15:25:03 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-14897f06-95d2-4866-b6a0-c80278ef170a tempest-VolumesGetTest-1961901732 None] Sending pool request DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots/v_3982de76-ea42-41a0-b8e5-4b24b489009b {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:25:03 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-14897f06-95d2-4866-b6a0-c80278ef170a tempest-VolumesGetTest-1961901732 None] Sending DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots/v_3982de76-ea42-41a0-b8e5-4b24b489009b {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:25:03 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-120c9c51-a7c0-483f-b2de-e5ce22a20a39 req-34fc5997-2964-49b3-b98a-4c267b80eb23 tempest-TaggedBootDevicesTest_v242-1317742003 None] create export for volume: 18475864-c202-4937-b18f-cc3e16c6d89f. {{(pid=84696) create_export /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:699}} Apr 17 15:25:03 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-120c9c51-a7c0-483f-b2de-e5ce22a20a39 req-34fc5997-2964-49b3-b98a-4c267b80eb23 tempest-TaggedBootDevicesTest_v242-1317742003 None] ensure volume 18475864-c202-4937-b18f-cc3e16c6d89f assigned to a proper target {{(pid=84696) _ensure_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:866}} Apr 17 15:25:03 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [req-120c9c51-a7c0-483f-b2de-e5ce22a20a39 req-34fc5997-2964-49b3-b98a-4c267b80eb23 tempest-TaggedBootDevicesTest_v242-1317742003 None] check if targe iqn.2010-10.org.openstack:18475864-c202-4937-b18f-cc3e16c6d89f exists {{(pid=84696) is_target /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:376}} Apr 17 15:25:03 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-120c9c51-a7c0-483f-b2de-e5ce22a20a39 req-34fc5997-2964-49b3-b98a-4c267b80eb23 tempest-TaggedBootDevicesTest_v242-1317742003 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:18475864-c202-4937-b18f-cc3e16c6d89f {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:25:03 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-120c9c51-a7c0-483f-b2de-e5ce22a20a39 req-34fc5997-2964-49b3-b98a-4c267b80eb23 tempest-TaggedBootDevicesTest_v242-1317742003 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:18475864-c202-4937-b18f-cc3e16c6d89f {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:25:03 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-120c9c51-a7c0-483f-b2de-e5ce22a20a39 req-34fc5997-2964-49b3-b98a-4c267b80eb23 tempest-TaggedBootDevicesTest_v242-1317742003 None] Geting {'code': 404, 'error': {'class': 'opene.exceptions.ItemNotFoundError', 'message': 'Target iqn.2010-10.org.openstack:18475864-c202-4937-b18f-cc3e16c6d89f not exists.', 'url': 'http://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:18475864-c202-4937-b18f-cc3e16c6d89f'}, 'data': None} from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:18475864-c202-4937-b18f-cc3e16c6d89f {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:25:03 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-120c9c51-a7c0-483f-b2de-e5ce22a20a39 req-34fc5997-2964-49b3-b98a-4c267b80eb23 tempest-TaggedBootDevicesTest_v242-1317742003 None] create target and attach volume 18475864-c202-4937-b18f-cc3e16c6d89f to it {{(pid=84696) _create_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:837}} Apr 17 15:25:03 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [req-120c9c51-a7c0-483f-b2de-e5ce22a20a39 req-34fc5997-2964-49b3-b98a-4c267b80eb23 tempest-TaggedBootDevicesTest_v242-1317742003 None] create target iqn.2010-10.org.openstack:18475864-c202-4937-b18f-cc3e16c6d89f {{(pid=84696) create_target /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:410}} Apr 17 15:25:03 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-120c9c51-a7c0-483f-b2de-e5ce22a20a39 req-34fc5997-2964-49b3-b98a-4c267b80eb23 tempest-TaggedBootDevicesTest_v242-1317742003 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:25:03 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-120c9c51-a7c0-483f-b2de-e5ce22a20a39 req-34fc5997-2964-49b3-b98a-4c267b80eb23 tempest-TaggedBootDevicesTest_v242-1317742003 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:25:04 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-56e853c8-d1bd-490d-9734-edcd1d06128c tempest-VolumeTypesTest-1824483253 None] Updating volume stats {{(pid=84696) _update_volume_stats /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:724}} Apr 17 15:25:04 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-56e853c8-d1bd-490d-9734-edcd1d06128c tempest-VolumeTypesTest-1824483253 None] Get pool Pool-0 fsprops {{(pid=84696) get_pool_stats /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:894}} Apr 17 15:25:04 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-56e853c8-d1bd-490d-9734-edcd1d06128c tempest-VolumeTypesTest-1824483253 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:25:04 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-56e853c8-d1bd-490d-9734-edcd1d06128c tempest-VolumeTypesTest-1824483253 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:25:05 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-25c80733-a960-453f-9159-ec587e0787bf tempest-VolumesExtendTest-93540250 None] Geting {'code': 204} from DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_31d34366-a0ee-4a52-822b-79de201b5a12 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:25:05 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-25c80733-a960-453f-9159-ec587e0787bf tempest-VolumesExtendTest-93540250 None] volume v_31d34366-a0ee-4a52-822b-79de201b5a12 deleted {{(pid=84696) delete_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:346}} Apr 17 15:25:05 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=69,cinder:UPDATE=26,cinder:DELETE=3,cinder:INSERT=2 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:25:05 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-14897f06-95d2-4866-b6a0-c80278ef170a tempest-VolumesGetTest-1961901732 None] Geting {'code': 204} from DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots/v_3982de76-ea42-41a0-b8e5-4b24b489009b {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:25:05 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-14897f06-95d2-4866-b6a0-c80278ef170a tempest-VolumesGetTest-1961901732 None] snapshot v_3982de76-ea42-41a0-b8e5-4b24b489009b deleted {{(pid=84696) delete_snapshot /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:772}} Apr 17 15:25:05 user cinder-volume[84696]: DEBUG cinder.quota [None req-25c80733-a960-453f-9159-ec587e0787bf tempest-VolumesExtendTest-93540250 None] Created reservations ['ff005e1f-b897-4724-82ee-b8bd26b864bc', 'd24e36ec-ed21-4515-9059-a05dcca08f96', '1d900cf6-f169-40e1-a876-5b44b6ada8e7', '6b39c5f0-ef3a-46d7-a89c-eb5531ab63d4'] {{(pid=84696) reserve /opt/stack/cinder/cinder/quota.py:755}} Apr 17 15:25:05 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:UPDATE=27,cinder:SELECT=65,cinder:INSERT=7,cinder:DELETE=1 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:25:05 user cinder-volume[84696]: DEBUG cinder.quota [None req-14897f06-95d2-4866-b6a0-c80278ef170a tempest-VolumesGetTest-1961901732 None] Created reservations ['cfefc5b8-ccd1-4416-a7d9-8baf6e2819ce', '0a3e25b4-9c7b-4d02-91a8-c722893cf273', '0dc2c83d-f0e9-491a-92d8-20c7b412b435', '797c786d-ed8d-4e73-9629-1f8d4ec2ca69'] {{(pid=84696) reserve /opt/stack/cinder/cinder/quota.py:755}} Apr 17 15:25:05 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-25c80733-a960-453f-9159-ec587e0787bf tempest-VolumesExtendTest-93540250 None] Updating volume stats {{(pid=84696) _update_volume_stats /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:724}} Apr 17 15:25:05 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-25c80733-a960-453f-9159-ec587e0787bf tempest-VolumesExtendTest-93540250 None] Get pool Pool-0 fsprops {{(pid=84696) get_pool_stats /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:894}} Apr 17 15:25:05 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-25c80733-a960-453f-9159-ec587e0787bf tempest-VolumesExtendTest-93540250 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:25:05 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-25c80733-a960-453f-9159-ec587e0787bf tempest-VolumesExtendTest-93540250 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:25:05 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-14897f06-95d2-4866-b6a0-c80278ef170a tempest-VolumesGetTest-1961901732 None] Updating volume stats {{(pid=84696) _update_volume_stats /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:724}} Apr 17 15:25:05 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-14897f06-95d2-4866-b6a0-c80278ef170a tempest-VolumesGetTest-1961901732 None] Get pool Pool-0 fsprops {{(pid=84696) get_pool_stats /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:894}} Apr 17 15:25:05 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-14897f06-95d2-4866-b6a0-c80278ef170a tempest-VolumesGetTest-1961901732 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:25:05 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-14897f06-95d2-4866-b6a0-c80278ef170a tempest-VolumesGetTest-1961901732 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:25:05 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-14897f06-95d2-4866-b6a0-c80278ef170a tempest-VolumesGetTest-1961901732 None] Geting {'code': 200, 'error': None, 'data': {'available': '113909825536', 'status': 30, 'import_status': {'import_steps': {'san_setup': True, 'vip_setup': True, 'zfs_import': True, 'mount_system_volume': True, 'nas_setup': True}, 'import_successful': True}, 'scan': None, 'encryption': {'enabled': False}, 'deduplication_rate': '1.00', 'sysvol_upgrade': 'not_needed', 'vdevs': [{'name': 'wwn-0x6000c296b793bf526a107a9d823e947a', 'iostats': {'read': '0', 'write': '0', 'chksum': '0'}, 'disks': [{'origin': 'local', 'slot': '', 'led': 'off', 'name': 'sdn', 'iostats': {'read': '0', 'write': '0', 'chksum': '0'}, 'alias': '', 'health': 'ONLINE', 'sn': '6000c296b793bf526a107a9d823e947a', 'trim_data': {'status': 'trim_unsupported', 'progress': None, 'trim_time': None}, 'path': 'pci-0000:03:00.0-scsi-0:0:14:0', 'model': 'Virtual disk', 'id': 'wwn-0x6000c296b793bf526a107a9d823e947a', 'size': 128849018880}], 'health': 'ONLINE', 'vdev_replacings': [], 'vdev_spares': [], 'type': ''}], 'id': '14181317334440137330', 'health': 'ONLINE', 'iostats': {'read': '0', 'write': '0', 'chksum': '0'}, 'operation': 'none', 'size': '127775277056', 'autotrim': False, 'name': 'Pool-0'}} from GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:25:05 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-14897f06-95d2-4866-b6a0-c80278ef170a tempest-VolumesGetTest-1961901732 None] Total capacity: 119, Free 106. {{(pid=84696) _update_volume_stats /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:757}} Apr 17 15:25:05 user cinder-volume[84696]: DEBUG cinder.manager [None req-14897f06-95d2-4866-b6a0-c80278ef170a tempest-VolumesGetTest-1961901732 None] Notifying Schedulers of capabilities ... {{(pid=84696) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 17 15:25:05 user cinder-volume[84696]: INFO cinder.volume.manager [None req-14897f06-95d2-4866-b6a0-c80278ef170a tempest-VolumesGetTest-1961901732 None] Deleted volume successfully. Apr 17 15:25:05 user cinder-volume[84696]: DEBUG cinder.coordination [None req-14897f06-95d2-4866-b6a0-c80278ef170a tempest-VolumesGetTest-1961901732 None] Lock "cinder-3982de76-ea42-41a0-b8e5-4b24b489009b-delete_volume" released by "delete_volume" :: held 3.417s {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:207}} Apr 17 15:25:05 user cinder-volume[84696]: DEBUG cinder.coordination [None req-f4767ed7-e4fa-4c0c-bd0b-b48a0034e6c0 tempest-VolumeTypesTest-1824483253 None] Acquiring lock "cinder-29534406-54ef-4f00-9568-69ae248eae21-delete_volume" by "delete_volume" {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:192}} Apr 17 15:25:05 user cinder-volume[84696]: DEBUG cinder.coordination [None req-f4767ed7-e4fa-4c0c-bd0b-b48a0034e6c0 tempest-VolumeTypesTest-1824483253 None] Lock "cinder-29534406-54ef-4f00-9568-69ae248eae21-delete_volume" acquired by "delete_volume" :: waited 0.011s {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:196}} Apr 17 15:25:05 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-f4767ed7-e4fa-4c0c-bd0b-b48a0034e6c0 tempest-VolumeTypesTest-1824483253 None] remove_export for volume: 29534406-54ef-4f00-9568-69ae248eae21. {{(pid=84696) remove_export /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:718}} Apr 17 15:25:05 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-f4767ed7-e4fa-4c0c-bd0b-b48a0034e6c0 tempest-VolumeTypesTest-1824483253 None] remove export {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:912}} Apr 17 15:25:05 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-f4767ed7-e4fa-4c0c-bd0b-b48a0034e6c0 tempest-VolumeTypesTest-1824483253 None] detach volume:Volume(_name_id=None,admin_metadata={},attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:25:00Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumeTypesTest-volume-2002497870',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=29534406-54ef-4f00-9568-69ae248eae21,launched_at=2023-04-17T15:25:03Z,metadata={},migration_status=None,multiattach=False,previous_status='available',project_id='1741e85eaa4b420cabe83f57ee957bb7',provider_auth='CHAP IiUnpCTr BHVRQXlgCrY7M4',provider_geometry=None,provider_id=None,provider_location='172.16.0.220:3260,1 iqn.2010-10.org.openstack:29534406-54ef-4f00-9568-69ae248eae21 0',replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:25:00Z,service_uuid='e628352e-c4cf-40eb-8389-cce40663026a',shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='deleting',terminated_at=2023-04-17T15:25:06Z,updated_at=2023-04-17T15:25:06Z,use_quota=True,user_id='685e853373e84057b2eb234a0815d817',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(4c6a6a14-19aa-4e5e-90a7-af70e8958929),volume_type_id=4c6a6a14-19aa-4e5e-90a7-af70e8958929) from target:iqn.2010-10.org.openstack:29534406-54ef-4f00-9568-69ae248eae21. {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:913}} Apr 17 15:25:05 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-f4767ed7-e4fa-4c0c-bd0b-b48a0034e6c0 tempest-VolumeTypesTest-1824483253 None] detach volume v_29534406-54ef-4f00-9568-69ae248eae21 from target iqn.2010-10.org.openstack:29534406-54ef-4f00-9568-69ae248eae21 {{(pid=84696) detach_target_vol /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:606}} Apr 17 15:25:05 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-f4767ed7-e4fa-4c0c-bd0b-b48a0034e6c0 tempest-VolumeTypesTest-1824483253 None] Sending pool request DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:29534406-54ef-4f00-9568-69ae248eae21/luns/v_29534406-54ef-4f00-9568-69ae248eae21 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:25:05 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-f4767ed7-e4fa-4c0c-bd0b-b48a0034e6c0 tempest-VolumeTypesTest-1824483253 None] Sending DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:29534406-54ef-4f00-9568-69ae248eae21/luns/v_29534406-54ef-4f00-9568-69ae248eae21 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:25:05 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-f4767ed7-e4fa-4c0c-bd0b-b48a0034e6c0 tempest-VolumeTypesTest-1824483253 None] Geting {'code': 404, 'error': {'class': 'opene.exceptions.ItemNotFoundError', 'message': 'Target iqn.2010-10.org.openstack:29534406-54ef-4f00-9568-69ae248eae21 not exists.', 'url': 'http://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:29534406-54ef-4f00-9568-69ae248eae21/luns/v_29534406-54ef-4f00-9568-69ae248eae21'}, 'data': None} from DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:29534406-54ef-4f00-9568-69ae248eae21/luns/v_29534406-54ef-4f00-9568-69ae248eae21 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:25:05 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-f4767ed7-e4fa-4c0c-bd0b-b48a0034e6c0 tempest-VolumeTypesTest-1824483253 None] failed to remove resource iqn.2010-10.org.openstack:29534406-54ef-4f00-9568-69ae248eae21 because of JDSS resource v_29534406-54ef-4f00-9568-69ae248eae21 DNE. {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:920}} Apr 17 15:25:05 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-f4767ed7-e4fa-4c0c-bd0b-b48a0034e6c0 tempest-VolumeTypesTest-1824483253 None] delete target: iqn.2010-10.org.openstack:29534406-54ef-4f00-9568-69ae248eae21 {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:930}} Apr 17 15:25:05 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-f4767ed7-e4fa-4c0c-bd0b-b48a0034e6c0 tempest-VolumeTypesTest-1824483253 None] delete target iqn.2010-10.org.openstack:29534406-54ef-4f00-9568-69ae248eae21 {{(pid=84696) delete_target /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:442}} Apr 17 15:25:05 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-f4767ed7-e4fa-4c0c-bd0b-b48a0034e6c0 tempest-VolumeTypesTest-1824483253 None] Sending pool request DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:29534406-54ef-4f00-9568-69ae248eae21 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:25:05 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-f4767ed7-e4fa-4c0c-bd0b-b48a0034e6c0 tempest-VolumeTypesTest-1824483253 None] Sending DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:29534406-54ef-4f00-9568-69ae248eae21 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:25:05 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-f4767ed7-e4fa-4c0c-bd0b-b48a0034e6c0 tempest-VolumeTypesTest-1824483253 None] Geting {'code': 404, 'error': {'class': 'opene.exceptions.ItemNotFoundError', 'message': 'Target iqn.2010-10.org.openstack:29534406-54ef-4f00-9568-69ae248eae21 not exists.', 'url': 'http://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:29534406-54ef-4f00-9568-69ae248eae21'}, 'data': None} from DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:29534406-54ef-4f00-9568-69ae248eae21 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:25:05 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-f4767ed7-e4fa-4c0c-bd0b-b48a0034e6c0 tempest-VolumeTypesTest-1824483253 None] failed to remove resource iqn.2010-10.org.openstack:29534406-54ef-4f00-9568-69ae248eae21 because of JDSS resource iqn.2010-10.org.openstack:29534406-54ef-4f00-9568-69ae248eae21 DNE. {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:935}} Apr 17 15:25:05 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-f4767ed7-e4fa-4c0c-bd0b-b48a0034e6c0 tempest-VolumeTypesTest-1824483253 None] deleating volume v_29534406-54ef-4f00-9568-69ae248eae21 {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 15:25:05 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-f4767ed7-e4fa-4c0c-bd0b-b48a0034e6c0 tempest-VolumeTypesTest-1824483253 None] get snapshots for volume v_29534406-54ef-4f00-9568-69ae248eae21 {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 15:25:05 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-f4767ed7-e4fa-4c0c-bd0b-b48a0034e6c0 tempest-VolumeTypesTest-1824483253 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_29534406-54ef-4f00-9568-69ae248eae21/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:25:05 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-f4767ed7-e4fa-4c0c-bd0b-b48a0034e6c0 tempest-VolumeTypesTest-1824483253 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_29534406-54ef-4f00-9568-69ae248eae21/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:25:06 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-25c80733-a960-453f-9159-ec587e0787bf tempest-VolumesExtendTest-93540250 None] Geting {'code': 200, 'error': None, 'data': {'available': '113909850112', 'status': 30, 'import_status': {'import_steps': {'san_setup': True, 'vip_setup': True, 'zfs_import': True, 'mount_system_volume': True, 'nas_setup': True}, 'import_successful': True}, 'scan': None, 'encryption': {'enabled': False}, 'deduplication_rate': '1.00', 'sysvol_upgrade': 'not_needed', 'vdevs': [{'name': 'wwn-0x6000c296b793bf526a107a9d823e947a', 'iostats': {'read': '0', 'write': '0', 'chksum': '0'}, 'disks': [{'origin': 'local', 'slot': '', 'led': 'off', 'name': 'sdn', 'iostats': {'read': '0', 'write': '0', 'chksum': '0'}, 'alias': '', 'health': 'ONLINE', 'sn': '6000c296b793bf526a107a9d823e947a', 'trim_data': {'status': 'trim_unsupported', 'progress': None, 'trim_time': None}, 'path': 'pci-0000:03:00.0-scsi-0:0:14:0', 'model': 'Virtual disk', 'id': 'wwn-0x6000c296b793bf526a107a9d823e947a', 'size': 128849018880}], 'health': 'ONLINE', 'vdev_replacings': [], 'vdev_spares': [], 'type': ''}], 'id': '14181317334440137330', 'health': 'ONLINE', 'iostats': {'read': '0', 'write': '0', 'chksum': '0'}, 'operation': 'none', 'size': '127775277056', 'autotrim': False, 'name': 'Pool-0'}} from GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:25:06 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-25c80733-a960-453f-9159-ec587e0787bf tempest-VolumesExtendTest-93540250 None] Total capacity: 119, Free 106. {{(pid=84696) _update_volume_stats /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:757}} Apr 17 15:25:06 user cinder-volume[84696]: DEBUG cinder.manager [None req-25c80733-a960-453f-9159-ec587e0787bf tempest-VolumesExtendTest-93540250 None] Notifying Schedulers of capabilities ... {{(pid=84696) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 17 15:25:06 user cinder-volume[84696]: INFO cinder.volume.manager [None req-25c80733-a960-453f-9159-ec587e0787bf tempest-VolumesExtendTest-93540250 None] Deleted volume successfully. Apr 17 15:25:06 user cinder-volume[84696]: DEBUG cinder.coordination [None req-25c80733-a960-453f-9159-ec587e0787bf tempest-VolumesExtendTest-93540250 None] Lock "cinder-31d34366-a0ee-4a52-822b-79de201b5a12-delete_volume" released by "delete_volume" :: held 4.644s {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:207}} Apr 17 15:25:06 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-f4767ed7-e4fa-4c0c-bd0b-b48a0034e6c0 tempest-VolumeTypesTest-1824483253 None] Geting {'code': 200, 'error': None, 'data': {'results': 0, 'entries': []}} from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_29534406-54ef-4f00-9568-69ae248eae21/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:25:06 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-f4767ed7-e4fa-4c0c-bd0b-b48a0034e6c0 tempest-VolumeTypesTest-1824483253 None] get volume v_29534406-54ef-4f00-9568-69ae248eae21 info {{(pid=84696) get_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:239}} Apr 17 15:25:06 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-f4767ed7-e4fa-4c0c-bd0b-b48a0034e6c0 tempest-VolumeTypesTest-1824483253 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_29534406-54ef-4f00-9568-69ae248eae21 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:25:06 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-f4767ed7-e4fa-4c0c-bd0b-b48a0034e6c0 tempest-VolumeTypesTest-1824483253 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_29534406-54ef-4f00-9568-69ae248eae21 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:25:06 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-f4767ed7-e4fa-4c0c-bd0b-b48a0034e6c0 tempest-VolumeTypesTest-1824483253 None] Geting {'code': 200, 'error': None, 'data': {'origin': None, 'relatime': None, 'acltype': None, 'vscan': None, 'full_name': 'Pool-0/v_29534406-54ef-4f00-9568-69ae248eae21', 'userrefs': None, 'primarycache': 'all', 'logbias': 'latency', 'creation': '1681746495', 'sync': 'always', 'is_clone': False, 'dedup': 'off', 'sharenfs': None, 'receive_resume_token': None, 'volsize': '1073741824', 'referenced': '57344', 'sharesmb': None, 'createtxg': '188012', 'reservation': '0', 'scontext': None, 'mountpoint': None, 'casesensitivity': None, 'guid': '4544586005323739435', 'usedbyrefreservation': '0', 'dnodesize': None, 'written': '57344', 'logicalused': '28672', 'compressratio': '1.00', 'rootcontext': 'none', 'default_scsi_id': 'b83236ece4cfb4dd', 'type': 'volume', 'compression': 'lz4', 'snapdir': None, 'overlay': None, 'encryption': 'off', 'xattr': None, 'volmode': 'default', 'copies': '1', 'snapshot_limit': '18446744073709551615', 'aclinherit': None, 'defcontext': 'none', 'readonly': 'off', 'version': None, 'recordsize': None, 'filesystem_limit': None, 'mounted': None, 'mlslabel': 'none', 'secondarycache': 'all', 'refreservation': '0', 'available': '113909850112', 'san:volume_id': 'b83236ece4cfb4dde83522256ec1f6afbb8490dc', 'encryptionroot': None, 'exec': None, 'refquota': None, 'refcompressratio': '1.00', 'quota': None, 'utf8only': None, 'keylocation': 'none', 'snapdev': 'hidden', 'snapshot_count': '18446744073709551615', 'fscontext': 'none', 'clones': None, 'canmount': None, 'keystatus': None, 'atime': None, 'usedbysnapshots': '0', 'normalization': None, 'usedbychildren': '0', 'volblocksize': '131072', 'usedbydataset': '57344', 'objsetid': '123179', 'name': 'v_29534406-54ef-4f00-9568-69ae248eae21', 'defer_destroy': None, 'pbkdf2iters': '0', 'checksum': 'on', 'redundant_metadata': 'all', 'filesystem_count': None, 'devices': None, 'keyformat': 'none', 'setuid': None, 'used': '57344', 'logicalreferenced': '28672', 'context': 'none', 'zoned': None, 'nbmand': None}} from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_29534406-54ef-4f00-9568-69ae248eae21 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:25:06 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-f4767ed7-e4fa-4c0c-bd0b-b48a0034e6c0 tempest-VolumeTypesTest-1824483253 None] delete volume:v_29534406-54ef-4f00-9568-69ae248eae21 recursively children:{'force_umount': True} {{(pid=84696) delete_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:335}} Apr 17 15:25:06 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-f4767ed7-e4fa-4c0c-bd0b-b48a0034e6c0 tempest-VolumeTypesTest-1824483253 None] Sending pool request DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_29534406-54ef-4f00-9568-69ae248eae21 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:25:06 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-f4767ed7-e4fa-4c0c-bd0b-b48a0034e6c0 tempest-VolumeTypesTest-1824483253 None] Sending DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_29534406-54ef-4f00-9568-69ae248eae21 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:25:06 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-120c9c51-a7c0-483f-b2de-e5ce22a20a39 req-34fc5997-2964-49b3-b98a-4c267b80eb23 tempest-TaggedBootDevicesTest_v242-1317742003 None] Geting {'code': 201, 'error': None, 'data': {'active': True, 'allow_ip': [], 'conflicted': False, 'deny_ip': [], 'incoming_users_active': True, 'name': 'iqn.2010-10.org.openstack:18475864-c202-4937-b18f-cc3e16c6d89f', 'outgoing_user': None}} from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:25:06 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [req-120c9c51-a7c0-483f-b2de-e5ce22a20a39 req-34fc5997-2964-49b3-b98a-4c267b80eb23 tempest-TaggedBootDevicesTest_v242-1317742003 None] atach volume v_18475864-c202-4937-b18f-cc3e16c6d89f to target iqn.2010-10.org.openstack:18475864-c202-4937-b18f-cc3e16c6d89f {{(pid=84696) attach_target_vol /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:576}} Apr 17 15:25:06 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-120c9c51-a7c0-483f-b2de-e5ce22a20a39 req-34fc5997-2964-49b3-b98a-4c267b80eb23 tempest-TaggedBootDevicesTest_v242-1317742003 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:18475864-c202-4937-b18f-cc3e16c6d89f/luns {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:25:06 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-120c9c51-a7c0-483f-b2de-e5ce22a20a39 req-34fc5997-2964-49b3-b98a-4c267b80eb23 tempest-TaggedBootDevicesTest_v242-1317742003 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:18475864-c202-4937-b18f-cc3e16c6d89f/luns {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:25:07 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-56e853c8-d1bd-490d-9734-edcd1d06128c tempest-VolumeTypesTest-1824483253 None] Geting {'code': 200, 'error': None, 'data': {'available': '113909850112', 'status': 30, 'import_status': {'import_steps': {'san_setup': True, 'vip_setup': True, 'zfs_import': True, 'mount_system_volume': True, 'nas_setup': True}, 'import_successful': True}, 'scan': None, 'encryption': {'enabled': False}, 'deduplication_rate': '1.00', 'sysvol_upgrade': 'not_needed', 'vdevs': [{'name': 'wwn-0x6000c296b793bf526a107a9d823e947a', 'iostats': {'read': '0', 'write': '0', 'chksum': '0'}, 'disks': [{'origin': 'local', 'slot': '', 'led': 'off', 'name': 'sdn', 'iostats': {'read': '0', 'write': '0', 'chksum': '0'}, 'alias': '', 'health': 'ONLINE', 'sn': '6000c296b793bf526a107a9d823e947a', 'trim_data': {'status': 'trim_unsupported', 'progress': None, 'trim_time': None}, 'path': 'pci-0000:03:00.0-scsi-0:0:14:0', 'model': 'Virtual disk', 'id': 'wwn-0x6000c296b793bf526a107a9d823e947a', 'size': 128849018880}], 'health': 'ONLINE', 'vdev_replacings': [], 'vdev_spares': [], 'type': ''}], 'id': '14181317334440137330', 'health': 'ONLINE', 'iostats': {'read': '0', 'write': '0', 'chksum': '0'}, 'operation': 'none', 'size': '127775277056', 'autotrim': False, 'name': 'Pool-0'}} from GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:25:07 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-56e853c8-d1bd-490d-9734-edcd1d06128c tempest-VolumeTypesTest-1824483253 None] Total capacity: 119, Free 106. {{(pid=84696) _update_volume_stats /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:757}} Apr 17 15:25:07 user cinder-volume[84696]: DEBUG cinder.manager [None req-56e853c8-d1bd-490d-9734-edcd1d06128c tempest-VolumeTypesTest-1824483253 None] Notifying Schedulers of capabilities ... {{(pid=84696) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 17 15:25:07 user cinder-volume[84696]: INFO cinder.volume.manager [None req-56e853c8-d1bd-490d-9734-edcd1d06128c tempest-VolumeTypesTest-1824483253 None] Retype volume completed successfully. Apr 17 15:25:09 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-f4767ed7-e4fa-4c0c-bd0b-b48a0034e6c0 tempest-VolumeTypesTest-1824483253 None] Geting {'code': 204} from DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_29534406-54ef-4f00-9568-69ae248eae21 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:25:09 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-f4767ed7-e4fa-4c0c-bd0b-b48a0034e6c0 tempest-VolumeTypesTest-1824483253 None] volume v_29534406-54ef-4f00-9568-69ae248eae21 deleted {{(pid=84696) delete_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:346}} Apr 17 15:25:09 user cinder-volume[84696]: WARNING cinder.quota [None req-f4767ed7-e4fa-4c0c-bd0b-b48a0034e6c0 tempest-VolumeTypesTest-1824483253 None] Deprecated: Default quota for resource: volumes_tempest-VolumeTypesTest-volume-type-1604645545 is set by the default quota flag: quota_volumes_tempest-VolumeTypesTest-volume-type-1604645545, it is now deprecated. Please use the default quota class for default quota. Apr 17 15:25:09 user cinder-volume[84696]: WARNING cinder.quota [None req-f4767ed7-e4fa-4c0c-bd0b-b48a0034e6c0 tempest-VolumeTypesTest-1824483253 None] Deprecated: Default quota for resource: gigabytes_tempest-VolumeTypesTest-volume-type-1604645545 is set by the default quota flag: quota_gigabytes_tempest-VolumeTypesTest-volume-type-1604645545, it is now deprecated. Please use the default quota class for default quota. Apr 17 15:25:09 user cinder-volume[84696]: DEBUG cinder.quota [None req-f4767ed7-e4fa-4c0c-bd0b-b48a0034e6c0 tempest-VolumeTypesTest-1824483253 None] Created reservations ['bb78af81-55d8-4c4e-a5c3-8c7aafbec9d0', '60a9311b-f0c4-45b6-9fa4-53580c918bd3', 'f66d87ec-e54a-42ca-9dfb-9206907636ea', 'a1ec473c-aeae-4510-970f-c14b5f4fa9d6'] {{(pid=84696) reserve /opt/stack/cinder/cinder/quota.py:755}} Apr 17 15:25:09 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-f4767ed7-e4fa-4c0c-bd0b-b48a0034e6c0 tempest-VolumeTypesTest-1824483253 None] Updating volume stats {{(pid=84696) _update_volume_stats /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:724}} Apr 17 15:25:09 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-f4767ed7-e4fa-4c0c-bd0b-b48a0034e6c0 tempest-VolumeTypesTest-1824483253 None] Get pool Pool-0 fsprops {{(pid=84696) get_pool_stats /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:894}} Apr 17 15:25:09 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-f4767ed7-e4fa-4c0c-bd0b-b48a0034e6c0 tempest-VolumeTypesTest-1824483253 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:25:09 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-f4767ed7-e4fa-4c0c-bd0b-b48a0034e6c0 tempest-VolumeTypesTest-1824483253 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:25:09 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-120c9c51-a7c0-483f-b2de-e5ce22a20a39 req-34fc5997-2964-49b3-b98a-4c267b80eb23 tempest-TaggedBootDevicesTest_v242-1317742003 None] Geting {'code': 201, 'error': None, 'data': {'block_size': 512, 'device_handler': 'vdisk_blockio', 'eui': '3861343637336131', 'lun': 0, 'mode': 'wt', 'name': 'v_18475864-c202-4937-b18f-cc3e16c6d89f', 'prod_id': 'Storage', 'scsi_id': '8a4673a10807941e'}} from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:18475864-c202-4937-b18f-cc3e16c6d89f/luns {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:25:09 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [req-120c9c51-a7c0-483f-b2de-e5ce22a20a39 req-34fc5997-2964-49b3-b98a-4c267b80eb23 tempest-TaggedBootDevicesTest_v242-1317742003 None] add credentails to target iqn.2010-10.org.openstack:18475864-c202-4937-b18f-cc3e16c6d89f {{(pid=84696) create_target_user /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:474}} Apr 17 15:25:09 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-120c9c51-a7c0-483f-b2de-e5ce22a20a39 req-34fc5997-2964-49b3-b98a-4c267b80eb23 tempest-TaggedBootDevicesTest_v242-1317742003 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:18475864-c202-4937-b18f-cc3e16c6d89f/incoming-users {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:25:09 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-120c9c51-a7c0-483f-b2de-e5ce22a20a39 req-34fc5997-2964-49b3-b98a-4c267b80eb23 tempest-TaggedBootDevicesTest_v242-1317742003 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:18475864-c202-4937-b18f-cc3e16c6d89f/incoming-users {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:25:09 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-120c9c51-a7c0-483f-b2de-e5ce22a20a39 req-34fc5997-2964-49b3-b98a-4c267b80eb23 tempest-TaggedBootDevicesTest_v242-1317742003 None] Geting {'code': 201, 'error': None, 'data': {'name': 'pmlSchIr'}} from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:18475864-c202-4937-b18f-cc3e16c6d89f/incoming-users {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:25:09 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-120c9c51-a7c0-483f-b2de-e5ce22a20a39 req-34fc5997-2964-49b3-b98a-4c267b80eb23 tempest-TaggedBootDevicesTest_v242-1317742003 None] initialize_connection for 18475864-c202-4937-b18f-cc3e16c6d89f 10.0.0.210. {{(pid=84696) initialize_connection /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:1018}} Apr 17 15:25:09 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-120c9c51-a7c0-483f-b2de-e5ce22a20a39 req-34fc5997-2964-49b3-b98a-4c267b80eb23 tempest-TaggedBootDevicesTest_v242-1317742003 None] Connection info returned from driver {'target_iqn': 'iqn.2010-10.org.openstack:18475864-c202-4937-b18f-cc3e16c6d89f', 'target_portal': '172.16.0.220:3260', 'target_discovered': False, 'auth_method': 'CHAP', 'auth_username': 'pmlSchIr', 'auth_password': '***', 'target_lun': 0, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False, 'driver_volume_type': 'iscsi', 'attachment_id': 'd21f5ce0-237b-4771-ac96-cb17b8159c18'} {{(pid=84696) _connection_create /opt/stack/cinder/cinder/volume/manager.py:4875}} Apr 17 15:25:09 user cinder-volume[84696]: INFO cinder.volume.manager [req-120c9c51-a7c0-483f-b2de-e5ce22a20a39 req-34fc5997-2964-49b3-b98a-4c267b80eb23 tempest-TaggedBootDevicesTest_v242-1317742003 None] attachment_update completed successfully. Apr 17 15:25:10 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-f4767ed7-e4fa-4c0c-bd0b-b48a0034e6c0 tempest-VolumeTypesTest-1824483253 None] Geting {'code': 200, 'error': None, 'data': {'available': '113909932032', 'status': 30, 'import_status': {'import_steps': {'san_setup': True, 'vip_setup': True, 'zfs_import': True, 'mount_system_volume': True, 'nas_setup': True}, 'import_successful': True}, 'scan': None, 'encryption': {'enabled': False}, 'deduplication_rate': '1.00', 'sysvol_upgrade': 'not_needed', 'vdevs': [{'name': 'wwn-0x6000c296b793bf526a107a9d823e947a', 'iostats': {'read': '0', 'write': '0', 'chksum': '0'}, 'disks': [{'origin': 'local', 'slot': '', 'led': 'off', 'name': 'sdn', 'iostats': {'read': '0', 'write': '0', 'chksum': '0'}, 'alias': '', 'health': 'ONLINE', 'sn': '6000c296b793bf526a107a9d823e947a', 'trim_data': {'status': 'trim_unsupported', 'progress': None, 'trim_time': None}, 'path': 'pci-0000:03:00.0-scsi-0:0:14:0', 'model': 'Virtual disk', 'id': 'wwn-0x6000c296b793bf526a107a9d823e947a', 'size': 128849018880}], 'health': 'ONLINE', 'vdev_replacings': [], 'vdev_spares': [], 'type': ''}], 'id': '14181317334440137330', 'health': 'ONLINE', 'iostats': {'read': '0', 'write': '0', 'chksum': '0'}, 'operation': 'none', 'size': '127775277056', 'autotrim': False, 'name': 'Pool-0'}} from GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:25:10 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-f4767ed7-e4fa-4c0c-bd0b-b48a0034e6c0 tempest-VolumeTypesTest-1824483253 None] Total capacity: 119, Free 106. {{(pid=84696) _update_volume_stats /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:757}} Apr 17 15:25:10 user cinder-volume[84696]: DEBUG cinder.manager [None req-f4767ed7-e4fa-4c0c-bd0b-b48a0034e6c0 tempest-VolumeTypesTest-1824483253 None] Notifying Schedulers of capabilities ... {{(pid=84696) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 17 15:25:10 user cinder-volume[84696]: INFO cinder.volume.manager [None req-f4767ed7-e4fa-4c0c-bd0b-b48a0034e6c0 tempest-VolumeTypesTest-1824483253 None] Deleted volume successfully. Apr 17 15:25:10 user cinder-volume[84696]: DEBUG cinder.coordination [None req-f4767ed7-e4fa-4c0c-bd0b-b48a0034e6c0 tempest-VolumeTypesTest-1824483253 None] Lock "cinder-29534406-54ef-4f00-9568-69ae248eae21-delete_volume" released by "delete_volume" :: held 4.337s {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:207}} Apr 17 15:25:10 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-120c9c51-a7c0-483f-b2de-e5ce22a20a39 req-39437dbe-513c-4c4a-94f4-3cf50493517a tempest-TaggedBootDevicesTest_v242-1317742003 None] create export for volume: 65a7c5eb-3af6-4afd-918f-ccb69a68641c. {{(pid=84696) create_export /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:699}} Apr 17 15:25:10 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-120c9c51-a7c0-483f-b2de-e5ce22a20a39 req-39437dbe-513c-4c4a-94f4-3cf50493517a tempest-TaggedBootDevicesTest_v242-1317742003 None] ensure volume 65a7c5eb-3af6-4afd-918f-ccb69a68641c assigned to a proper target {{(pid=84696) _ensure_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:866}} Apr 17 15:25:10 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [req-120c9c51-a7c0-483f-b2de-e5ce22a20a39 req-39437dbe-513c-4c4a-94f4-3cf50493517a tempest-TaggedBootDevicesTest_v242-1317742003 None] check if targe iqn.2010-10.org.openstack:65a7c5eb-3af6-4afd-918f-ccb69a68641c exists {{(pid=84696) is_target /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:376}} Apr 17 15:25:10 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-120c9c51-a7c0-483f-b2de-e5ce22a20a39 req-39437dbe-513c-4c4a-94f4-3cf50493517a tempest-TaggedBootDevicesTest_v242-1317742003 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:65a7c5eb-3af6-4afd-918f-ccb69a68641c {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:25:10 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-120c9c51-a7c0-483f-b2de-e5ce22a20a39 req-39437dbe-513c-4c4a-94f4-3cf50493517a tempest-TaggedBootDevicesTest_v242-1317742003 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:65a7c5eb-3af6-4afd-918f-ccb69a68641c {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:25:10 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-120c9c51-a7c0-483f-b2de-e5ce22a20a39 req-39437dbe-513c-4c4a-94f4-3cf50493517a tempest-TaggedBootDevicesTest_v242-1317742003 None] Geting {'code': 404, 'error': {'class': 'opene.exceptions.ItemNotFoundError', 'message': 'Target iqn.2010-10.org.openstack:65a7c5eb-3af6-4afd-918f-ccb69a68641c not exists.', 'url': 'http://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:65a7c5eb-3af6-4afd-918f-ccb69a68641c'}, 'data': None} from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:65a7c5eb-3af6-4afd-918f-ccb69a68641c {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:25:10 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-120c9c51-a7c0-483f-b2de-e5ce22a20a39 req-39437dbe-513c-4c4a-94f4-3cf50493517a tempest-TaggedBootDevicesTest_v242-1317742003 None] create target and attach volume 65a7c5eb-3af6-4afd-918f-ccb69a68641c to it {{(pid=84696) _create_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:837}} Apr 17 15:25:10 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [req-120c9c51-a7c0-483f-b2de-e5ce22a20a39 req-39437dbe-513c-4c4a-94f4-3cf50493517a tempest-TaggedBootDevicesTest_v242-1317742003 None] create target iqn.2010-10.org.openstack:65a7c5eb-3af6-4afd-918f-ccb69a68641c {{(pid=84696) create_target /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:410}} Apr 17 15:25:10 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-120c9c51-a7c0-483f-b2de-e5ce22a20a39 req-39437dbe-513c-4c4a-94f4-3cf50493517a tempest-TaggedBootDevicesTest_v242-1317742003 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:25:10 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-120c9c51-a7c0-483f-b2de-e5ce22a20a39 req-39437dbe-513c-4c4a-94f4-3cf50493517a tempest-TaggedBootDevicesTest_v242-1317742003 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:25:12 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-120c9c51-a7c0-483f-b2de-e5ce22a20a39 req-39437dbe-513c-4c4a-94f4-3cf50493517a tempest-TaggedBootDevicesTest_v242-1317742003 None] Geting {'code': 201, 'error': None, 'data': {'active': True, 'allow_ip': [], 'conflicted': False, 'deny_ip': [], 'incoming_users_active': True, 'name': 'iqn.2010-10.org.openstack:65a7c5eb-3af6-4afd-918f-ccb69a68641c', 'outgoing_user': None}} from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:25:12 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [req-120c9c51-a7c0-483f-b2de-e5ce22a20a39 req-39437dbe-513c-4c4a-94f4-3cf50493517a tempest-TaggedBootDevicesTest_v242-1317742003 None] atach volume v_65a7c5eb-3af6-4afd-918f-ccb69a68641c to target iqn.2010-10.org.openstack:65a7c5eb-3af6-4afd-918f-ccb69a68641c {{(pid=84696) attach_target_vol /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:576}} Apr 17 15:25:12 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-120c9c51-a7c0-483f-b2de-e5ce22a20a39 req-39437dbe-513c-4c4a-94f4-3cf50493517a tempest-TaggedBootDevicesTest_v242-1317742003 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:65a7c5eb-3af6-4afd-918f-ccb69a68641c/luns {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:25:12 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-120c9c51-a7c0-483f-b2de-e5ce22a20a39 req-39437dbe-513c-4c4a-94f4-3cf50493517a tempest-TaggedBootDevicesTest_v242-1317742003 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:65a7c5eb-3af6-4afd-918f-ccb69a68641c/luns {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:25:14 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-120c9c51-a7c0-483f-b2de-e5ce22a20a39 req-39437dbe-513c-4c4a-94f4-3cf50493517a tempest-TaggedBootDevicesTest_v242-1317742003 None] Geting {'code': 201, 'error': None, 'data': {'block_size': 512, 'device_handler': 'vdisk_blockio', 'eui': '3064666433666363', 'lun': 0, 'mode': 'wt', 'name': 'v_65a7c5eb-3af6-4afd-918f-ccb69a68641c', 'prod_id': 'Storage', 'scsi_id': '0dfd3fcc160aa28c'}} from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:65a7c5eb-3af6-4afd-918f-ccb69a68641c/luns {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:25:14 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [req-120c9c51-a7c0-483f-b2de-e5ce22a20a39 req-39437dbe-513c-4c4a-94f4-3cf50493517a tempest-TaggedBootDevicesTest_v242-1317742003 None] add credentails to target iqn.2010-10.org.openstack:65a7c5eb-3af6-4afd-918f-ccb69a68641c {{(pid=84696) create_target_user /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:474}} Apr 17 15:25:14 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-120c9c51-a7c0-483f-b2de-e5ce22a20a39 req-39437dbe-513c-4c4a-94f4-3cf50493517a tempest-TaggedBootDevicesTest_v242-1317742003 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:65a7c5eb-3af6-4afd-918f-ccb69a68641c/incoming-users {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:25:14 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-120c9c51-a7c0-483f-b2de-e5ce22a20a39 req-39437dbe-513c-4c4a-94f4-3cf50493517a tempest-TaggedBootDevicesTest_v242-1317742003 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:65a7c5eb-3af6-4afd-918f-ccb69a68641c/incoming-users {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:25:14 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-120c9c51-a7c0-483f-b2de-e5ce22a20a39 req-39437dbe-513c-4c4a-94f4-3cf50493517a tempest-TaggedBootDevicesTest_v242-1317742003 None] Geting {'code': 201, 'error': None, 'data': {'name': 'wouATrvO'}} from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:65a7c5eb-3af6-4afd-918f-ccb69a68641c/incoming-users {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:25:14 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-120c9c51-a7c0-483f-b2de-e5ce22a20a39 req-39437dbe-513c-4c4a-94f4-3cf50493517a tempest-TaggedBootDevicesTest_v242-1317742003 None] initialize_connection for 65a7c5eb-3af6-4afd-918f-ccb69a68641c 10.0.0.210. {{(pid=84696) initialize_connection /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:1018}} Apr 17 15:25:14 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-120c9c51-a7c0-483f-b2de-e5ce22a20a39 req-39437dbe-513c-4c4a-94f4-3cf50493517a tempest-TaggedBootDevicesTest_v242-1317742003 None] Connection info returned from driver {'target_iqn': 'iqn.2010-10.org.openstack:65a7c5eb-3af6-4afd-918f-ccb69a68641c', 'target_portal': '172.16.0.220:3260', 'target_discovered': False, 'auth_method': 'CHAP', 'auth_username': 'wouATrvO', 'auth_password': '***', 'target_lun': 0, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False, 'driver_volume_type': 'iscsi', 'attachment_id': 'c57372fa-5821-4a41-9048-fb252947e1d0'} {{(pid=84696) _connection_create /opt/stack/cinder/cinder/volume/manager.py:4875}} Apr 17 15:25:14 user cinder-volume[84696]: INFO cinder.volume.manager [req-120c9c51-a7c0-483f-b2de-e5ce22a20a39 req-39437dbe-513c-4c4a-94f4-3cf50493517a tempest-TaggedBootDevicesTest_v242-1317742003 None] attachment_update completed successfully. Apr 17 15:25:15 user cinder-volume[84696]: DEBUG oslo_service.periodic_task [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Running periodic task VolumeManager.publish_service_capabilities {{(pid=84696) run_periodic_tasks /usr/local/lib/python3.10/dist-packages/oslo_service/periodic_task.py:210}} Apr 17 15:25:15 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Updating volume stats {{(pid=84696) _update_volume_stats /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:724}} Apr 17 15:25:15 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Get pool Pool-0 fsprops {{(pid=84696) get_pool_stats /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:894}} Apr 17 15:25:15 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:25:15 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:25:16 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Geting {'code': 200, 'error': None, 'data': {'available': '113909932032', 'status': 30, 'import_status': {'import_steps': {'san_setup': True, 'vip_setup': True, 'zfs_import': True, 'mount_system_volume': True, 'nas_setup': True}, 'import_successful': True}, 'scan': None, 'encryption': {'enabled': False}, 'deduplication_rate': '1.00', 'sysvol_upgrade': 'not_needed', 'vdevs': [{'name': 'wwn-0x6000c296b793bf526a107a9d823e947a', 'iostats': {'read': '0', 'write': '0', 'chksum': '0'}, 'disks': [{'origin': 'local', 'slot': '', 'led': 'off', 'name': 'sdn', 'iostats': {'read': '0', 'write': '0', 'chksum': '0'}, 'alias': '', 'health': 'ONLINE', 'sn': '6000c296b793bf526a107a9d823e947a', 'trim_data': {'status': 'trim_unsupported', 'progress': None, 'trim_time': None}, 'path': 'pci-0000:03:00.0-scsi-0:0:14:0', 'model': 'Virtual disk', 'id': 'wwn-0x6000c296b793bf526a107a9d823e947a', 'size': 128849018880}], 'health': 'ONLINE', 'vdev_replacings': [], 'vdev_spares': [], 'type': ''}], 'id': '14181317334440137330', 'health': 'ONLINE', 'iostats': {'read': '0', 'write': '0', 'chksum': '0'}, 'operation': 'none', 'size': '127775277056', 'autotrim': False, 'name': 'Pool-0'}} from GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:25:16 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Total capacity: 119, Free 106. {{(pid=84696) _update_volume_stats /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:757}} Apr 17 15:25:16 user cinder-volume[84696]: DEBUG cinder.manager [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Notifying Schedulers of capabilities ... {{(pid=84696) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 17 15:25:25 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-99b77dd0-765c-4f80-8074-39dde517bfa3 tempest-TestShelveInstance-1683612675 None] Volume reschedule parameters: True retry: {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 15:25:25 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-99b77dd0-765c-4f80-8074-39dde517bfa3 tempest-TestShelveInstance-1683612675 None] Flow 'volume_create_manager' (28acc1ad-7c32-4072-b51f-04d2ce189977) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:25:25 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-99b77dd0-765c-4f80-8074-39dde517bfa3 tempest-TestShelveInstance-1683612675 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (44fa4923-5c71-45af-90d5-1cad6bb28912) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:25:25 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-99b77dd0-765c-4f80-8074-39dde517bfa3 tempest-TestShelveInstance-1683612675 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (44fa4923-5c71-45af-90d5-1cad6bb28912) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:25:25Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-tempest.common.compute-volume-1444682483',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=d895d11d-6880-4418-a9ad-feca813ed5d6,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='a2df343eb0d64207a5f9f91aa0810999',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:25:25Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:25:25Z,use_quota=True,user_id='909c09f2587f42f895352916d3f8636a',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:25:25 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-99b77dd0-765c-4f80-8074-39dde517bfa3 tempest-TestShelveInstance-1683612675 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (e71abd77-84ca-4d78-9fa2-11494e5d04a7) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:25:25 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-99b77dd0-765c-4f80-8074-39dde517bfa3 tempest-TestShelveInstance-1683612675 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (e71abd77-84ca-4d78-9fa2-11494e5d04a7) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:25:25 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-99b77dd0-765c-4f80-8074-39dde517bfa3 tempest-TestShelveInstance-1683612675 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (9530ba85-f486-47b6-bd7f-06e6ad32e1d5) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:25:25 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-99b77dd0-765c-4f80-8074-39dde517bfa3 tempest-TestShelveInstance-1683612675 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (9530ba85-f486-47b6-bd7f-06e6ad32e1d5) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'image', 'volume_id': 'd895d11d-6880-4418-a9ad-feca813ed5d6', 'volume_name': 'volume-d895d11d-6880-4418-a9ad-feca813ed5d6', 'volume_size': 1, 'image_id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'image_location': (None, None), 'image_meta': {'name': 'cirros-0.5.2-x86_64-disk', 'disk_format': 'qcow2', 'container_format': 'bare', 'visibility': 'public', 'size': 16300544, 'virtual_size': 117440512, 'status': 'active', 'checksum': 'b874c39491a2377b8490f5f1e89761a4', 'protected': False, 'min_ram': 0, 'min_disk': 0, 'owner': 'a43c211b4ed542538799349489b1d884', 'os_hidden': False, 'os_hash_algo': 'sha512', 'os_hash_value': '6b813aa46bb90b4da216a4d19376593fa3f4fc7e617f03a92b7fe11e9a3981cbe8f0959dbebe36225e5f53dc4492341a4863cac4ed1ee0909f3fc78ef9c3e869', 'id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'created_at': datetime.datetime(2023, 4, 17, 15, 19, 53, tzinfo=datetime.timezone.utc), 'updated_at': datetime.datetime(2023, 4, 17, 15, 19, 55, tzinfo=datetime.timezone.utc), 'tags': [], 'file': '/v2/images/b3f0dec1-7758-44a8-82e0-76ada8837428/file', 'properties': {'hw_rng_model': 'virtio', 'owner_specified.openstack.md5': '', 'owner_specified.openstack.object': 'images/cirros-0.5.2-x86_64-disk', 'owner_specified.openstack.sha256': ''}}, 'image_service': }' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:25:25 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-99b77dd0-765c-4f80-8074-39dde517bfa3 tempest-TestShelveInstance-1683612675 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (ef282e8c-e79a-4bf6-9a5a-914a60435a72) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:25:25 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-99b77dd0-765c-4f80-8074-39dde517bfa3 tempest-TestShelveInstance-1683612675 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (ef282e8c-e79a-4bf6-9a5a-914a60435a72) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:25:25 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-99b77dd0-765c-4f80-8074-39dde517bfa3 tempest-TestShelveInstance-1683612675 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (a08bdee8-f56d-46f3-a696-ae7ad07e890a) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:25:25 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-99b77dd0-765c-4f80-8074-39dde517bfa3 tempest-TestShelveInstance-1683612675 None] Volume d895d11d-6880-4418-a9ad-feca813ed5d6: being created as image with specification: {'status': 'creating', 'volume_name': 'volume-d895d11d-6880-4418-a9ad-feca813ed5d6', 'volume_size': 1, 'image_id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'image_location': (None, None), 'image_meta': {'name': 'cirros-0.5.2-x86_64-disk', 'disk_format': 'qcow2', 'container_format': 'bare', 'visibility': 'public', 'size': 16300544, 'virtual_size': 117440512, 'status': 'active', 'checksum': 'b874c39491a2377b8490f5f1e89761a4', 'protected': False, 'min_ram': 0, 'min_disk': 0, 'owner': 'a43c211b4ed542538799349489b1d884', 'os_hidden': False, 'os_hash_algo': 'sha512', 'os_hash_value': '6b813aa46bb90b4da216a4d19376593fa3f4fc7e617f03a92b7fe11e9a3981cbe8f0959dbebe36225e5f53dc4492341a4863cac4ed1ee0909f3fc78ef9c3e869', 'id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'created_at': datetime.datetime(2023, 4, 17, 15, 19, 53, tzinfo=datetime.timezone.utc), 'updated_at': datetime.datetime(2023, 4, 17, 15, 19, 55, tzinfo=datetime.timezone.utc), 'tags': [], 'file': '/v2/images/b3f0dec1-7758-44a8-82e0-76ada8837428/file', 'properties': {'hw_rng_model': 'virtio', 'owner_specified.openstack.md5': '', 'owner_specified.openstack.object': 'images/cirros-0.5.2-x86_64-disk', 'owner_specified.openstack.sha256': ''}}, 'image_service': } Apr 17 15:25:25 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-99b77dd0-765c-4f80-8074-39dde517bfa3 tempest-TestShelveInstance-1683612675 None] Cloning d895d11d-6880-4418-a9ad-feca813ed5d6 from image b3f0dec1-7758-44a8-82e0-76ada8837428 at location (None, None). {{(pid=84696) _create_from_image /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1071}} Apr 17 15:25:25 user cinder-volume[84696]: DEBUG cinder.coordination [None req-99b77dd0-765c-4f80-8074-39dde517bfa3 tempest-TestShelveInstance-1683612675 None] Acquiring lock "cinder-b3f0dec1-7758-44a8-82e0-76ada8837428" by "_prepare_image_cache_entry" {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:192}} Apr 17 15:25:25 user cinder-volume[84696]: DEBUG cinder.coordination [None req-99b77dd0-765c-4f80-8074-39dde517bfa3 tempest-TestShelveInstance-1683612675 None] Lock "cinder-b3f0dec1-7758-44a8-82e0-76ada8837428" acquired by "_prepare_image_cache_entry" :: waited 0.008s {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:196}} Apr 17 15:25:25 user cinder-volume[84696]: DEBUG cinder.image.cache [None req-99b77dd0-765c-4f80-8074-39dde517bfa3 tempest-TestShelveInstance-1683612675 None] Found image-volume cache entry: {'id': 1, 'image_id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'volume_id': '006f20ae-5306-44fe-afdf-e433f79284e8', 'host': 'user@joviandss-2#joviandss-2', 'size': 1, 'image_updated_at': datetime.datetime(2023, 4, 17, 15, 19, 55), 'last_used': datetime.datetime(2023, 4, 17, 15, 25, 25, 461152)}. {{(pid=84696) get_entry /opt/stack/cinder/cinder/image/cache.py:76}} Apr 17 15:25:25 user cinder-volume[84696]: DEBUG cinder.image.cache [None req-99b77dd0-765c-4f80-8074-39dde517bfa3 tempest-TestShelveInstance-1683612675 None] Image-volume cache entry image_update_at = 2023-04-17 15:19:55+00:00, requested image updated_at = 2023-04-17 15:19:55+00:00. {{(pid=84696) _should_update_entry /opt/stack/cinder/cinder/image/cache.py:259}} Apr 17 15:25:25 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-99b77dd0-765c-4f80-8074-39dde517bfa3 tempest-TestShelveInstance-1683612675 None] Found cache entry for image = b3f0dec1-7758-44a8-82e0-76ada8837428 on host user@joviandss-2#joviandss-2. {{(pid=84696) _prepare_image_cache_entry /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:886}} Apr 17 15:25:25 user cinder-volume[84696]: DEBUG cinder.coordination [None req-99b77dd0-765c-4f80-8074-39dde517bfa3 tempest-TestShelveInstance-1683612675 None] Lock "cinder-b3f0dec1-7758-44a8-82e0-76ada8837428" released by "_prepare_image_cache_entry" :: held 0.013s {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:207}} Apr 17 15:25:25 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-99b77dd0-765c-4f80-8074-39dde517bfa3 tempest-TestShelveInstance-1683612675 None] Attempting to retrieve cache entry for image = b3f0dec1-7758-44a8-82e0-76ada8837428 on host user@joviandss-2#joviandss-2. {{(pid=84696) _create_from_image_cache /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:827}} Apr 17 15:25:25 user cinder-volume[84696]: DEBUG cinder.image.cache [None req-99b77dd0-765c-4f80-8074-39dde517bfa3 tempest-TestShelveInstance-1683612675 None] Found image-volume cache entry: {'id': 1, 'image_id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'volume_id': '006f20ae-5306-44fe-afdf-e433f79284e8', 'host': 'user@joviandss-2#joviandss-2', 'size': 1, 'image_updated_at': datetime.datetime(2023, 4, 17, 15, 19, 55), 'last_used': datetime.datetime(2023, 4, 17, 15, 25, 25, 476678)}. {{(pid=84696) get_entry /opt/stack/cinder/cinder/image/cache.py:76}} Apr 17 15:25:25 user cinder-volume[84696]: DEBUG cinder.image.cache [None req-99b77dd0-765c-4f80-8074-39dde517bfa3 tempest-TestShelveInstance-1683612675 None] Image-volume cache entry image_update_at = 2023-04-17 15:19:55+00:00, requested image updated_at = 2023-04-17 15:19:55+00:00. {{(pid=84696) _should_update_entry /opt/stack/cinder/cinder/image/cache.py:259}} Apr 17 15:25:25 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-99b77dd0-765c-4f80-8074-39dde517bfa3 tempest-TestShelveInstance-1683612675 None] Creating from source image-volume 006f20ae-5306-44fe-afdf-e433f79284e8 {{(pid=84696) _create_from_image_cache /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:841}} Apr 17 15:25:25 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-99b77dd0-765c-4f80-8074-39dde517bfa3 tempest-TestShelveInstance-1683612675 None] cloned volume 006f20ae-5306-44fe-afdf-e433f79284e8 to d895d11d-6880-4418-a9ad-feca813ed5d6 {{(pid=84696) create_cloned_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:521}} Apr 17 15:25:25 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-99b77dd0-765c-4f80-8074-39dde517bfa3 tempest-TestShelveInstance-1683612675 None] cloning v_006f20ae-5306-44fe-afdf-e433f79284e8 to v_d895d11d-6880-4418-a9ad-feca813ed5d6 {{(pid=84696) _clone_object /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:456}} Apr 17 15:25:25 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-99b77dd0-765c-4f80-8074-39dde517bfa3 tempest-TestShelveInstance-1683612675 None] create snapshot v_d895d11d-6880-4418-a9ad-feca813ed5d6 {{(pid=84696) create_snapshot /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:635}} Apr 17 15:25:25 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-99b77dd0-765c-4f80-8074-39dde517bfa3 tempest-TestShelveInstance-1683612675 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:25:25 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-99b77dd0-765c-4f80-8074-39dde517bfa3 tempest-TestShelveInstance-1683612675 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:25:27 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-99b77dd0-765c-4f80-8074-39dde517bfa3 tempest-TestShelveInstance-1683612675 None] Geting {'code': 200, 'error': None, 'data': {'name': 'v_d895d11d-6880-4418-a9ad-feca813ed5d6'}} from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:25:27 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-99b77dd0-765c-4f80-8074-39dde517bfa3 tempest-TestShelveInstance-1683612675 None] create volume v_d895d11d-6880-4418-a9ad-feca813ed5d6 from snapshot v_d895d11d-6880-4418-a9ad-feca813ed5d6 {{(pid=84696) create_volume_from_snapshot /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:674}} Apr 17 15:25:27 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-99b77dd0-765c-4f80-8074-39dde517bfa3 tempest-TestShelveInstance-1683612675 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/clone {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:25:27 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-99b77dd0-765c-4f80-8074-39dde517bfa3 tempest-TestShelveInstance-1683612675 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/clone {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:25:28 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-99b77dd0-765c-4f80-8074-39dde517bfa3 tempest-TestShelveInstance-1683612675 None] Geting {'code': 200, 'error': None, 'data': {'origin': 'Pool-0/v_006f20ae-5306-44fe-afdf-e433f79284e8@v_d895d11d-6880-4418-a9ad-feca813ed5d6', 'is_clone': True, 'full_name': 'Pool-0/v_d895d11d-6880-4418-a9ad-feca813ed5d6', 'name': 'v_d895d11d-6880-4418-a9ad-feca813ed5d6'}} from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/clone {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:25:28 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-99b77dd0-765c-4f80-8074-39dde517bfa3 tempest-TestShelveInstance-1683612675 None] get volume v_d895d11d-6880-4418-a9ad-feca813ed5d6 info {{(pid=84696) get_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:239}} Apr 17 15:25:28 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-99b77dd0-765c-4f80-8074-39dde517bfa3 tempest-TestShelveInstance-1683612675 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_d895d11d-6880-4418-a9ad-feca813ed5d6 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:25:28 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-99b77dd0-765c-4f80-8074-39dde517bfa3 tempest-TestShelveInstance-1683612675 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_d895d11d-6880-4418-a9ad-feca813ed5d6 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:25:28 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-99b77dd0-765c-4f80-8074-39dde517bfa3 tempest-TestShelveInstance-1683612675 None] Geting {'code': 200, 'error': None, 'data': {'origin': 'Pool-0/v_006f20ae-5306-44fe-afdf-e433f79284e8@v_d895d11d-6880-4418-a9ad-feca813ed5d6', 'relatime': None, 'acltype': None, 'vscan': None, 'full_name': 'Pool-0/v_d895d11d-6880-4418-a9ad-feca813ed5d6', 'userrefs': None, 'primarycache': 'all', 'logbias': 'latency', 'creation': '1681746520', 'sync': 'always', 'is_clone': True, 'dedup': 'off', 'sharenfs': None, 'receive_resume_token': None, 'volsize': '1073741824', 'referenced': '16285696', 'sharesmb': None, 'createtxg': '188034', 'reservation': '0', 'scontext': None, 'mountpoint': None, 'casesensitivity': None, 'guid': '1691245292466290146', 'usedbyrefreservation': '0', 'dnodesize': None, 'written': '0', 'logicalused': '0', 'compressratio': '1.00', 'rootcontext': 'none', 'default_scsi_id': '1c1bcfc8b40c3c32', 'type': 'volume', 'compression': 'lz4', 'snapdir': None, 'overlay': None, 'encryption': 'off', 'xattr': None, 'volmode': 'default', 'copies': '1', 'snapshot_limit': '18446744073709551615', 'aclinherit': None, 'defcontext': 'none', 'readonly': 'off', 'version': None, 'recordsize': None, 'filesystem_limit': None, 'mounted': None, 'mlslabel': 'none', 'secondarycache': 'all', 'refreservation': '0', 'available': '113909907456', 'used': '0', 'encryptionroot': None, 'exec': None, 'refquota': None, 'refcompressratio': '1.17', 'quota': None, 'utf8only': None, 'keylocation': 'none', 'snapdev': 'hidden', 'snapshot_count': '18446744073709551615', 'fscontext': 'none', 'clones': None, 'canmount': None, 'keystatus': None, 'atime': None, 'usedbysnapshots': '0', 'normalization': None, 'usedbychildren': '0', 'volblocksize': '131072', 'usedbydataset': '0', 'objsetid': '122052', 'name': 'v_d895d11d-6880-4418-a9ad-feca813ed5d6', 'defer_destroy': None, 'pbkdf2iters': '0', 'checksum': 'on', 'redundant_metadata': 'all', 'filesystem_count': None, 'devices': None, 'keyformat': 'none', 'setuid': None, 'logicalreferenced': '19042304', 'context': 'none', 'zoned': None, 'nbmand': None}} from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_d895d11d-6880-4418-a9ad-feca813ed5d6 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:25:28 user cinder-volume[84696]: DEBUG cinder.volume.volume_utils [None req-99b77dd0-765c-4f80-8074-39dde517bfa3 tempest-TestShelveInstance-1683612675 None] Marking volume d895d11d-6880-4418-a9ad-feca813ed5d6 as bootable. {{(pid=84696) enable_bootable_flag /opt/stack/cinder/cinder/volume/volume_utils.py:1140}} Apr 17 15:25:28 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-99b77dd0-765c-4f80-8074-39dde517bfa3 tempest-TestShelveInstance-1683612675 None] Copying metadata from image b3f0dec1-7758-44a8-82e0-76ada8837428 to d895d11d-6880-4418-a9ad-feca813ed5d6. {{(pid=84696) _handle_bootable_volume_glance_meta /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:452}} Apr 17 15:25:28 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-99b77dd0-765c-4f80-8074-39dde517bfa3 tempest-TestShelveInstance-1683612675 None] Creating volume glance metadata for volume d895d11d-6880-4418-a9ad-feca813ed5d6 backed by image b3f0dec1-7758-44a8-82e0-76ada8837428 with: {'hw_rng_model': 'virtio', 'owner_specified.openstack.md5': '', 'owner_specified.openstack.object': 'images/cirros-0.5.2-x86_64-disk', 'owner_specified.openstack.sha256': '', 'image_id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'image_name': 'cirros-0.5.2-x86_64-disk', 'checksum': 'b874c39491a2377b8490f5f1e89761a4', 'container_format': 'bare', 'disk_format': 'qcow2', 'min_disk': 0, 'min_ram': 0, 'size': 16300544}. {{(pid=84696) _capture_volume_image_metadata /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:683}} Apr 17 15:25:28 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=57,cinder:INSERT=10,cinder:UPDATE=33 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:25:28 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-99b77dd0-765c-4f80-8074-39dde517bfa3 tempest-TestShelveInstance-1683612675 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (a08bdee8-f56d-46f3-a696-ae7ad07e890a) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'volume_name': 'volume-d895d11d-6880-4418-a9ad-feca813ed5d6', 'volume_size': 1, 'image_id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'image_location': (None, None), 'image_meta': {'name': 'cirros-0.5.2-x86_64-disk', 'disk_format': 'qcow2', 'container_format': 'bare', 'visibility': 'public', 'size': 16300544, 'virtual_size': 117440512, 'status': 'active', 'checksum': 'b874c39491a2377b8490f5f1e89761a4', 'protected': False, 'min_ram': 0, 'min_disk': 0, 'owner': 'a43c211b4ed542538799349489b1d884', 'os_hidden': False, 'os_hash_algo': 'sha512', 'os_hash_value': '6b813aa46bb90b4da216a4d19376593fa3f4fc7e617f03a92b7fe11e9a3981cbe8f0959dbebe36225e5f53dc4492341a4863cac4ed1ee0909f3fc78ef9c3e869', 'id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'created_at': datetime.datetime(2023, 4, 17, 15, 19, 53, tzinfo=datetime.timezone.utc), 'updated_at': datetime.datetime(2023, 4, 17, 15, 19, 55, tzinfo=datetime.timezone.utc), 'tags': [], 'file': '/v2/images/b3f0dec1-7758-44a8-82e0-76ada8837428/file', 'properties': {'hw_rng_model': 'virtio', 'owner_specified.openstack.md5': '', 'owner_specified.openstack.object': 'images/cirros-0.5.2-x86_64-disk', 'owner_specified.openstack.sha256': ''}}, 'image_service': }' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:25:28 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-99b77dd0-765c-4f80-8074-39dde517bfa3 tempest-TestShelveInstance-1683612675 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeOnFinishTask;volume:create, create.end' (8bc858d4-a844-467a-9c02-92ba9f113762) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:25:28 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-99b77dd0-765c-4f80-8074-39dde517bfa3 tempest-TestShelveInstance-1683612675 None] Volume volume-d895d11d-6880-4418-a9ad-feca813ed5d6 (d895d11d-6880-4418-a9ad-feca813ed5d6): created successfully Apr 17 15:25:28 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-99b77dd0-765c-4f80-8074-39dde517bfa3 tempest-TestShelveInstance-1683612675 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeOnFinishTask;volume:create, create.end' (8bc858d4-a844-467a-9c02-92ba9f113762) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:25:28 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-99b77dd0-765c-4f80-8074-39dde517bfa3 tempest-TestShelveInstance-1683612675 None] Flow 'volume_create_manager' (28acc1ad-7c32-4072-b51f-04d2ce189977) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:25:28 user cinder-volume[84696]: INFO cinder.volume.manager [None req-99b77dd0-765c-4f80-8074-39dde517bfa3 tempest-TestShelveInstance-1683612675 None] Created volume successfully. Apr 17 15:25:31 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-e68739af-adba-4118-be06-d8d33ae11364 req-70a95b81-2e50-49e5-bf8c-fd71c1e50576 tempest-TestShelveInstance-1683612675 None] create export for volume: d895d11d-6880-4418-a9ad-feca813ed5d6. {{(pid=84696) create_export /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:699}} Apr 17 15:25:31 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-e68739af-adba-4118-be06-d8d33ae11364 req-70a95b81-2e50-49e5-bf8c-fd71c1e50576 tempest-TestShelveInstance-1683612675 None] ensure volume d895d11d-6880-4418-a9ad-feca813ed5d6 assigned to a proper target {{(pid=84696) _ensure_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:866}} Apr 17 15:25:31 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [req-e68739af-adba-4118-be06-d8d33ae11364 req-70a95b81-2e50-49e5-bf8c-fd71c1e50576 tempest-TestShelveInstance-1683612675 None] check if targe iqn.2010-10.org.openstack:d895d11d-6880-4418-a9ad-feca813ed5d6 exists {{(pid=84696) is_target /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:376}} Apr 17 15:25:31 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-e68739af-adba-4118-be06-d8d33ae11364 req-70a95b81-2e50-49e5-bf8c-fd71c1e50576 tempest-TestShelveInstance-1683612675 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:d895d11d-6880-4418-a9ad-feca813ed5d6 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:25:31 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-e68739af-adba-4118-be06-d8d33ae11364 req-70a95b81-2e50-49e5-bf8c-fd71c1e50576 tempest-TestShelveInstance-1683612675 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:d895d11d-6880-4418-a9ad-feca813ed5d6 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:25:31 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-e68739af-adba-4118-be06-d8d33ae11364 req-70a95b81-2e50-49e5-bf8c-fd71c1e50576 tempest-TestShelveInstance-1683612675 None] Geting {'code': 404, 'error': {'class': 'opene.exceptions.ItemNotFoundError', 'message': 'Target iqn.2010-10.org.openstack:d895d11d-6880-4418-a9ad-feca813ed5d6 not exists.', 'url': 'http://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:d895d11d-6880-4418-a9ad-feca813ed5d6'}, 'data': None} from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:d895d11d-6880-4418-a9ad-feca813ed5d6 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:25:31 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-e68739af-adba-4118-be06-d8d33ae11364 req-70a95b81-2e50-49e5-bf8c-fd71c1e50576 tempest-TestShelveInstance-1683612675 None] create target and attach volume d895d11d-6880-4418-a9ad-feca813ed5d6 to it {{(pid=84696) _create_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:837}} Apr 17 15:25:31 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [req-e68739af-adba-4118-be06-d8d33ae11364 req-70a95b81-2e50-49e5-bf8c-fd71c1e50576 tempest-TestShelveInstance-1683612675 None] create target iqn.2010-10.org.openstack:d895d11d-6880-4418-a9ad-feca813ed5d6 {{(pid=84696) create_target /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:410}} Apr 17 15:25:31 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-e68739af-adba-4118-be06-d8d33ae11364 req-70a95b81-2e50-49e5-bf8c-fd71c1e50576 tempest-TestShelveInstance-1683612675 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:25:31 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-e68739af-adba-4118-be06-d8d33ae11364 req-70a95b81-2e50-49e5-bf8c-fd71c1e50576 tempest-TestShelveInstance-1683612675 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:25:33 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-a728e6aa-455a-49d8-9c3f-ede168768691 tempest-DeleteServersTestJSON-1458225442 None] Volume reschedule parameters: True retry: {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 15:25:33 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-a728e6aa-455a-49d8-9c3f-ede168768691 tempest-DeleteServersTestJSON-1458225442 None] Flow 'volume_create_manager' (3319bdb3-d24a-480d-8576-eff92e0d9e3c) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:25:33 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-a728e6aa-455a-49d8-9c3f-ede168768691 tempest-DeleteServersTestJSON-1458225442 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (8ab7d5a1-d991-4843-988c-0c5342ad60bf) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:25:33 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-a728e6aa-455a-49d8-9c3f-ede168768691 tempest-DeleteServersTestJSON-1458225442 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (8ab7d5a1-d991-4843-988c-0c5342ad60bf) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:25:34Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-DeleteServersTestJSON-volume-1479973060',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=a5f9d059-9f1b-4cf5-ae89-92bcf6c50326,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='8aecd98c92d74f6b9879e6f57125ebde',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:25:34Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:25:34Z,use_quota=True,user_id='fec7c09a38ee413db724c659acc576ce',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:25:33 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-a728e6aa-455a-49d8-9c3f-ede168768691 tempest-DeleteServersTestJSON-1458225442 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (fa16184d-d85f-4f9a-8eb3-c31cf64f943a) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:25:33 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-a728e6aa-455a-49d8-9c3f-ede168768691 tempest-DeleteServersTestJSON-1458225442 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (fa16184d-d85f-4f9a-8eb3-c31cf64f943a) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:25:33 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-a728e6aa-455a-49d8-9c3f-ede168768691 tempest-DeleteServersTestJSON-1458225442 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (d9ae0da0-3d2d-47b9-91cf-53779b432134) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:25:33 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-a728e6aa-455a-49d8-9c3f-ede168768691 tempest-DeleteServersTestJSON-1458225442 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (d9ae0da0-3d2d-47b9-91cf-53779b432134) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': 'a5f9d059-9f1b-4cf5-ae89-92bcf6c50326', 'volume_name': 'volume-a5f9d059-9f1b-4cf5-ae89-92bcf6c50326', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:25:33 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-a728e6aa-455a-49d8-9c3f-ede168768691 tempest-DeleteServersTestJSON-1458225442 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (a9aae958-13f0-4420-a3c0-20ba5ef4a64a) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:25:33 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-a728e6aa-455a-49d8-9c3f-ede168768691 tempest-DeleteServersTestJSON-1458225442 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (a9aae958-13f0-4420-a3c0-20ba5ef4a64a) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:25:33 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-a728e6aa-455a-49d8-9c3f-ede168768691 tempest-DeleteServersTestJSON-1458225442 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (aab77fb8-c424-4c76-95b7-5728d7409027) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:25:33 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-a728e6aa-455a-49d8-9c3f-ede168768691 tempest-DeleteServersTestJSON-1458225442 None] Volume a5f9d059-9f1b-4cf5-ae89-92bcf6c50326: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-a5f9d059-9f1b-4cf5-ae89-92bcf6c50326', 'volume_size': 1} Apr 17 15:25:33 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-a728e6aa-455a-49d8-9c3f-ede168768691 tempest-DeleteServersTestJSON-1458225442 None] creating volume v_a5f9d059-9f1b-4cf5-ae89-92bcf6c50326. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 15:25:33 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-a728e6aa-455a-49d8-9c3f-ede168768691 tempest-DeleteServersTestJSON-1458225442 None] create volume {'name': 'v_a5f9d059-9f1b-4cf5-ae89-92bcf6c50326', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 15:25:33 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-a728e6aa-455a-49d8-9c3f-ede168768691 tempest-DeleteServersTestJSON-1458225442 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:25:33 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-a728e6aa-455a-49d8-9c3f-ede168768691 tempest-DeleteServersTestJSON-1458225442 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:25:33 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:UPDATE=27,cinder:SELECT=61,cinder:DELETE=3,cinder:INSERT=9 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:25:34 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-e68739af-adba-4118-be06-d8d33ae11364 req-70a95b81-2e50-49e5-bf8c-fd71c1e50576 tempest-TestShelveInstance-1683612675 None] Geting {'code': 201, 'error': None, 'data': {'active': True, 'allow_ip': [], 'conflicted': False, 'deny_ip': [], 'incoming_users_active': True, 'name': 'iqn.2010-10.org.openstack:d895d11d-6880-4418-a9ad-feca813ed5d6', 'outgoing_user': None}} from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:25:34 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [req-e68739af-adba-4118-be06-d8d33ae11364 req-70a95b81-2e50-49e5-bf8c-fd71c1e50576 tempest-TestShelveInstance-1683612675 None] atach volume v_d895d11d-6880-4418-a9ad-feca813ed5d6 to target iqn.2010-10.org.openstack:d895d11d-6880-4418-a9ad-feca813ed5d6 {{(pid=84696) attach_target_vol /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:576}} Apr 17 15:25:34 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-e68739af-adba-4118-be06-d8d33ae11364 req-70a95b81-2e50-49e5-bf8c-fd71c1e50576 tempest-TestShelveInstance-1683612675 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:d895d11d-6880-4418-a9ad-feca813ed5d6/luns {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:25:34 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-e68739af-adba-4118-be06-d8d33ae11364 req-70a95b81-2e50-49e5-bf8c-fd71c1e50576 tempest-TestShelveInstance-1683612675 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:d895d11d-6880-4418-a9ad-feca813ed5d6/luns {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:25:35 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-a728e6aa-455a-49d8-9c3f-ede168768691 tempest-DeleteServersTestJSON-1458225442 None] Geting {'code': 201, 'error': None, 'data': {'full_name': 'Pool-0/v_a5f9d059-9f1b-4cf5-ae89-92bcf6c50326', 'is_clone': False, 'name': 'v_a5f9d059-9f1b-4cf5-ae89-92bcf6c50326'}} from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:25:35 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-a728e6aa-455a-49d8-9c3f-ede168768691 tempest-DeleteServersTestJSON-1458225442 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (aab77fb8-c424-4c76-95b7-5728d7409027) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'volume_name': 'volume-a5f9d059-9f1b-4cf5-ae89-92bcf6c50326', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:25:35 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-a728e6aa-455a-49d8-9c3f-ede168768691 tempest-DeleteServersTestJSON-1458225442 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeOnFinishTask;volume:create, create.end' (d7d83942-0b88-46ba-9057-693a3ab059f2) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:25:35 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-a728e6aa-455a-49d8-9c3f-ede168768691 tempest-DeleteServersTestJSON-1458225442 None] Volume volume-a5f9d059-9f1b-4cf5-ae89-92bcf6c50326 (a5f9d059-9f1b-4cf5-ae89-92bcf6c50326): created successfully Apr 17 15:25:35 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-a728e6aa-455a-49d8-9c3f-ede168768691 tempest-DeleteServersTestJSON-1458225442 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeOnFinishTask;volume:create, create.end' (d7d83942-0b88-46ba-9057-693a3ab059f2) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:25:35 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-a728e6aa-455a-49d8-9c3f-ede168768691 tempest-DeleteServersTestJSON-1458225442 None] Flow 'volume_create_manager' (3319bdb3-d24a-480d-8576-eff92e0d9e3c) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:25:35 user cinder-volume[84696]: INFO cinder.volume.manager [None req-a728e6aa-455a-49d8-9c3f-ede168768691 tempest-DeleteServersTestJSON-1458225442 None] Created volume successfully. Apr 17 15:25:35 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-e68739af-adba-4118-be06-d8d33ae11364 req-70a95b81-2e50-49e5-bf8c-fd71c1e50576 tempest-TestShelveInstance-1683612675 None] Geting {'code': 201, 'error': None, 'data': {'block_size': 512, 'device_handler': 'vdisk_blockio', 'eui': '3163316263666338', 'lun': 0, 'mode': 'wt', 'name': 'v_d895d11d-6880-4418-a9ad-feca813ed5d6', 'prod_id': 'Storage', 'scsi_id': '1c1bcfc8b40c3c32'}} from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:d895d11d-6880-4418-a9ad-feca813ed5d6/luns {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:25:35 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [req-e68739af-adba-4118-be06-d8d33ae11364 req-70a95b81-2e50-49e5-bf8c-fd71c1e50576 tempest-TestShelveInstance-1683612675 None] add credentails to target iqn.2010-10.org.openstack:d895d11d-6880-4418-a9ad-feca813ed5d6 {{(pid=84696) create_target_user /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:474}} Apr 17 15:25:35 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-e68739af-adba-4118-be06-d8d33ae11364 req-70a95b81-2e50-49e5-bf8c-fd71c1e50576 tempest-TestShelveInstance-1683612675 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:d895d11d-6880-4418-a9ad-feca813ed5d6/incoming-users {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:25:35 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-e68739af-adba-4118-be06-d8d33ae11364 req-70a95b81-2e50-49e5-bf8c-fd71c1e50576 tempest-TestShelveInstance-1683612675 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:d895d11d-6880-4418-a9ad-feca813ed5d6/incoming-users {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:25:36 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-e68739af-adba-4118-be06-d8d33ae11364 req-70a95b81-2e50-49e5-bf8c-fd71c1e50576 tempest-TestShelveInstance-1683612675 None] Geting {'code': 201, 'error': None, 'data': {'name': 'FVRqMWdJ'}} from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:d895d11d-6880-4418-a9ad-feca813ed5d6/incoming-users {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:25:36 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-e68739af-adba-4118-be06-d8d33ae11364 req-70a95b81-2e50-49e5-bf8c-fd71c1e50576 tempest-TestShelveInstance-1683612675 None] initialize_connection for d895d11d-6880-4418-a9ad-feca813ed5d6 10.0.0.210. {{(pid=84696) initialize_connection /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:1018}} Apr 17 15:25:36 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-e68739af-adba-4118-be06-d8d33ae11364 req-70a95b81-2e50-49e5-bf8c-fd71c1e50576 tempest-TestShelveInstance-1683612675 None] Connection info returned from driver {'target_iqn': 'iqn.2010-10.org.openstack:d895d11d-6880-4418-a9ad-feca813ed5d6', 'target_portal': '172.16.0.220:3260', 'target_discovered': False, 'auth_method': 'CHAP', 'auth_username': 'FVRqMWdJ', 'auth_password': '***', 'target_lun': 0, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False, 'driver_volume_type': 'iscsi', 'attachment_id': 'c0eb6e83-df81-4c2b-920c-8280d74fcde6'} {{(pid=84696) _connection_create /opt/stack/cinder/cinder/volume/manager.py:4875}} Apr 17 15:25:36 user cinder-volume[84696]: INFO cinder.volume.manager [req-e68739af-adba-4118-be06-d8d33ae11364 req-70a95b81-2e50-49e5-bf8c-fd71c1e50576 tempest-TestShelveInstance-1683612675 None] attachment_update completed successfully. Apr 17 15:25:36 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-664238cc-8c86-4ea1-a1cc-92af4eab666f req-a4dc200d-916f-47a0-8c7c-4eb61ffc5c6d tempest-DeleteServersTestJSON-1458225442 None] create export for volume: a5f9d059-9f1b-4cf5-ae89-92bcf6c50326. {{(pid=84696) create_export /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:699}} Apr 17 15:25:36 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-664238cc-8c86-4ea1-a1cc-92af4eab666f req-a4dc200d-916f-47a0-8c7c-4eb61ffc5c6d tempest-DeleteServersTestJSON-1458225442 None] ensure volume a5f9d059-9f1b-4cf5-ae89-92bcf6c50326 assigned to a proper target {{(pid=84696) _ensure_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:866}} Apr 17 15:25:36 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [req-664238cc-8c86-4ea1-a1cc-92af4eab666f req-a4dc200d-916f-47a0-8c7c-4eb61ffc5c6d tempest-DeleteServersTestJSON-1458225442 None] check if targe iqn.2010-10.org.openstack:a5f9d059-9f1b-4cf5-ae89-92bcf6c50326 exists {{(pid=84696) is_target /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:376}} Apr 17 15:25:36 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-664238cc-8c86-4ea1-a1cc-92af4eab666f req-a4dc200d-916f-47a0-8c7c-4eb61ffc5c6d tempest-DeleteServersTestJSON-1458225442 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:a5f9d059-9f1b-4cf5-ae89-92bcf6c50326 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:25:36 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-664238cc-8c86-4ea1-a1cc-92af4eab666f req-a4dc200d-916f-47a0-8c7c-4eb61ffc5c6d tempest-DeleteServersTestJSON-1458225442 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:a5f9d059-9f1b-4cf5-ae89-92bcf6c50326 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:25:36 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-664238cc-8c86-4ea1-a1cc-92af4eab666f req-a4dc200d-916f-47a0-8c7c-4eb61ffc5c6d tempest-DeleteServersTestJSON-1458225442 None] Geting {'code': 404, 'error': {'class': 'opene.exceptions.ItemNotFoundError', 'message': 'Target iqn.2010-10.org.openstack:a5f9d059-9f1b-4cf5-ae89-92bcf6c50326 not exists.', 'url': 'http://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:a5f9d059-9f1b-4cf5-ae89-92bcf6c50326'}, 'data': None} from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:a5f9d059-9f1b-4cf5-ae89-92bcf6c50326 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:25:36 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-664238cc-8c86-4ea1-a1cc-92af4eab666f req-a4dc200d-916f-47a0-8c7c-4eb61ffc5c6d tempest-DeleteServersTestJSON-1458225442 None] create target and attach volume a5f9d059-9f1b-4cf5-ae89-92bcf6c50326 to it {{(pid=84696) _create_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:837}} Apr 17 15:25:36 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [req-664238cc-8c86-4ea1-a1cc-92af4eab666f req-a4dc200d-916f-47a0-8c7c-4eb61ffc5c6d tempest-DeleteServersTestJSON-1458225442 None] create target iqn.2010-10.org.openstack:a5f9d059-9f1b-4cf5-ae89-92bcf6c50326 {{(pid=84696) create_target /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:410}} Apr 17 15:25:36 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-664238cc-8c86-4ea1-a1cc-92af4eab666f req-a4dc200d-916f-47a0-8c7c-4eb61ffc5c6d tempest-DeleteServersTestJSON-1458225442 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:25:36 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-664238cc-8c86-4ea1-a1cc-92af4eab666f req-a4dc200d-916f-47a0-8c7c-4eb61ffc5c6d tempest-DeleteServersTestJSON-1458225442 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:25:39 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-664238cc-8c86-4ea1-a1cc-92af4eab666f req-a4dc200d-916f-47a0-8c7c-4eb61ffc5c6d tempest-DeleteServersTestJSON-1458225442 None] Geting {'code': 201, 'error': None, 'data': {'active': True, 'allow_ip': [], 'conflicted': False, 'deny_ip': [], 'incoming_users_active': True, 'name': 'iqn.2010-10.org.openstack:a5f9d059-9f1b-4cf5-ae89-92bcf6c50326', 'outgoing_user': None}} from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:25:39 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [req-664238cc-8c86-4ea1-a1cc-92af4eab666f req-a4dc200d-916f-47a0-8c7c-4eb61ffc5c6d tempest-DeleteServersTestJSON-1458225442 None] atach volume v_a5f9d059-9f1b-4cf5-ae89-92bcf6c50326 to target iqn.2010-10.org.openstack:a5f9d059-9f1b-4cf5-ae89-92bcf6c50326 {{(pid=84696) attach_target_vol /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:576}} Apr 17 15:25:39 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-664238cc-8c86-4ea1-a1cc-92af4eab666f req-a4dc200d-916f-47a0-8c7c-4eb61ffc5c6d tempest-DeleteServersTestJSON-1458225442 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:a5f9d059-9f1b-4cf5-ae89-92bcf6c50326/luns {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:25:39 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-664238cc-8c86-4ea1-a1cc-92af4eab666f req-a4dc200d-916f-47a0-8c7c-4eb61ffc5c6d tempest-DeleteServersTestJSON-1458225442 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:a5f9d059-9f1b-4cf5-ae89-92bcf6c50326/luns {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:25:41 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-664238cc-8c86-4ea1-a1cc-92af4eab666f req-a4dc200d-916f-47a0-8c7c-4eb61ffc5c6d tempest-DeleteServersTestJSON-1458225442 None] Geting {'code': 201, 'error': None, 'data': {'block_size': 512, 'device_handler': 'vdisk_blockio', 'eui': '3966613166316366', 'lun': 0, 'mode': 'wt', 'name': 'v_a5f9d059-9f1b-4cf5-ae89-92bcf6c50326', 'prod_id': 'Storage', 'scsi_id': '9fa1f1cf9f3015b5'}} from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:a5f9d059-9f1b-4cf5-ae89-92bcf6c50326/luns {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:25:41 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [req-664238cc-8c86-4ea1-a1cc-92af4eab666f req-a4dc200d-916f-47a0-8c7c-4eb61ffc5c6d tempest-DeleteServersTestJSON-1458225442 None] add credentails to target iqn.2010-10.org.openstack:a5f9d059-9f1b-4cf5-ae89-92bcf6c50326 {{(pid=84696) create_target_user /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:474}} Apr 17 15:25:41 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-664238cc-8c86-4ea1-a1cc-92af4eab666f req-a4dc200d-916f-47a0-8c7c-4eb61ffc5c6d tempest-DeleteServersTestJSON-1458225442 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:a5f9d059-9f1b-4cf5-ae89-92bcf6c50326/incoming-users {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:25:41 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-664238cc-8c86-4ea1-a1cc-92af4eab666f req-a4dc200d-916f-47a0-8c7c-4eb61ffc5c6d tempest-DeleteServersTestJSON-1458225442 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:a5f9d059-9f1b-4cf5-ae89-92bcf6c50326/incoming-users {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:25:41 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-664238cc-8c86-4ea1-a1cc-92af4eab666f req-a4dc200d-916f-47a0-8c7c-4eb61ffc5c6d tempest-DeleteServersTestJSON-1458225442 None] Geting {'code': 201, 'error': None, 'data': {'name': 'rxEFJebi'}} from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:a5f9d059-9f1b-4cf5-ae89-92bcf6c50326/incoming-users {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:25:41 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-664238cc-8c86-4ea1-a1cc-92af4eab666f req-a4dc200d-916f-47a0-8c7c-4eb61ffc5c6d tempest-DeleteServersTestJSON-1458225442 None] initialize_connection for a5f9d059-9f1b-4cf5-ae89-92bcf6c50326 10.0.0.210. {{(pid=84696) initialize_connection /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:1018}} Apr 17 15:25:41 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-664238cc-8c86-4ea1-a1cc-92af4eab666f req-a4dc200d-916f-47a0-8c7c-4eb61ffc5c6d tempest-DeleteServersTestJSON-1458225442 None] Connection info returned from driver {'target_iqn': 'iqn.2010-10.org.openstack:a5f9d059-9f1b-4cf5-ae89-92bcf6c50326', 'target_portal': '172.16.0.220:3260', 'target_discovered': False, 'auth_method': 'CHAP', 'auth_username': 'rxEFJebi', 'auth_password': '***', 'target_lun': 0, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False, 'driver_volume_type': 'iscsi', 'attachment_id': '48883706-c9f1-4af4-93c4-d1fe08ad33fa'} {{(pid=84696) _connection_create /opt/stack/cinder/cinder/volume/manager.py:4875}} Apr 17 15:25:41 user cinder-volume[84696]: INFO cinder.volume.manager [req-664238cc-8c86-4ea1-a1cc-92af4eab666f req-a4dc200d-916f-47a0-8c7c-4eb61ffc5c6d tempest-DeleteServersTestJSON-1458225442 None] attachment_update completed successfully. Apr 17 15:25:46 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-292d6e35-8ff9-4d0c-87e6-e4b27b69e96d req-f6ca2c48-e724-4083-8a12-6e07397f8ab0 tempest-DeleteServersTestJSON-1458225442 None] terminate connection for a5f9d059-9f1b-4cf5-ae89-92bcf6c50326 {{(pid=84696) terminate_connection /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:1032}} Apr 17 15:25:46 user cinder-volume[84696]: INFO cinder.volume.manager [req-292d6e35-8ff9-4d0c-87e6-e4b27b69e96d req-f6ca2c48-e724-4083-8a12-6e07397f8ab0 tempest-DeleteServersTestJSON-1458225442 None] Terminate volume connection completed successfully. Apr 17 15:25:46 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-292d6e35-8ff9-4d0c-87e6-e4b27b69e96d req-f6ca2c48-e724-4083-8a12-6e07397f8ab0 tempest-DeleteServersTestJSON-1458225442 None] Deleting attachment 48883706-c9f1-4af4-93c4-d1fe08ad33fa. {{(pid=84696) attachment_delete /opt/stack/cinder/cinder/volume/manager.py:5006}} Apr 17 15:25:46 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-292d6e35-8ff9-4d0c-87e6-e4b27b69e96d req-f6ca2c48-e724-4083-8a12-6e07397f8ab0 tempest-DeleteServersTestJSON-1458225442 None] remove_export for volume: a5f9d059-9f1b-4cf5-ae89-92bcf6c50326. {{(pid=84696) remove_export /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:718}} Apr 17 15:25:46 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-292d6e35-8ff9-4d0c-87e6-e4b27b69e96d req-f6ca2c48-e724-4083-8a12-6e07397f8ab0 tempest-DeleteServersTestJSON-1458225442 None] remove export {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:912}} Apr 17 15:25:46 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-292d6e35-8ff9-4d0c-87e6-e4b27b69e96d req-f6ca2c48-e724-4083-8a12-6e07397f8ab0 tempest-DeleteServersTestJSON-1458225442 None] detach volume:Volume(_name_id=None,admin_metadata=,attach_status='attached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:25:34Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-DeleteServersTestJSON-volume-1479973060',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=a5f9d059-9f1b-4cf5-ae89-92bcf6c50326,launched_at=2023-04-17T15:25:35Z,metadata=,migration_status=None,multiattach=False,previous_status=None,project_id='8aecd98c92d74f6b9879e6f57125ebde',provider_auth='CHAP rxEFJebi 7ein8k2dpHAS4V',provider_geometry=None,provider_id=None,provider_location='172.16.0.220:3260,1 iqn.2010-10.org.openstack:a5f9d059-9f1b-4cf5-ae89-92bcf6c50326 0',replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:25:34Z,service_uuid='e628352e-c4cf-40eb-8389-cce40663026a',shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='in-use',terminated_at=None,updated_at=2023-04-17T15:25:43Z,use_quota=True,user_id='fec7c09a38ee413db724c659acc576ce',volume_attachment=,volume_type=,volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c) from target:iqn.2010-10.org.openstack:a5f9d059-9f1b-4cf5-ae89-92bcf6c50326. {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:913}} Apr 17 15:25:46 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [req-292d6e35-8ff9-4d0c-87e6-e4b27b69e96d req-f6ca2c48-e724-4083-8a12-6e07397f8ab0 tempest-DeleteServersTestJSON-1458225442 None] detach volume v_a5f9d059-9f1b-4cf5-ae89-92bcf6c50326 from target iqn.2010-10.org.openstack:a5f9d059-9f1b-4cf5-ae89-92bcf6c50326 {{(pid=84696) detach_target_vol /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:606}} Apr 17 15:25:46 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-292d6e35-8ff9-4d0c-87e6-e4b27b69e96d req-f6ca2c48-e724-4083-8a12-6e07397f8ab0 tempest-DeleteServersTestJSON-1458225442 None] Sending pool request DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:a5f9d059-9f1b-4cf5-ae89-92bcf6c50326/luns/v_a5f9d059-9f1b-4cf5-ae89-92bcf6c50326 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:25:46 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-292d6e35-8ff9-4d0c-87e6-e4b27b69e96d req-f6ca2c48-e724-4083-8a12-6e07397f8ab0 tempest-DeleteServersTestJSON-1458225442 None] Sending DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:a5f9d059-9f1b-4cf5-ae89-92bcf6c50326/luns/v_a5f9d059-9f1b-4cf5-ae89-92bcf6c50326 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:25:46 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-292d6e35-8ff9-4d0c-87e6-e4b27b69e96d req-f6ca2c48-e724-4083-8a12-6e07397f8ab0 tempest-DeleteServersTestJSON-1458225442 None] Geting {'code': 204} from DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:a5f9d059-9f1b-4cf5-ae89-92bcf6c50326/luns/v_a5f9d059-9f1b-4cf5-ae89-92bcf6c50326 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:25:46 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-292d6e35-8ff9-4d0c-87e6-e4b27b69e96d req-f6ca2c48-e724-4083-8a12-6e07397f8ab0 tempest-DeleteServersTestJSON-1458225442 None] delete target: iqn.2010-10.org.openstack:a5f9d059-9f1b-4cf5-ae89-92bcf6c50326 {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:930}} Apr 17 15:25:46 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [req-292d6e35-8ff9-4d0c-87e6-e4b27b69e96d req-f6ca2c48-e724-4083-8a12-6e07397f8ab0 tempest-DeleteServersTestJSON-1458225442 None] delete target iqn.2010-10.org.openstack:a5f9d059-9f1b-4cf5-ae89-92bcf6c50326 {{(pid=84696) delete_target /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:442}} Apr 17 15:25:46 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-292d6e35-8ff9-4d0c-87e6-e4b27b69e96d req-f6ca2c48-e724-4083-8a12-6e07397f8ab0 tempest-DeleteServersTestJSON-1458225442 None] Sending pool request DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:a5f9d059-9f1b-4cf5-ae89-92bcf6c50326 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:25:46 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-292d6e35-8ff9-4d0c-87e6-e4b27b69e96d req-f6ca2c48-e724-4083-8a12-6e07397f8ab0 tempest-DeleteServersTestJSON-1458225442 None] Sending DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:a5f9d059-9f1b-4cf5-ae89-92bcf6c50326 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:25:46 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-292d6e35-8ff9-4d0c-87e6-e4b27b69e96d req-f6ca2c48-e724-4083-8a12-6e07397f8ab0 tempest-DeleteServersTestJSON-1458225442 None] Geting {'code': 204} from DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:a5f9d059-9f1b-4cf5-ae89-92bcf6c50326 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:25:46 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [req-292d6e35-8ff9-4d0c-87e6-e4b27b69e96d req-f6ca2c48-e724-4083-8a12-6e07397f8ab0 tempest-DeleteServersTestJSON-1458225442 None] target iqn.2010-10.org.openstack:a5f9d059-9f1b-4cf5-ae89-92bcf6c50326 deleted {{(pid=84696) delete_target /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:447}} Apr 17 15:25:53 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-e7888f03-9769-4460-9438-25f7adb5e7b9 tempest-AttachVolumeShelveTestJSON-1745617049 None] Volume reschedule parameters: True retry: {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 15:25:53 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e7888f03-9769-4460-9438-25f7adb5e7b9 tempest-AttachVolumeShelveTestJSON-1745617049 None] Flow 'volume_create_manager' (63245a1e-aaa3-4cd9-9f39-e1ba608e300d) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:25:53 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e7888f03-9769-4460-9438-25f7adb5e7b9 tempest-AttachVolumeShelveTestJSON-1745617049 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (1036f75c-4336-44d3-9e98-dfa76a8000d1) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:25:53 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e7888f03-9769-4460-9438-25f7adb5e7b9 tempest-AttachVolumeShelveTestJSON-1745617049 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (1036f75c-4336-44d3-9e98-dfa76a8000d1) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:25:53Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-AttachVolumeShelveTestJSON-volume-1457837636',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=13bf568b-585c-4556-ab78-e9930ad50214,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='214a0f233f114d7aa22e64048a837c0b',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:25:53Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:25:53Z,use_quota=True,user_id='83b9d77de46143d8b9277a55f5ad4d72',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:25:53 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e7888f03-9769-4460-9438-25f7adb5e7b9 tempest-AttachVolumeShelveTestJSON-1745617049 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (9250ae28-2d0a-4525-b750-37b602b8ae94) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:25:53 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e7888f03-9769-4460-9438-25f7adb5e7b9 tempest-AttachVolumeShelveTestJSON-1745617049 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (9250ae28-2d0a-4525-b750-37b602b8ae94) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:25:53 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e7888f03-9769-4460-9438-25f7adb5e7b9 tempest-AttachVolumeShelveTestJSON-1745617049 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (1d0a0b60-ab10-4e7c-8b1c-5e5aaa96d98f) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:25:53 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e7888f03-9769-4460-9438-25f7adb5e7b9 tempest-AttachVolumeShelveTestJSON-1745617049 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (1d0a0b60-ab10-4e7c-8b1c-5e5aaa96d98f) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': '13bf568b-585c-4556-ab78-e9930ad50214', 'volume_name': 'volume-13bf568b-585c-4556-ab78-e9930ad50214', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:25:53 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e7888f03-9769-4460-9438-25f7adb5e7b9 tempest-AttachVolumeShelveTestJSON-1745617049 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (98705c06-3780-4f0d-a924-61faa59cec09) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:25:53 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e7888f03-9769-4460-9438-25f7adb5e7b9 tempest-AttachVolumeShelveTestJSON-1745617049 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (98705c06-3780-4f0d-a924-61faa59cec09) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:25:53 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e7888f03-9769-4460-9438-25f7adb5e7b9 tempest-AttachVolumeShelveTestJSON-1745617049 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (6624c08c-27cb-4afb-ac7a-31f1366c93fe) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:25:53 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-e7888f03-9769-4460-9438-25f7adb5e7b9 tempest-AttachVolumeShelveTestJSON-1745617049 None] Volume 13bf568b-585c-4556-ab78-e9930ad50214: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-13bf568b-585c-4556-ab78-e9930ad50214', 'volume_size': 1} Apr 17 15:25:53 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-e7888f03-9769-4460-9438-25f7adb5e7b9 tempest-AttachVolumeShelveTestJSON-1745617049 None] creating volume v_13bf568b-585c-4556-ab78-e9930ad50214. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 15:25:53 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-e7888f03-9769-4460-9438-25f7adb5e7b9 tempest-AttachVolumeShelveTestJSON-1745617049 None] create volume {'name': 'v_13bf568b-585c-4556-ab78-e9930ad50214', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 15:25:53 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-e7888f03-9769-4460-9438-25f7adb5e7b9 tempest-AttachVolumeShelveTestJSON-1745617049 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:25:53 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-e7888f03-9769-4460-9438-25f7adb5e7b9 tempest-AttachVolumeShelveTestJSON-1745617049 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:25:55 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-e7888f03-9769-4460-9438-25f7adb5e7b9 tempest-AttachVolumeShelveTestJSON-1745617049 None] Geting {'code': 201, 'error': None, 'data': {'full_name': 'Pool-0/v_13bf568b-585c-4556-ab78-e9930ad50214', 'is_clone': False, 'name': 'v_13bf568b-585c-4556-ab78-e9930ad50214'}} from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:25:55 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e7888f03-9769-4460-9438-25f7adb5e7b9 tempest-AttachVolumeShelveTestJSON-1745617049 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (6624c08c-27cb-4afb-ac7a-31f1366c93fe) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'volume_name': 'volume-13bf568b-585c-4556-ab78-e9930ad50214', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:25:55 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e7888f03-9769-4460-9438-25f7adb5e7b9 tempest-AttachVolumeShelveTestJSON-1745617049 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeOnFinishTask;volume:create, create.end' (e355ce5d-bc92-451c-b097-f5c09ab44498) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:25:55 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-e7888f03-9769-4460-9438-25f7adb5e7b9 tempest-AttachVolumeShelveTestJSON-1745617049 None] Volume volume-13bf568b-585c-4556-ab78-e9930ad50214 (13bf568b-585c-4556-ab78-e9930ad50214): created successfully Apr 17 15:25:55 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e7888f03-9769-4460-9438-25f7adb5e7b9 tempest-AttachVolumeShelveTestJSON-1745617049 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeOnFinishTask;volume:create, create.end' (e355ce5d-bc92-451c-b097-f5c09ab44498) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:25:55 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e7888f03-9769-4460-9438-25f7adb5e7b9 tempest-AttachVolumeShelveTestJSON-1745617049 None] Flow 'volume_create_manager' (63245a1e-aaa3-4cd9-9f39-e1ba608e300d) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:25:55 user cinder-volume[84696]: INFO cinder.volume.manager [None req-e7888f03-9769-4460-9438-25f7adb5e7b9 tempest-AttachVolumeShelveTestJSON-1745617049 None] Created volume successfully. Apr 17 15:26:03 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=46,cinder:INSERT=2,cinder:DELETE=1,cinder:UPDATE=12 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:26:08 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=33,cinder:UPDATE=7,cinder:DELETE=2 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:26:08 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-8de17766-ca0a-45d7-9093-3c083ab4c48d tempest-VolumesAdminNegativeTest-1285212149 None] Volume reschedule parameters: True retry: {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 15:26:08 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-8de17766-ca0a-45d7-9093-3c083ab4c48d tempest-VolumesAdminNegativeTest-1285212149 None] Flow 'volume_create_manager' (a6b284e9-a357-46de-9a34-b53e5fbd0e6b) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:26:08 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-8de17766-ca0a-45d7-9093-3c083ab4c48d tempest-VolumesAdminNegativeTest-1285212149 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (fc009856-8e18-4338-81e3-dda50c7dbce8) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:26:08 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-8de17766-ca0a-45d7-9093-3c083ab4c48d tempest-VolumesAdminNegativeTest-1285212149 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (fc009856-8e18-4338-81e3-dda50c7dbce8) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:26:08Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesAdminNegativeTest-volume-850487146',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=215a6a5b-25ca-48fa-a8e3-72416d7a39e2,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='b334c5bc33f143c484518d9bd0e750dd',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:26:08Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:26:08Z,use_quota=True,user_id='3dfadfd30c394b46abe75ec930007850',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:26:08 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-8de17766-ca0a-45d7-9093-3c083ab4c48d tempest-VolumesAdminNegativeTest-1285212149 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (719540fb-3c62-47c3-a7dc-f0fd4011ef70) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:26:08 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-8de17766-ca0a-45d7-9093-3c083ab4c48d tempest-VolumesAdminNegativeTest-1285212149 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (719540fb-3c62-47c3-a7dc-f0fd4011ef70) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:26:08 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-8de17766-ca0a-45d7-9093-3c083ab4c48d tempest-VolumesAdminNegativeTest-1285212149 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (b81d9997-75dc-4ad7-a90d-c16773723b4b) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:26:08 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-8de17766-ca0a-45d7-9093-3c083ab4c48d tempest-VolumesAdminNegativeTest-1285212149 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (b81d9997-75dc-4ad7-a90d-c16773723b4b) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': '215a6a5b-25ca-48fa-a8e3-72416d7a39e2', 'volume_name': 'volume-215a6a5b-25ca-48fa-a8e3-72416d7a39e2', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:26:08 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-8de17766-ca0a-45d7-9093-3c083ab4c48d tempest-VolumesAdminNegativeTest-1285212149 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (3db6a31a-bac2-4fa7-94bc-de4ee9725ffc) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:26:08 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-8de17766-ca0a-45d7-9093-3c083ab4c48d tempest-VolumesAdminNegativeTest-1285212149 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (3db6a31a-bac2-4fa7-94bc-de4ee9725ffc) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:26:08 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-8de17766-ca0a-45d7-9093-3c083ab4c48d tempest-VolumesAdminNegativeTest-1285212149 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (cceb3d86-d593-41b0-bec9-8cbf9792f519) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:26:08 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-8de17766-ca0a-45d7-9093-3c083ab4c48d tempest-VolumesAdminNegativeTest-1285212149 None] Volume 215a6a5b-25ca-48fa-a8e3-72416d7a39e2: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-215a6a5b-25ca-48fa-a8e3-72416d7a39e2', 'volume_size': 1} Apr 17 15:26:08 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-8de17766-ca0a-45d7-9093-3c083ab4c48d tempest-VolumesAdminNegativeTest-1285212149 None] creating volume v_215a6a5b-25ca-48fa-a8e3-72416d7a39e2. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 15:26:08 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-8de17766-ca0a-45d7-9093-3c083ab4c48d tempest-VolumesAdminNegativeTest-1285212149 None] create volume {'name': 'v_215a6a5b-25ca-48fa-a8e3-72416d7a39e2', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 15:26:08 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-8de17766-ca0a-45d7-9093-3c083ab4c48d tempest-VolumesAdminNegativeTest-1285212149 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:26:08 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-8de17766-ca0a-45d7-9093-3c083ab4c48d tempest-VolumesAdminNegativeTest-1285212149 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:26:09 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-8de17766-ca0a-45d7-9093-3c083ab4c48d tempest-VolumesAdminNegativeTest-1285212149 None] Geting {'code': 201, 'error': None, 'data': {'full_name': 'Pool-0/v_215a6a5b-25ca-48fa-a8e3-72416d7a39e2', 'is_clone': False, 'name': 'v_215a6a5b-25ca-48fa-a8e3-72416d7a39e2'}} from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:26:09 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-8de17766-ca0a-45d7-9093-3c083ab4c48d tempest-VolumesAdminNegativeTest-1285212149 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (cceb3d86-d593-41b0-bec9-8cbf9792f519) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'volume_name': 'volume-215a6a5b-25ca-48fa-a8e3-72416d7a39e2', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:26:09 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-8de17766-ca0a-45d7-9093-3c083ab4c48d tempest-VolumesAdminNegativeTest-1285212149 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeOnFinishTask;volume:create, create.end' (605d3898-809c-446e-b88c-a6771c496561) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:26:09 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-8de17766-ca0a-45d7-9093-3c083ab4c48d tempest-VolumesAdminNegativeTest-1285212149 None] Volume volume-215a6a5b-25ca-48fa-a8e3-72416d7a39e2 (215a6a5b-25ca-48fa-a8e3-72416d7a39e2): created successfully Apr 17 15:26:09 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-8de17766-ca0a-45d7-9093-3c083ab4c48d tempest-VolumesAdminNegativeTest-1285212149 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeOnFinishTask;volume:create, create.end' (605d3898-809c-446e-b88c-a6771c496561) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:26:09 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-8de17766-ca0a-45d7-9093-3c083ab4c48d tempest-VolumesAdminNegativeTest-1285212149 None] Flow 'volume_create_manager' (a6b284e9-a357-46de-9a34-b53e5fbd0e6b) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:26:09 user cinder-volume[84696]: INFO cinder.volume.manager [None req-8de17766-ca0a-45d7-9093-3c083ab4c48d tempest-VolumesAdminNegativeTest-1285212149 None] Created volume successfully. Apr 17 15:26:11 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-16569f56-3a92-48d7-bd36-396b19817e9c req-f2f421a7-1285-46d9-8032-1fe2416254cf tempest-VolumesAdminNegativeTest-1285212149 None] create export for volume: 215a6a5b-25ca-48fa-a8e3-72416d7a39e2. {{(pid=84696) create_export /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:699}} Apr 17 15:26:11 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-16569f56-3a92-48d7-bd36-396b19817e9c req-f2f421a7-1285-46d9-8032-1fe2416254cf tempest-VolumesAdminNegativeTest-1285212149 None] ensure volume 215a6a5b-25ca-48fa-a8e3-72416d7a39e2 assigned to a proper target {{(pid=84696) _ensure_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:866}} Apr 17 15:26:11 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [req-16569f56-3a92-48d7-bd36-396b19817e9c req-f2f421a7-1285-46d9-8032-1fe2416254cf tempest-VolumesAdminNegativeTest-1285212149 None] check if targe iqn.2010-10.org.openstack:215a6a5b-25ca-48fa-a8e3-72416d7a39e2 exists {{(pid=84696) is_target /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:376}} Apr 17 15:26:11 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-16569f56-3a92-48d7-bd36-396b19817e9c req-f2f421a7-1285-46d9-8032-1fe2416254cf tempest-VolumesAdminNegativeTest-1285212149 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:215a6a5b-25ca-48fa-a8e3-72416d7a39e2 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:26:11 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-16569f56-3a92-48d7-bd36-396b19817e9c req-f2f421a7-1285-46d9-8032-1fe2416254cf tempest-VolumesAdminNegativeTest-1285212149 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:215a6a5b-25ca-48fa-a8e3-72416d7a39e2 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:26:11 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-16569f56-3a92-48d7-bd36-396b19817e9c req-f2f421a7-1285-46d9-8032-1fe2416254cf tempest-VolumesAdminNegativeTest-1285212149 None] Geting {'code': 404, 'error': {'class': 'opene.exceptions.ItemNotFoundError', 'message': 'Target iqn.2010-10.org.openstack:215a6a5b-25ca-48fa-a8e3-72416d7a39e2 not exists.', 'url': 'http://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:215a6a5b-25ca-48fa-a8e3-72416d7a39e2'}, 'data': None} from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:215a6a5b-25ca-48fa-a8e3-72416d7a39e2 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:26:11 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-16569f56-3a92-48d7-bd36-396b19817e9c req-f2f421a7-1285-46d9-8032-1fe2416254cf tempest-VolumesAdminNegativeTest-1285212149 None] create target and attach volume 215a6a5b-25ca-48fa-a8e3-72416d7a39e2 to it {{(pid=84696) _create_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:837}} Apr 17 15:26:11 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [req-16569f56-3a92-48d7-bd36-396b19817e9c req-f2f421a7-1285-46d9-8032-1fe2416254cf tempest-VolumesAdminNegativeTest-1285212149 None] create target iqn.2010-10.org.openstack:215a6a5b-25ca-48fa-a8e3-72416d7a39e2 {{(pid=84696) create_target /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:410}} Apr 17 15:26:11 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-16569f56-3a92-48d7-bd36-396b19817e9c req-f2f421a7-1285-46d9-8032-1fe2416254cf tempest-VolumesAdminNegativeTest-1285212149 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:26:11 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-16569f56-3a92-48d7-bd36-396b19817e9c req-f2f421a7-1285-46d9-8032-1fe2416254cf tempest-VolumesAdminNegativeTest-1285212149 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:26:14 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-16569f56-3a92-48d7-bd36-396b19817e9c req-f2f421a7-1285-46d9-8032-1fe2416254cf tempest-VolumesAdminNegativeTest-1285212149 None] Geting {'code': 201, 'error': None, 'data': {'active': True, 'allow_ip': [], 'conflicted': False, 'deny_ip': [], 'incoming_users_active': True, 'name': 'iqn.2010-10.org.openstack:215a6a5b-25ca-48fa-a8e3-72416d7a39e2', 'outgoing_user': None}} from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:26:14 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [req-16569f56-3a92-48d7-bd36-396b19817e9c req-f2f421a7-1285-46d9-8032-1fe2416254cf tempest-VolumesAdminNegativeTest-1285212149 None] atach volume v_215a6a5b-25ca-48fa-a8e3-72416d7a39e2 to target iqn.2010-10.org.openstack:215a6a5b-25ca-48fa-a8e3-72416d7a39e2 {{(pid=84696) attach_target_vol /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:576}} Apr 17 15:26:14 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-16569f56-3a92-48d7-bd36-396b19817e9c req-f2f421a7-1285-46d9-8032-1fe2416254cf tempest-VolumesAdminNegativeTest-1285212149 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:215a6a5b-25ca-48fa-a8e3-72416d7a39e2/luns {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:26:14 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-16569f56-3a92-48d7-bd36-396b19817e9c req-f2f421a7-1285-46d9-8032-1fe2416254cf tempest-VolumesAdminNegativeTest-1285212149 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:215a6a5b-25ca-48fa-a8e3-72416d7a39e2/luns {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:26:14 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-70844f65-4304-4cd9-8174-1f52cac4dcfd tempest-AttachSCSIVolumeTestJSON-944917744 None] Volume reschedule parameters: True retry: {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 15:26:14 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-70844f65-4304-4cd9-8174-1f52cac4dcfd tempest-AttachSCSIVolumeTestJSON-944917744 None] Flow 'volume_create_manager' (3ec4a8f1-f153-4557-aca2-3d12dbd7420e) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:26:14 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-70844f65-4304-4cd9-8174-1f52cac4dcfd tempest-AttachSCSIVolumeTestJSON-944917744 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (ce82be16-a0b4-4683-aa60-13688afdcaba) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:26:14 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-70844f65-4304-4cd9-8174-1f52cac4dcfd tempest-AttachSCSIVolumeTestJSON-944917744 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (ce82be16-a0b4-4683-aa60-13688afdcaba) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:26:14Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-AttachSCSIVolumeTestJSON-volume-1764995674',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=bf63d38e-00b8-46cb-acce-fda3fd9b6a9f,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='0fd4ceec29874bc38e8da456cab2048d',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:26:14Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:26:14Z,use_quota=True,user_id='b1734894909e44118bd60e233dc501a0',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:26:14 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-70844f65-4304-4cd9-8174-1f52cac4dcfd tempest-AttachSCSIVolumeTestJSON-944917744 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (449e4e1d-b0a0-4369-8605-fb9a7923b16d) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:26:14 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-70844f65-4304-4cd9-8174-1f52cac4dcfd tempest-AttachSCSIVolumeTestJSON-944917744 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (449e4e1d-b0a0-4369-8605-fb9a7923b16d) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:26:14 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-70844f65-4304-4cd9-8174-1f52cac4dcfd tempest-AttachSCSIVolumeTestJSON-944917744 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (f32a4a1d-9c7e-4e1d-b1e8-67791e97c880) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:26:14 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-70844f65-4304-4cd9-8174-1f52cac4dcfd tempest-AttachSCSIVolumeTestJSON-944917744 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (f32a4a1d-9c7e-4e1d-b1e8-67791e97c880) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': 'bf63d38e-00b8-46cb-acce-fda3fd9b6a9f', 'volume_name': 'volume-bf63d38e-00b8-46cb-acce-fda3fd9b6a9f', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:26:14 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-70844f65-4304-4cd9-8174-1f52cac4dcfd tempest-AttachSCSIVolumeTestJSON-944917744 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (d2c5d43b-8f66-4082-a13c-092073553db9) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:26:14 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-70844f65-4304-4cd9-8174-1f52cac4dcfd tempest-AttachSCSIVolumeTestJSON-944917744 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (d2c5d43b-8f66-4082-a13c-092073553db9) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:26:14 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-70844f65-4304-4cd9-8174-1f52cac4dcfd tempest-AttachSCSIVolumeTestJSON-944917744 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (8d4f4d83-db7d-4f01-80aa-399ec250ed11) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:26:14 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-70844f65-4304-4cd9-8174-1f52cac4dcfd tempest-AttachSCSIVolumeTestJSON-944917744 None] Volume bf63d38e-00b8-46cb-acce-fda3fd9b6a9f: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-bf63d38e-00b8-46cb-acce-fda3fd9b6a9f', 'volume_size': 1} Apr 17 15:26:14 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-70844f65-4304-4cd9-8174-1f52cac4dcfd tempest-AttachSCSIVolumeTestJSON-944917744 None] creating volume v_bf63d38e-00b8-46cb-acce-fda3fd9b6a9f. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 15:26:14 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-70844f65-4304-4cd9-8174-1f52cac4dcfd tempest-AttachSCSIVolumeTestJSON-944917744 None] create volume {'name': 'v_bf63d38e-00b8-46cb-acce-fda3fd9b6a9f', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 15:26:14 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-70844f65-4304-4cd9-8174-1f52cac4dcfd tempest-AttachSCSIVolumeTestJSON-944917744 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:26:14 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-70844f65-4304-4cd9-8174-1f52cac4dcfd tempest-AttachSCSIVolumeTestJSON-944917744 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:26:15 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-47322ce3-978b-4118-82f4-5d45e902b32e req-dbe2da4e-c498-4346-aa59-0d1da1e07c67 tempest-TestVolumeSwap-1714109745 None] create export for volume: 66000c93-eb12-4c23-8a69-71e1aa6b521f. {{(pid=84696) create_export /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:699}} Apr 17 15:26:15 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-47322ce3-978b-4118-82f4-5d45e902b32e req-dbe2da4e-c498-4346-aa59-0d1da1e07c67 tempest-TestVolumeSwap-1714109745 None] ensure volume 66000c93-eb12-4c23-8a69-71e1aa6b521f assigned to a proper target {{(pid=84696) _ensure_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:866}} Apr 17 15:26:15 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [req-47322ce3-978b-4118-82f4-5d45e902b32e req-dbe2da4e-c498-4346-aa59-0d1da1e07c67 tempest-TestVolumeSwap-1714109745 None] check if targe iqn.2010-10.org.openstack:66000c93-eb12-4c23-8a69-71e1aa6b521f exists {{(pid=84696) is_target /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:376}} Apr 17 15:26:15 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-47322ce3-978b-4118-82f4-5d45e902b32e req-dbe2da4e-c498-4346-aa59-0d1da1e07c67 tempest-TestVolumeSwap-1714109745 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:66000c93-eb12-4c23-8a69-71e1aa6b521f {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:26:15 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-47322ce3-978b-4118-82f4-5d45e902b32e req-dbe2da4e-c498-4346-aa59-0d1da1e07c67 tempest-TestVolumeSwap-1714109745 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:66000c93-eb12-4c23-8a69-71e1aa6b521f {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:26:15 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-16569f56-3a92-48d7-bd36-396b19817e9c req-f2f421a7-1285-46d9-8032-1fe2416254cf tempest-VolumesAdminNegativeTest-1285212149 None] Geting {'code': 201, 'error': None, 'data': {'block_size': 512, 'device_handler': 'vdisk_blockio', 'eui': '6631343631356166', 'lun': 0, 'mode': 'wt', 'name': 'v_215a6a5b-25ca-48fa-a8e3-72416d7a39e2', 'prod_id': 'Storage', 'scsi_id': 'f14615afb1c173e1'}} from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:215a6a5b-25ca-48fa-a8e3-72416d7a39e2/luns {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:26:15 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [req-16569f56-3a92-48d7-bd36-396b19817e9c req-f2f421a7-1285-46d9-8032-1fe2416254cf tempest-VolumesAdminNegativeTest-1285212149 None] add credentails to target iqn.2010-10.org.openstack:215a6a5b-25ca-48fa-a8e3-72416d7a39e2 {{(pid=84696) create_target_user /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:474}} Apr 17 15:26:15 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-16569f56-3a92-48d7-bd36-396b19817e9c req-f2f421a7-1285-46d9-8032-1fe2416254cf tempest-VolumesAdminNegativeTest-1285212149 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:215a6a5b-25ca-48fa-a8e3-72416d7a39e2/incoming-users {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:26:15 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-16569f56-3a92-48d7-bd36-396b19817e9c req-f2f421a7-1285-46d9-8032-1fe2416254cf tempest-VolumesAdminNegativeTest-1285212149 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:215a6a5b-25ca-48fa-a8e3-72416d7a39e2/incoming-users {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:26:15 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-47322ce3-978b-4118-82f4-5d45e902b32e req-dbe2da4e-c498-4346-aa59-0d1da1e07c67 tempest-TestVolumeSwap-1714109745 None] Geting {'code': 404, 'error': {'class': 'opene.exceptions.ItemNotFoundError', 'message': 'Target iqn.2010-10.org.openstack:66000c93-eb12-4c23-8a69-71e1aa6b521f not exists.', 'url': 'http://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:66000c93-eb12-4c23-8a69-71e1aa6b521f'}, 'data': None} from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:66000c93-eb12-4c23-8a69-71e1aa6b521f {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:26:15 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-47322ce3-978b-4118-82f4-5d45e902b32e req-dbe2da4e-c498-4346-aa59-0d1da1e07c67 tempest-TestVolumeSwap-1714109745 None] create target and attach volume 66000c93-eb12-4c23-8a69-71e1aa6b521f to it {{(pid=84696) _create_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:837}} Apr 17 15:26:15 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [req-47322ce3-978b-4118-82f4-5d45e902b32e req-dbe2da4e-c498-4346-aa59-0d1da1e07c67 tempest-TestVolumeSwap-1714109745 None] create target iqn.2010-10.org.openstack:66000c93-eb12-4c23-8a69-71e1aa6b521f {{(pid=84696) create_target /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:410}} Apr 17 15:26:15 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-47322ce3-978b-4118-82f4-5d45e902b32e req-dbe2da4e-c498-4346-aa59-0d1da1e07c67 tempest-TestVolumeSwap-1714109745 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:26:15 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-47322ce3-978b-4118-82f4-5d45e902b32e req-dbe2da4e-c498-4346-aa59-0d1da1e07c67 tempest-TestVolumeSwap-1714109745 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:26:15 user cinder-volume[84696]: DEBUG oslo_service.periodic_task [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Running periodic task VolumeManager.publish_service_capabilities {{(pid=84696) run_periodic_tasks /usr/local/lib/python3.10/dist-packages/oslo_service/periodic_task.py:210}} Apr 17 15:26:15 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Updating volume stats {{(pid=84696) _update_volume_stats /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:724}} Apr 17 15:26:15 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Get pool Pool-0 fsprops {{(pid=84696) get_pool_stats /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:894}} Apr 17 15:26:15 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:26:15 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:26:16 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Geting {'code': 200, 'error': None, 'data': {'available': '113889308672', 'status': 30, 'import_status': {'import_steps': {'san_setup': True, 'vip_setup': True, 'zfs_import': True, 'mount_system_volume': True, 'nas_setup': True}, 'import_successful': True}, 'scan': None, 'encryption': {'enabled': False}, 'deduplication_rate': '1.00', 'sysvol_upgrade': 'not_needed', 'vdevs': [{'name': 'wwn-0x6000c296b793bf526a107a9d823e947a', 'iostats': {'read': '0', 'write': '0', 'chksum': '0'}, 'disks': [{'origin': 'local', 'slot': '', 'led': 'off', 'name': 'sdn', 'iostats': {'read': '0', 'write': '0', 'chksum': '0'}, 'alias': '', 'health': 'ONLINE', 'sn': '6000c296b793bf526a107a9d823e947a', 'trim_data': {'status': 'trim_unsupported', 'progress': None, 'trim_time': None}, 'path': 'pci-0000:03:00.0-scsi-0:0:14:0', 'model': 'Virtual disk', 'id': 'wwn-0x6000c296b793bf526a107a9d823e947a', 'size': 128849018880}], 'health': 'ONLINE', 'vdev_replacings': [], 'vdev_spares': [], 'type': ''}], 'id': '14181317334440137330', 'health': 'ONLINE', 'iostats': {'read': '0', 'write': '0', 'chksum': '0'}, 'operation': 'none', 'size': '127775277056', 'autotrim': False, 'name': 'Pool-0'}} from GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:26:16 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Total capacity: 119, Free 106. {{(pid=84696) _update_volume_stats /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:757}} Apr 17 15:26:16 user cinder-volume[84696]: DEBUG cinder.manager [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Notifying Schedulers of capabilities ... {{(pid=84696) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 17 15:26:16 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-70844f65-4304-4cd9-8174-1f52cac4dcfd tempest-AttachSCSIVolumeTestJSON-944917744 None] Geting {'code': 201, 'error': None, 'data': {'full_name': 'Pool-0/v_bf63d38e-00b8-46cb-acce-fda3fd9b6a9f', 'is_clone': False, 'name': 'v_bf63d38e-00b8-46cb-acce-fda3fd9b6a9f'}} from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:26:17 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-70844f65-4304-4cd9-8174-1f52cac4dcfd tempest-AttachSCSIVolumeTestJSON-944917744 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (8d4f4d83-db7d-4f01-80aa-399ec250ed11) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'volume_name': 'volume-bf63d38e-00b8-46cb-acce-fda3fd9b6a9f', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:26:17 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-70844f65-4304-4cd9-8174-1f52cac4dcfd tempest-AttachSCSIVolumeTestJSON-944917744 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeOnFinishTask;volume:create, create.end' (d46243ad-29ce-46dc-9931-7c12c9f23ea9) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:26:17 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-70844f65-4304-4cd9-8174-1f52cac4dcfd tempest-AttachSCSIVolumeTestJSON-944917744 None] Volume volume-bf63d38e-00b8-46cb-acce-fda3fd9b6a9f (bf63d38e-00b8-46cb-acce-fda3fd9b6a9f): created successfully Apr 17 15:26:17 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-70844f65-4304-4cd9-8174-1f52cac4dcfd tempest-AttachSCSIVolumeTestJSON-944917744 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeOnFinishTask;volume:create, create.end' (d46243ad-29ce-46dc-9931-7c12c9f23ea9) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:26:17 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-70844f65-4304-4cd9-8174-1f52cac4dcfd tempest-AttachSCSIVolumeTestJSON-944917744 None] Flow 'volume_create_manager' (3ec4a8f1-f153-4557-aca2-3d12dbd7420e) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:26:17 user cinder-volume[84696]: INFO cinder.volume.manager [None req-70844f65-4304-4cd9-8174-1f52cac4dcfd tempest-AttachSCSIVolumeTestJSON-944917744 None] Created volume successfully. Apr 17 15:26:17 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-16569f56-3a92-48d7-bd36-396b19817e9c req-f2f421a7-1285-46d9-8032-1fe2416254cf tempest-VolumesAdminNegativeTest-1285212149 None] Geting {'code': 201, 'error': None, 'data': {'name': 'wIJnMDkU'}} from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:215a6a5b-25ca-48fa-a8e3-72416d7a39e2/incoming-users {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:26:17 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-16569f56-3a92-48d7-bd36-396b19817e9c req-f2f421a7-1285-46d9-8032-1fe2416254cf tempest-VolumesAdminNegativeTest-1285212149 None] initialize_connection for 215a6a5b-25ca-48fa-a8e3-72416d7a39e2 10.0.0.210. {{(pid=84696) initialize_connection /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:1018}} Apr 17 15:26:17 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-16569f56-3a92-48d7-bd36-396b19817e9c req-f2f421a7-1285-46d9-8032-1fe2416254cf tempest-VolumesAdminNegativeTest-1285212149 None] Connection info returned from driver {'target_iqn': 'iqn.2010-10.org.openstack:215a6a5b-25ca-48fa-a8e3-72416d7a39e2', 'target_portal': '172.16.0.220:3260', 'target_discovered': False, 'auth_method': 'CHAP', 'auth_username': 'wIJnMDkU', 'auth_password': '***', 'target_lun': 0, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False, 'driver_volume_type': 'iscsi', 'attachment_id': '5b4d8d41-f27c-477b-9bbd-0437f5a2d910'} {{(pid=84696) _connection_create /opt/stack/cinder/cinder/volume/manager.py:4875}} Apr 17 15:26:17 user cinder-volume[84696]: INFO cinder.volume.manager [req-16569f56-3a92-48d7-bd36-396b19817e9c req-f2f421a7-1285-46d9-8032-1fe2416254cf tempest-VolumesAdminNegativeTest-1285212149 None] attachment_update completed successfully. Apr 17 15:26:17 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-de849bf8-d9df-401e-9b04-0eb5d5f7d726 req-0fdbe885-b9a4-4b52-a62d-6737ac516ed4 tempest-AttachSCSIVolumeTestJSON-944917744 None] create export for volume: bf63d38e-00b8-46cb-acce-fda3fd9b6a9f. {{(pid=84696) create_export /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:699}} Apr 17 15:26:17 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-de849bf8-d9df-401e-9b04-0eb5d5f7d726 req-0fdbe885-b9a4-4b52-a62d-6737ac516ed4 tempest-AttachSCSIVolumeTestJSON-944917744 None] ensure volume bf63d38e-00b8-46cb-acce-fda3fd9b6a9f assigned to a proper target {{(pid=84696) _ensure_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:866}} Apr 17 15:26:17 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [req-de849bf8-d9df-401e-9b04-0eb5d5f7d726 req-0fdbe885-b9a4-4b52-a62d-6737ac516ed4 tempest-AttachSCSIVolumeTestJSON-944917744 None] check if targe iqn.2010-10.org.openstack:bf63d38e-00b8-46cb-acce-fda3fd9b6a9f exists {{(pid=84696) is_target /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:376}} Apr 17 15:26:17 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-de849bf8-d9df-401e-9b04-0eb5d5f7d726 req-0fdbe885-b9a4-4b52-a62d-6737ac516ed4 tempest-AttachSCSIVolumeTestJSON-944917744 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:bf63d38e-00b8-46cb-acce-fda3fd9b6a9f {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:26:17 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-de849bf8-d9df-401e-9b04-0eb5d5f7d726 req-0fdbe885-b9a4-4b52-a62d-6737ac516ed4 tempest-AttachSCSIVolumeTestJSON-944917744 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:bf63d38e-00b8-46cb-acce-fda3fd9b6a9f {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:26:19 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-fd4f584b-d59b-4452-9b9d-4c5ce060bfcd req-96ff4b61-b76a-41d9-ac83-cf34bbbd176e tempest-TaggedAttachmentsTest-1309189231 None] create export for volume: 3c05f4f0-17a8-4f4c-9987-1fcb502ffd3a. {{(pid=84696) create_export /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:699}} Apr 17 15:26:19 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-fd4f584b-d59b-4452-9b9d-4c5ce060bfcd req-96ff4b61-b76a-41d9-ac83-cf34bbbd176e tempest-TaggedAttachmentsTest-1309189231 None] ensure volume 3c05f4f0-17a8-4f4c-9987-1fcb502ffd3a assigned to a proper target {{(pid=84696) _ensure_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:866}} Apr 17 15:26:19 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [req-fd4f584b-d59b-4452-9b9d-4c5ce060bfcd req-96ff4b61-b76a-41d9-ac83-cf34bbbd176e tempest-TaggedAttachmentsTest-1309189231 None] check if targe iqn.2010-10.org.openstack:3c05f4f0-17a8-4f4c-9987-1fcb502ffd3a exists {{(pid=84696) is_target /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:376}} Apr 17 15:26:19 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-fd4f584b-d59b-4452-9b9d-4c5ce060bfcd req-96ff4b61-b76a-41d9-ac83-cf34bbbd176e tempest-TaggedAttachmentsTest-1309189231 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:3c05f4f0-17a8-4f4c-9987-1fcb502ffd3a {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:26:19 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-fd4f584b-d59b-4452-9b9d-4c5ce060bfcd req-96ff4b61-b76a-41d9-ac83-cf34bbbd176e tempest-TaggedAttachmentsTest-1309189231 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:3c05f4f0-17a8-4f4c-9987-1fcb502ffd3a {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:26:20 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-47322ce3-978b-4118-82f4-5d45e902b32e req-dbe2da4e-c498-4346-aa59-0d1da1e07c67 tempest-TestVolumeSwap-1714109745 None] Geting {'code': 201, 'error': None, 'data': {'active': True, 'allow_ip': [], 'conflicted': False, 'deny_ip': [], 'incoming_users_active': True, 'name': 'iqn.2010-10.org.openstack:66000c93-eb12-4c23-8a69-71e1aa6b521f', 'outgoing_user': None}} from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:26:20 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [req-47322ce3-978b-4118-82f4-5d45e902b32e req-dbe2da4e-c498-4346-aa59-0d1da1e07c67 tempest-TestVolumeSwap-1714109745 None] atach volume v_66000c93-eb12-4c23-8a69-71e1aa6b521f to target iqn.2010-10.org.openstack:66000c93-eb12-4c23-8a69-71e1aa6b521f {{(pid=84696) attach_target_vol /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:576}} Apr 17 15:26:20 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-47322ce3-978b-4118-82f4-5d45e902b32e req-dbe2da4e-c498-4346-aa59-0d1da1e07c67 tempest-TestVolumeSwap-1714109745 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:66000c93-eb12-4c23-8a69-71e1aa6b521f/luns {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:26:20 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-47322ce3-978b-4118-82f4-5d45e902b32e req-dbe2da4e-c498-4346-aa59-0d1da1e07c67 tempest-TestVolumeSwap-1714109745 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:66000c93-eb12-4c23-8a69-71e1aa6b521f/luns {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:26:20 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-de849bf8-d9df-401e-9b04-0eb5d5f7d726 req-0fdbe885-b9a4-4b52-a62d-6737ac516ed4 tempest-AttachSCSIVolumeTestJSON-944917744 None] Geting {'code': 404, 'error': {'class': 'opene.exceptions.ItemNotFoundError', 'message': 'Target iqn.2010-10.org.openstack:bf63d38e-00b8-46cb-acce-fda3fd9b6a9f not exists.', 'url': 'http://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:bf63d38e-00b8-46cb-acce-fda3fd9b6a9f'}, 'data': None} from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:bf63d38e-00b8-46cb-acce-fda3fd9b6a9f {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:26:20 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-de849bf8-d9df-401e-9b04-0eb5d5f7d726 req-0fdbe885-b9a4-4b52-a62d-6737ac516ed4 tempest-AttachSCSIVolumeTestJSON-944917744 None] create target and attach volume bf63d38e-00b8-46cb-acce-fda3fd9b6a9f to it {{(pid=84696) _create_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:837}} Apr 17 15:26:20 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [req-de849bf8-d9df-401e-9b04-0eb5d5f7d726 req-0fdbe885-b9a4-4b52-a62d-6737ac516ed4 tempest-AttachSCSIVolumeTestJSON-944917744 None] create target iqn.2010-10.org.openstack:bf63d38e-00b8-46cb-acce-fda3fd9b6a9f {{(pid=84696) create_target /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:410}} Apr 17 15:26:20 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-de849bf8-d9df-401e-9b04-0eb5d5f7d726 req-0fdbe885-b9a4-4b52-a62d-6737ac516ed4 tempest-AttachSCSIVolumeTestJSON-944917744 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:26:20 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-de849bf8-d9df-401e-9b04-0eb5d5f7d726 req-0fdbe885-b9a4-4b52-a62d-6737ac516ed4 tempest-AttachSCSIVolumeTestJSON-944917744 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:26:20 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-fd4f584b-d59b-4452-9b9d-4c5ce060bfcd req-96ff4b61-b76a-41d9-ac83-cf34bbbd176e tempest-TaggedAttachmentsTest-1309189231 None] Geting {'code': 404, 'error': {'class': 'opene.exceptions.ItemNotFoundError', 'message': 'Target iqn.2010-10.org.openstack:3c05f4f0-17a8-4f4c-9987-1fcb502ffd3a not exists.', 'url': 'http://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:3c05f4f0-17a8-4f4c-9987-1fcb502ffd3a'}, 'data': None} from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:3c05f4f0-17a8-4f4c-9987-1fcb502ffd3a {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:26:20 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-fd4f584b-d59b-4452-9b9d-4c5ce060bfcd req-96ff4b61-b76a-41d9-ac83-cf34bbbd176e tempest-TaggedAttachmentsTest-1309189231 None] create target and attach volume 3c05f4f0-17a8-4f4c-9987-1fcb502ffd3a to it {{(pid=84696) _create_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:837}} Apr 17 15:26:20 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [req-fd4f584b-d59b-4452-9b9d-4c5ce060bfcd req-96ff4b61-b76a-41d9-ac83-cf34bbbd176e tempest-TaggedAttachmentsTest-1309189231 None] create target iqn.2010-10.org.openstack:3c05f4f0-17a8-4f4c-9987-1fcb502ffd3a {{(pid=84696) create_target /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:410}} Apr 17 15:26:20 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-fd4f584b-d59b-4452-9b9d-4c5ce060bfcd req-96ff4b61-b76a-41d9-ac83-cf34bbbd176e tempest-TaggedAttachmentsTest-1309189231 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:26:20 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-fd4f584b-d59b-4452-9b9d-4c5ce060bfcd req-96ff4b61-b76a-41d9-ac83-cf34bbbd176e tempest-TaggedAttachmentsTest-1309189231 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:26:21 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-47322ce3-978b-4118-82f4-5d45e902b32e req-dbe2da4e-c498-4346-aa59-0d1da1e07c67 tempest-TestVolumeSwap-1714109745 None] Geting {'code': 201, 'error': None, 'data': {'block_size': 512, 'device_handler': 'vdisk_blockio', 'eui': '3338626565376231', 'lun': 0, 'mode': 'wt', 'name': 'v_66000c93-eb12-4c23-8a69-71e1aa6b521f', 'prod_id': 'Storage', 'scsi_id': '38bee7b12b313f31'}} from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:66000c93-eb12-4c23-8a69-71e1aa6b521f/luns {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:26:21 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [req-47322ce3-978b-4118-82f4-5d45e902b32e req-dbe2da4e-c498-4346-aa59-0d1da1e07c67 tempest-TestVolumeSwap-1714109745 None] add credentails to target iqn.2010-10.org.openstack:66000c93-eb12-4c23-8a69-71e1aa6b521f {{(pid=84696) create_target_user /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:474}} Apr 17 15:26:21 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-47322ce3-978b-4118-82f4-5d45e902b32e req-dbe2da4e-c498-4346-aa59-0d1da1e07c67 tempest-TestVolumeSwap-1714109745 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:66000c93-eb12-4c23-8a69-71e1aa6b521f/incoming-users {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:26:21 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-47322ce3-978b-4118-82f4-5d45e902b32e req-dbe2da4e-c498-4346-aa59-0d1da1e07c67 tempest-TestVolumeSwap-1714109745 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:66000c93-eb12-4c23-8a69-71e1aa6b521f/incoming-users {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:26:24 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-de849bf8-d9df-401e-9b04-0eb5d5f7d726 req-0fdbe885-b9a4-4b52-a62d-6737ac516ed4 tempest-AttachSCSIVolumeTestJSON-944917744 None] Geting {'code': 201, 'error': None, 'data': {'active': True, 'allow_ip': [], 'conflicted': False, 'deny_ip': [], 'incoming_users_active': True, 'name': 'iqn.2010-10.org.openstack:bf63d38e-00b8-46cb-acce-fda3fd9b6a9f', 'outgoing_user': None}} from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:26:24 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [req-de849bf8-d9df-401e-9b04-0eb5d5f7d726 req-0fdbe885-b9a4-4b52-a62d-6737ac516ed4 tempest-AttachSCSIVolumeTestJSON-944917744 None] atach volume v_bf63d38e-00b8-46cb-acce-fda3fd9b6a9f to target iqn.2010-10.org.openstack:bf63d38e-00b8-46cb-acce-fda3fd9b6a9f {{(pid=84696) attach_target_vol /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:576}} Apr 17 15:26:24 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-de849bf8-d9df-401e-9b04-0eb5d5f7d726 req-0fdbe885-b9a4-4b52-a62d-6737ac516ed4 tempest-AttachSCSIVolumeTestJSON-944917744 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:bf63d38e-00b8-46cb-acce-fda3fd9b6a9f/luns {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:26:24 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-de849bf8-d9df-401e-9b04-0eb5d5f7d726 req-0fdbe885-b9a4-4b52-a62d-6737ac516ed4 tempest-AttachSCSIVolumeTestJSON-944917744 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:bf63d38e-00b8-46cb-acce-fda3fd9b6a9f/luns {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:26:27 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-fd4f584b-d59b-4452-9b9d-4c5ce060bfcd req-96ff4b61-b76a-41d9-ac83-cf34bbbd176e tempest-TaggedAttachmentsTest-1309189231 None] Geting {'code': 201, 'error': None, 'data': {'active': True, 'allow_ip': [], 'conflicted': False, 'deny_ip': [], 'incoming_users_active': True, 'name': 'iqn.2010-10.org.openstack:3c05f4f0-17a8-4f4c-9987-1fcb502ffd3a', 'outgoing_user': None}} from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:26:27 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [req-fd4f584b-d59b-4452-9b9d-4c5ce060bfcd req-96ff4b61-b76a-41d9-ac83-cf34bbbd176e tempest-TaggedAttachmentsTest-1309189231 None] atach volume v_3c05f4f0-17a8-4f4c-9987-1fcb502ffd3a to target iqn.2010-10.org.openstack:3c05f4f0-17a8-4f4c-9987-1fcb502ffd3a {{(pid=84696) attach_target_vol /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:576}} Apr 17 15:26:27 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-fd4f584b-d59b-4452-9b9d-4c5ce060bfcd req-96ff4b61-b76a-41d9-ac83-cf34bbbd176e tempest-TaggedAttachmentsTest-1309189231 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:3c05f4f0-17a8-4f4c-9987-1fcb502ffd3a/luns {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:26:27 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-fd4f584b-d59b-4452-9b9d-4c5ce060bfcd req-96ff4b61-b76a-41d9-ac83-cf34bbbd176e tempest-TaggedAttachmentsTest-1309189231 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:3c05f4f0-17a8-4f4c-9987-1fcb502ffd3a/luns {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:26:27 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-47322ce3-978b-4118-82f4-5d45e902b32e req-dbe2da4e-c498-4346-aa59-0d1da1e07c67 tempest-TestVolumeSwap-1714109745 None] Geting {'code': 201, 'error': None, 'data': {'name': 'ygxXCJfw'}} from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:66000c93-eb12-4c23-8a69-71e1aa6b521f/incoming-users {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:26:27 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-47322ce3-978b-4118-82f4-5d45e902b32e req-dbe2da4e-c498-4346-aa59-0d1da1e07c67 tempest-TestVolumeSwap-1714109745 None] initialize_connection for 66000c93-eb12-4c23-8a69-71e1aa6b521f 10.0.0.210. {{(pid=84696) initialize_connection /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:1018}} Apr 17 15:26:27 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-47322ce3-978b-4118-82f4-5d45e902b32e req-dbe2da4e-c498-4346-aa59-0d1da1e07c67 tempest-TestVolumeSwap-1714109745 None] Connection info returned from driver {'target_iqn': 'iqn.2010-10.org.openstack:66000c93-eb12-4c23-8a69-71e1aa6b521f', 'target_portal': '172.16.0.220:3260', 'target_discovered': False, 'auth_method': 'CHAP', 'auth_username': 'ygxXCJfw', 'auth_password': '***', 'target_lun': 0, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False, 'driver_volume_type': 'iscsi', 'attachment_id': 'ed5eb659-cd00-475c-bec2-daa30716b18a'} {{(pid=84696) _connection_create /opt/stack/cinder/cinder/volume/manager.py:4875}} Apr 17 15:26:27 user cinder-volume[84696]: INFO cinder.volume.manager [req-47322ce3-978b-4118-82f4-5d45e902b32e req-dbe2da4e-c498-4346-aa59-0d1da1e07c67 tempest-TestVolumeSwap-1714109745 None] attachment_update completed successfully. Apr 17 15:26:28 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-de849bf8-d9df-401e-9b04-0eb5d5f7d726 req-0fdbe885-b9a4-4b52-a62d-6737ac516ed4 tempest-AttachSCSIVolumeTestJSON-944917744 None] Geting {'code': 201, 'error': None, 'data': {'block_size': 512, 'device_handler': 'vdisk_blockio', 'eui': '3932643732623665', 'lun': 0, 'mode': 'wt', 'name': 'v_bf63d38e-00b8-46cb-acce-fda3fd9b6a9f', 'prod_id': 'Storage', 'scsi_id': '92d72b6e9a809d92'}} from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:bf63d38e-00b8-46cb-acce-fda3fd9b6a9f/luns {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:26:28 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [req-de849bf8-d9df-401e-9b04-0eb5d5f7d726 req-0fdbe885-b9a4-4b52-a62d-6737ac516ed4 tempest-AttachSCSIVolumeTestJSON-944917744 None] add credentails to target iqn.2010-10.org.openstack:bf63d38e-00b8-46cb-acce-fda3fd9b6a9f {{(pid=84696) create_target_user /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:474}} Apr 17 15:26:28 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-de849bf8-d9df-401e-9b04-0eb5d5f7d726 req-0fdbe885-b9a4-4b52-a62d-6737ac516ed4 tempest-AttachSCSIVolumeTestJSON-944917744 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:bf63d38e-00b8-46cb-acce-fda3fd9b6a9f/incoming-users {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:26:28 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-de849bf8-d9df-401e-9b04-0eb5d5f7d726 req-0fdbe885-b9a4-4b52-a62d-6737ac516ed4 tempest-AttachSCSIVolumeTestJSON-944917744 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:bf63d38e-00b8-46cb-acce-fda3fd9b6a9f/incoming-users {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:26:29 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-dae38c21-d185-4d26-9a00-f5ceddfb2d2b req-8125de46-2f4d-4ee3-95cb-e845c4d93eb5 tempest-VolumesAdminNegativeTest-1285212149 None] terminate connection for 215a6a5b-25ca-48fa-a8e3-72416d7a39e2 {{(pid=84696) terminate_connection /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:1032}} Apr 17 15:26:29 user cinder-volume[84696]: INFO cinder.volume.manager [req-dae38c21-d185-4d26-9a00-f5ceddfb2d2b req-8125de46-2f4d-4ee3-95cb-e845c4d93eb5 tempest-VolumesAdminNegativeTest-1285212149 None] Terminate volume connection completed successfully. Apr 17 15:26:29 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-dae38c21-d185-4d26-9a00-f5ceddfb2d2b req-8125de46-2f4d-4ee3-95cb-e845c4d93eb5 tempest-VolumesAdminNegativeTest-1285212149 None] Deleting attachment 5b4d8d41-f27c-477b-9bbd-0437f5a2d910. {{(pid=84696) attachment_delete /opt/stack/cinder/cinder/volume/manager.py:5006}} Apr 17 15:26:29 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-dae38c21-d185-4d26-9a00-f5ceddfb2d2b req-8125de46-2f4d-4ee3-95cb-e845c4d93eb5 tempest-VolumesAdminNegativeTest-1285212149 None] remove_export for volume: 215a6a5b-25ca-48fa-a8e3-72416d7a39e2. {{(pid=84696) remove_export /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:718}} Apr 17 15:26:29 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-dae38c21-d185-4d26-9a00-f5ceddfb2d2b req-8125de46-2f4d-4ee3-95cb-e845c4d93eb5 tempest-VolumesAdminNegativeTest-1285212149 None] remove export {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:912}} Apr 17 15:26:29 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-dae38c21-d185-4d26-9a00-f5ceddfb2d2b req-8125de46-2f4d-4ee3-95cb-e845c4d93eb5 tempest-VolumesAdminNegativeTest-1285212149 None] detach volume:Volume(_name_id=None,admin_metadata=,attach_status='attached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:26:08Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesAdminNegativeTest-volume-850487146',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=215a6a5b-25ca-48fa-a8e3-72416d7a39e2,launched_at=2023-04-17T15:26:10Z,metadata=,migration_status=None,multiattach=False,previous_status=None,project_id='b334c5bc33f143c484518d9bd0e750dd',provider_auth='CHAP wIJnMDkU zdn9a38rfO7goh',provider_geometry=None,provider_id=None,provider_location='172.16.0.220:3260,1 iqn.2010-10.org.openstack:215a6a5b-25ca-48fa-a8e3-72416d7a39e2 0',replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:26:08Z,service_uuid='e628352e-c4cf-40eb-8389-cce40663026a',shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='detaching',terminated_at=None,updated_at=2023-04-17T15:26:28Z,use_quota=True,user_id='3dfadfd30c394b46abe75ec930007850',volume_attachment=,volume_type=,volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c) from target:iqn.2010-10.org.openstack:215a6a5b-25ca-48fa-a8e3-72416d7a39e2. {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:913}} Apr 17 15:26:29 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [req-dae38c21-d185-4d26-9a00-f5ceddfb2d2b req-8125de46-2f4d-4ee3-95cb-e845c4d93eb5 tempest-VolumesAdminNegativeTest-1285212149 None] detach volume v_215a6a5b-25ca-48fa-a8e3-72416d7a39e2 from target iqn.2010-10.org.openstack:215a6a5b-25ca-48fa-a8e3-72416d7a39e2 {{(pid=84696) detach_target_vol /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:606}} Apr 17 15:26:29 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-dae38c21-d185-4d26-9a00-f5ceddfb2d2b req-8125de46-2f4d-4ee3-95cb-e845c4d93eb5 tempest-VolumesAdminNegativeTest-1285212149 None] Sending pool request DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:215a6a5b-25ca-48fa-a8e3-72416d7a39e2/luns/v_215a6a5b-25ca-48fa-a8e3-72416d7a39e2 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:26:29 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-dae38c21-d185-4d26-9a00-f5ceddfb2d2b req-8125de46-2f4d-4ee3-95cb-e845c4d93eb5 tempest-VolumesAdminNegativeTest-1285212149 None] Sending DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:215a6a5b-25ca-48fa-a8e3-72416d7a39e2/luns/v_215a6a5b-25ca-48fa-a8e3-72416d7a39e2 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:26:30 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-fd4f584b-d59b-4452-9b9d-4c5ce060bfcd req-96ff4b61-b76a-41d9-ac83-cf34bbbd176e tempest-TaggedAttachmentsTest-1309189231 None] Geting {'code': 201, 'error': None, 'data': {'block_size': 512, 'device_handler': 'vdisk_blockio', 'eui': '6336303633356264', 'lun': 0, 'mode': 'wt', 'name': 'v_3c05f4f0-17a8-4f4c-9987-1fcb502ffd3a', 'prod_id': 'Storage', 'scsi_id': 'c60635bd02741eff'}} from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:3c05f4f0-17a8-4f4c-9987-1fcb502ffd3a/luns {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:26:30 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [req-fd4f584b-d59b-4452-9b9d-4c5ce060bfcd req-96ff4b61-b76a-41d9-ac83-cf34bbbd176e tempest-TaggedAttachmentsTest-1309189231 None] add credentails to target iqn.2010-10.org.openstack:3c05f4f0-17a8-4f4c-9987-1fcb502ffd3a {{(pid=84696) create_target_user /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:474}} Apr 17 15:26:30 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-fd4f584b-d59b-4452-9b9d-4c5ce060bfcd req-96ff4b61-b76a-41d9-ac83-cf34bbbd176e tempest-TaggedAttachmentsTest-1309189231 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:3c05f4f0-17a8-4f4c-9987-1fcb502ffd3a/incoming-users {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:26:30 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-fd4f584b-d59b-4452-9b9d-4c5ce060bfcd req-96ff4b61-b76a-41d9-ac83-cf34bbbd176e tempest-TaggedAttachmentsTest-1309189231 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:3c05f4f0-17a8-4f4c-9987-1fcb502ffd3a/incoming-users {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:26:30 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-dae38c21-d185-4d26-9a00-f5ceddfb2d2b req-8125de46-2f4d-4ee3-95cb-e845c4d93eb5 tempest-VolumesAdminNegativeTest-1285212149 None] Geting {'code': 204} from DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:215a6a5b-25ca-48fa-a8e3-72416d7a39e2/luns/v_215a6a5b-25ca-48fa-a8e3-72416d7a39e2 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:26:30 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-dae38c21-d185-4d26-9a00-f5ceddfb2d2b req-8125de46-2f4d-4ee3-95cb-e845c4d93eb5 tempest-VolumesAdminNegativeTest-1285212149 None] delete target: iqn.2010-10.org.openstack:215a6a5b-25ca-48fa-a8e3-72416d7a39e2 {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:930}} Apr 17 15:26:30 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [req-dae38c21-d185-4d26-9a00-f5ceddfb2d2b req-8125de46-2f4d-4ee3-95cb-e845c4d93eb5 tempest-VolumesAdminNegativeTest-1285212149 None] delete target iqn.2010-10.org.openstack:215a6a5b-25ca-48fa-a8e3-72416d7a39e2 {{(pid=84696) delete_target /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:442}} Apr 17 15:26:30 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-dae38c21-d185-4d26-9a00-f5ceddfb2d2b req-8125de46-2f4d-4ee3-95cb-e845c4d93eb5 tempest-VolumesAdminNegativeTest-1285212149 None] Sending pool request DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:215a6a5b-25ca-48fa-a8e3-72416d7a39e2 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:26:30 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-dae38c21-d185-4d26-9a00-f5ceddfb2d2b req-8125de46-2f4d-4ee3-95cb-e845c4d93eb5 tempest-VolumesAdminNegativeTest-1285212149 None] Sending DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:215a6a5b-25ca-48fa-a8e3-72416d7a39e2 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:26:31 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-de849bf8-d9df-401e-9b04-0eb5d5f7d726 req-0fdbe885-b9a4-4b52-a62d-6737ac516ed4 tempest-AttachSCSIVolumeTestJSON-944917744 None] Geting {'code': 201, 'error': None, 'data': {'name': 'xSVisnfg'}} from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:bf63d38e-00b8-46cb-acce-fda3fd9b6a9f/incoming-users {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:26:31 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-de849bf8-d9df-401e-9b04-0eb5d5f7d726 req-0fdbe885-b9a4-4b52-a62d-6737ac516ed4 tempest-AttachSCSIVolumeTestJSON-944917744 None] initialize_connection for bf63d38e-00b8-46cb-acce-fda3fd9b6a9f 10.0.0.210. {{(pid=84696) initialize_connection /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:1018}} Apr 17 15:26:31 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-de849bf8-d9df-401e-9b04-0eb5d5f7d726 req-0fdbe885-b9a4-4b52-a62d-6737ac516ed4 tempest-AttachSCSIVolumeTestJSON-944917744 None] Connection info returned from driver {'target_iqn': 'iqn.2010-10.org.openstack:bf63d38e-00b8-46cb-acce-fda3fd9b6a9f', 'target_portal': '172.16.0.220:3260', 'target_discovered': False, 'auth_method': 'CHAP', 'auth_username': 'xSVisnfg', 'auth_password': '***', 'target_lun': 0, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False, 'driver_volume_type': 'iscsi', 'attachment_id': 'f80a0217-be74-40ad-a93c-7295129cd655'} {{(pid=84696) _connection_create /opt/stack/cinder/cinder/volume/manager.py:4875}} Apr 17 15:26:31 user cinder-volume[84696]: INFO cinder.volume.manager [req-de849bf8-d9df-401e-9b04-0eb5d5f7d726 req-0fdbe885-b9a4-4b52-a62d-6737ac516ed4 tempest-AttachSCSIVolumeTestJSON-944917744 None] attachment_update completed successfully. Apr 17 15:26:31 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-fd4f584b-d59b-4452-9b9d-4c5ce060bfcd req-96ff4b61-b76a-41d9-ac83-cf34bbbd176e tempest-TaggedAttachmentsTest-1309189231 None] Geting {'code': 201, 'error': None, 'data': {'name': 'vjBFoMiS'}} from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:3c05f4f0-17a8-4f4c-9987-1fcb502ffd3a/incoming-users {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:26:31 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-fd4f584b-d59b-4452-9b9d-4c5ce060bfcd req-96ff4b61-b76a-41d9-ac83-cf34bbbd176e tempest-TaggedAttachmentsTest-1309189231 None] initialize_connection for 3c05f4f0-17a8-4f4c-9987-1fcb502ffd3a 10.0.0.210. {{(pid=84696) initialize_connection /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:1018}} Apr 17 15:26:31 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-fd4f584b-d59b-4452-9b9d-4c5ce060bfcd req-96ff4b61-b76a-41d9-ac83-cf34bbbd176e tempest-TaggedAttachmentsTest-1309189231 None] Connection info returned from driver {'target_iqn': 'iqn.2010-10.org.openstack:3c05f4f0-17a8-4f4c-9987-1fcb502ffd3a', 'target_portal': '172.16.0.220:3260', 'target_discovered': False, 'auth_method': 'CHAP', 'auth_username': 'vjBFoMiS', 'auth_password': '***', 'target_lun': 0, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False, 'driver_volume_type': 'iscsi', 'attachment_id': '3ada793a-9b05-4c8b-9dfb-27728f2edf3f'} {{(pid=84696) _connection_create /opt/stack/cinder/cinder/volume/manager.py:4875}} Apr 17 15:26:31 user cinder-volume[84696]: INFO cinder.volume.manager [req-fd4f584b-d59b-4452-9b9d-4c5ce060bfcd req-96ff4b61-b76a-41d9-ac83-cf34bbbd176e tempest-TaggedAttachmentsTest-1309189231 None] attachment_update completed successfully. Apr 17 15:26:31 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-dae38c21-d185-4d26-9a00-f5ceddfb2d2b req-8125de46-2f4d-4ee3-95cb-e845c4d93eb5 tempest-VolumesAdminNegativeTest-1285212149 None] Geting {'code': 204} from DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:215a6a5b-25ca-48fa-a8e3-72416d7a39e2 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:26:31 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [req-dae38c21-d185-4d26-9a00-f5ceddfb2d2b req-8125de46-2f4d-4ee3-95cb-e845c4d93eb5 tempest-VolumesAdminNegativeTest-1285212149 None] target iqn.2010-10.org.openstack:215a6a5b-25ca-48fa-a8e3-72416d7a39e2 deleted {{(pid=84696) delete_target /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:447}} Apr 17 15:26:33 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-b239e302-6fec-456c-b14a-e56af141016e req-50246045-33bf-4669-b8b9-131dfb9a20fc tempest-TestVolumeSwap-1210043716 None] create export for volume: 485ea70c-618d-4373-a78f-a460914fc095. {{(pid=84696) create_export /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:699}} Apr 17 15:26:33 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-b239e302-6fec-456c-b14a-e56af141016e req-50246045-33bf-4669-b8b9-131dfb9a20fc tempest-TestVolumeSwap-1210043716 None] ensure volume 485ea70c-618d-4373-a78f-a460914fc095 assigned to a proper target {{(pid=84696) _ensure_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:866}} Apr 17 15:26:33 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [req-b239e302-6fec-456c-b14a-e56af141016e req-50246045-33bf-4669-b8b9-131dfb9a20fc tempest-TestVolumeSwap-1210043716 None] check if targe iqn.2010-10.org.openstack:485ea70c-618d-4373-a78f-a460914fc095 exists {{(pid=84696) is_target /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:376}} Apr 17 15:26:33 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-b239e302-6fec-456c-b14a-e56af141016e req-50246045-33bf-4669-b8b9-131dfb9a20fc tempest-TestVolumeSwap-1210043716 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:485ea70c-618d-4373-a78f-a460914fc095 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:26:33 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-b239e302-6fec-456c-b14a-e56af141016e req-50246045-33bf-4669-b8b9-131dfb9a20fc tempest-TestVolumeSwap-1210043716 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:485ea70c-618d-4373-a78f-a460914fc095 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:26:33 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-b239e302-6fec-456c-b14a-e56af141016e req-50246045-33bf-4669-b8b9-131dfb9a20fc tempest-TestVolumeSwap-1210043716 None] Geting {'code': 404, 'error': {'class': 'opene.exceptions.ItemNotFoundError', 'message': 'Target iqn.2010-10.org.openstack:485ea70c-618d-4373-a78f-a460914fc095 not exists.', 'url': 'http://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:485ea70c-618d-4373-a78f-a460914fc095'}, 'data': None} from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:485ea70c-618d-4373-a78f-a460914fc095 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:26:33 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-b239e302-6fec-456c-b14a-e56af141016e req-50246045-33bf-4669-b8b9-131dfb9a20fc tempest-TestVolumeSwap-1210043716 None] create target and attach volume 485ea70c-618d-4373-a78f-a460914fc095 to it {{(pid=84696) _create_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:837}} Apr 17 15:26:33 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [req-b239e302-6fec-456c-b14a-e56af141016e req-50246045-33bf-4669-b8b9-131dfb9a20fc tempest-TestVolumeSwap-1210043716 None] create target iqn.2010-10.org.openstack:485ea70c-618d-4373-a78f-a460914fc095 {{(pid=84696) create_target /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:410}} Apr 17 15:26:33 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-b239e302-6fec-456c-b14a-e56af141016e req-50246045-33bf-4669-b8b9-131dfb9a20fc tempest-TestVolumeSwap-1210043716 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:26:33 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-b239e302-6fec-456c-b14a-e56af141016e req-50246045-33bf-4669-b8b9-131dfb9a20fc tempest-TestVolumeSwap-1210043716 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:26:33 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=58,cinder:UPDATE=11,cinder:DELETE=2 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:26:34 user cinder-volume[84696]: DEBUG cinder.coordination [None req-928c2c4b-8430-48c9-b964-08ab40043521 tempest-DeleteServersTestJSON-1458225442 None] Acquiring lock "cinder-a5f9d059-9f1b-4cf5-ae89-92bcf6c50326-delete_volume" by "delete_volume" {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:192}} Apr 17 15:26:34 user cinder-volume[84696]: DEBUG cinder.coordination [None req-928c2c4b-8430-48c9-b964-08ab40043521 tempest-DeleteServersTestJSON-1458225442 None] Lock "cinder-a5f9d059-9f1b-4cf5-ae89-92bcf6c50326-delete_volume" acquired by "delete_volume" :: waited 0.017s {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:196}} Apr 17 15:26:34 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-928c2c4b-8430-48c9-b964-08ab40043521 tempest-DeleteServersTestJSON-1458225442 None] remove_export for volume: a5f9d059-9f1b-4cf5-ae89-92bcf6c50326. {{(pid=84696) remove_export /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:718}} Apr 17 15:26:34 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-928c2c4b-8430-48c9-b964-08ab40043521 tempest-DeleteServersTestJSON-1458225442 None] remove export {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:912}} Apr 17 15:26:34 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-928c2c4b-8430-48c9-b964-08ab40043521 tempest-DeleteServersTestJSON-1458225442 None] detach volume:Volume(_name_id=None,admin_metadata={},attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:25:34Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-DeleteServersTestJSON-volume-1479973060',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=a5f9d059-9f1b-4cf5-ae89-92bcf6c50326,launched_at=2023-04-17T15:25:35Z,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='8aecd98c92d74f6b9879e6f57125ebde',provider_auth='CHAP rxEFJebi 7ein8k2dpHAS4V',provider_geometry=None,provider_id=None,provider_location='172.16.0.220:3260,1 iqn.2010-10.org.openstack:a5f9d059-9f1b-4cf5-ae89-92bcf6c50326 0',replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:25:34Z,service_uuid='e628352e-c4cf-40eb-8389-cce40663026a',shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='deleting',terminated_at=2023-04-17T15:26:35Z,updated_at=2023-04-17T15:26:35Z,use_quota=True,user_id='fec7c09a38ee413db724c659acc576ce',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c) from target:iqn.2010-10.org.openstack:a5f9d059-9f1b-4cf5-ae89-92bcf6c50326. {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:913}} Apr 17 15:26:34 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-928c2c4b-8430-48c9-b964-08ab40043521 tempest-DeleteServersTestJSON-1458225442 None] detach volume v_a5f9d059-9f1b-4cf5-ae89-92bcf6c50326 from target iqn.2010-10.org.openstack:a5f9d059-9f1b-4cf5-ae89-92bcf6c50326 {{(pid=84696) detach_target_vol /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:606}} Apr 17 15:26:34 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-928c2c4b-8430-48c9-b964-08ab40043521 tempest-DeleteServersTestJSON-1458225442 None] Sending pool request DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:a5f9d059-9f1b-4cf5-ae89-92bcf6c50326/luns/v_a5f9d059-9f1b-4cf5-ae89-92bcf6c50326 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:26:34 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-928c2c4b-8430-48c9-b964-08ab40043521 tempest-DeleteServersTestJSON-1458225442 None] Sending DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:a5f9d059-9f1b-4cf5-ae89-92bcf6c50326/luns/v_a5f9d059-9f1b-4cf5-ae89-92bcf6c50326 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:26:34 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-709e8d1d-5a28-4240-8e7a-adfedc29ea50 req-559d7de6-cd8d-4397-90b5-72ff22badc88 tempest-AttachSCSIVolumeTestJSON-944917744 None] terminate connection for bf63d38e-00b8-46cb-acce-fda3fd9b6a9f {{(pid=84696) terminate_connection /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:1032}} Apr 17 15:26:34 user cinder-volume[84696]: INFO cinder.volume.manager [req-709e8d1d-5a28-4240-8e7a-adfedc29ea50 req-559d7de6-cd8d-4397-90b5-72ff22badc88 tempest-AttachSCSIVolumeTestJSON-944917744 None] Terminate volume connection completed successfully. Apr 17 15:26:34 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-709e8d1d-5a28-4240-8e7a-adfedc29ea50 req-559d7de6-cd8d-4397-90b5-72ff22badc88 tempest-AttachSCSIVolumeTestJSON-944917744 None] Deleting attachment f80a0217-be74-40ad-a93c-7295129cd655. {{(pid=84696) attachment_delete /opt/stack/cinder/cinder/volume/manager.py:5006}} Apr 17 15:26:34 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-709e8d1d-5a28-4240-8e7a-adfedc29ea50 req-559d7de6-cd8d-4397-90b5-72ff22badc88 tempest-AttachSCSIVolumeTestJSON-944917744 None] remove_export for volume: bf63d38e-00b8-46cb-acce-fda3fd9b6a9f. {{(pid=84696) remove_export /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:718}} Apr 17 15:26:34 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-709e8d1d-5a28-4240-8e7a-adfedc29ea50 req-559d7de6-cd8d-4397-90b5-72ff22badc88 tempest-AttachSCSIVolumeTestJSON-944917744 None] remove export {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:912}} Apr 17 15:26:34 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-709e8d1d-5a28-4240-8e7a-adfedc29ea50 req-559d7de6-cd8d-4397-90b5-72ff22badc88 tempest-AttachSCSIVolumeTestJSON-944917744 None] detach volume:Volume(_name_id=None,admin_metadata=,attach_status='attached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:26:14Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-AttachSCSIVolumeTestJSON-volume-1764995674',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=bf63d38e-00b8-46cb-acce-fda3fd9b6a9f,launched_at=2023-04-17T15:26:17Z,metadata=,migration_status=None,multiattach=False,previous_status=None,project_id='0fd4ceec29874bc38e8da456cab2048d',provider_auth='CHAP xSVisnfg 8KuvVtUOwfxz5X',provider_geometry=None,provider_id=None,provider_location='172.16.0.220:3260,1 iqn.2010-10.org.openstack:bf63d38e-00b8-46cb-acce-fda3fd9b6a9f 0',replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:26:14Z,service_uuid='e628352e-c4cf-40eb-8389-cce40663026a',shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='detaching',terminated_at=None,updated_at=2023-04-17T15:26:34Z,use_quota=True,user_id='b1734894909e44118bd60e233dc501a0',volume_attachment=,volume_type=,volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c) from target:iqn.2010-10.org.openstack:bf63d38e-00b8-46cb-acce-fda3fd9b6a9f. {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:913}} Apr 17 15:26:34 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [req-709e8d1d-5a28-4240-8e7a-adfedc29ea50 req-559d7de6-cd8d-4397-90b5-72ff22badc88 tempest-AttachSCSIVolumeTestJSON-944917744 None] detach volume v_bf63d38e-00b8-46cb-acce-fda3fd9b6a9f from target iqn.2010-10.org.openstack:bf63d38e-00b8-46cb-acce-fda3fd9b6a9f {{(pid=84696) detach_target_vol /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:606}} Apr 17 15:26:34 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-709e8d1d-5a28-4240-8e7a-adfedc29ea50 req-559d7de6-cd8d-4397-90b5-72ff22badc88 tempest-AttachSCSIVolumeTestJSON-944917744 None] Sending pool request DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:bf63d38e-00b8-46cb-acce-fda3fd9b6a9f/luns/v_bf63d38e-00b8-46cb-acce-fda3fd9b6a9f {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:26:34 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-709e8d1d-5a28-4240-8e7a-adfedc29ea50 req-559d7de6-cd8d-4397-90b5-72ff22badc88 tempest-AttachSCSIVolumeTestJSON-944917744 None] Sending DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:bf63d38e-00b8-46cb-acce-fda3fd9b6a9f/luns/v_bf63d38e-00b8-46cb-acce-fda3fd9b6a9f {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:26:36 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-8aa52899-63af-4722-b9ff-f7493ed0a17e req-5e0be7d2-298e-46a4-9ff5-3a608e3ec33a tempest-TaggedAttachmentsTest-1309189231 None] terminate connection for 3c05f4f0-17a8-4f4c-9987-1fcb502ffd3a {{(pid=84696) terminate_connection /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:1032}} Apr 17 15:26:36 user cinder-volume[84696]: INFO cinder.volume.manager [req-8aa52899-63af-4722-b9ff-f7493ed0a17e req-5e0be7d2-298e-46a4-9ff5-3a608e3ec33a tempest-TaggedAttachmentsTest-1309189231 None] Terminate volume connection completed successfully. Apr 17 15:26:36 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-8aa52899-63af-4722-b9ff-f7493ed0a17e req-5e0be7d2-298e-46a4-9ff5-3a608e3ec33a tempest-TaggedAttachmentsTest-1309189231 None] Deleting attachment 3ada793a-9b05-4c8b-9dfb-27728f2edf3f. {{(pid=84696) attachment_delete /opt/stack/cinder/cinder/volume/manager.py:5006}} Apr 17 15:26:36 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-8aa52899-63af-4722-b9ff-f7493ed0a17e req-5e0be7d2-298e-46a4-9ff5-3a608e3ec33a tempest-TaggedAttachmentsTest-1309189231 None] remove_export for volume: 3c05f4f0-17a8-4f4c-9987-1fcb502ffd3a. {{(pid=84696) remove_export /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:718}} Apr 17 15:26:36 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-8aa52899-63af-4722-b9ff-f7493ed0a17e req-5e0be7d2-298e-46a4-9ff5-3a608e3ec33a tempest-TaggedAttachmentsTest-1309189231 None] remove export {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:912}} Apr 17 15:26:36 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-8aa52899-63af-4722-b9ff-f7493ed0a17e req-5e0be7d2-298e-46a4-9ff5-3a608e3ec33a tempest-TaggedAttachmentsTest-1309189231 None] detach volume:Volume(_name_id=None,admin_metadata=,attach_status='attached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:24:14Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TaggedAttachmentsTest-volume-2098521363',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=3c05f4f0-17a8-4f4c-9987-1fcb502ffd3a,launched_at=2023-04-17T15:24:17Z,metadata=,migration_status=None,multiattach=False,previous_status=None,project_id='9dfa73323c3c4c689f4012505d8b4833',provider_auth='CHAP vjBFoMiS JvaW8BYGPNwxzR',provider_geometry=None,provider_id=None,provider_location='172.16.0.220:3260,1 iqn.2010-10.org.openstack:3c05f4f0-17a8-4f4c-9987-1fcb502ffd3a 0',replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:24:15Z,service_uuid='e628352e-c4cf-40eb-8389-cce40663026a',shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='detaching',terminated_at=None,updated_at=2023-04-17T15:26:35Z,use_quota=True,user_id='f081ecb75ec14053b68d74cacf61c802',volume_attachment=,volume_type=,volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c) from target:iqn.2010-10.org.openstack:3c05f4f0-17a8-4f4c-9987-1fcb502ffd3a. {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:913}} Apr 17 15:26:36 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [req-8aa52899-63af-4722-b9ff-f7493ed0a17e req-5e0be7d2-298e-46a4-9ff5-3a608e3ec33a tempest-TaggedAttachmentsTest-1309189231 None] detach volume v_3c05f4f0-17a8-4f4c-9987-1fcb502ffd3a from target iqn.2010-10.org.openstack:3c05f4f0-17a8-4f4c-9987-1fcb502ffd3a {{(pid=84696) detach_target_vol /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:606}} Apr 17 15:26:36 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-8aa52899-63af-4722-b9ff-f7493ed0a17e req-5e0be7d2-298e-46a4-9ff5-3a608e3ec33a tempest-TaggedAttachmentsTest-1309189231 None] Sending pool request DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:3c05f4f0-17a8-4f4c-9987-1fcb502ffd3a/luns/v_3c05f4f0-17a8-4f4c-9987-1fcb502ffd3a {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:26:36 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-8aa52899-63af-4722-b9ff-f7493ed0a17e req-5e0be7d2-298e-46a4-9ff5-3a608e3ec33a tempest-TaggedAttachmentsTest-1309189231 None] Sending DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:3c05f4f0-17a8-4f4c-9987-1fcb502ffd3a/luns/v_3c05f4f0-17a8-4f4c-9987-1fcb502ffd3a {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:26:36 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-b239e302-6fec-456c-b14a-e56af141016e req-50246045-33bf-4669-b8b9-131dfb9a20fc tempest-TestVolumeSwap-1210043716 None] Geting {'code': 201, 'error': None, 'data': {'active': True, 'allow_ip': [], 'conflicted': False, 'deny_ip': [], 'incoming_users_active': True, 'name': 'iqn.2010-10.org.openstack:485ea70c-618d-4373-a78f-a460914fc095', 'outgoing_user': None}} from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:26:36 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [req-b239e302-6fec-456c-b14a-e56af141016e req-50246045-33bf-4669-b8b9-131dfb9a20fc tempest-TestVolumeSwap-1210043716 None] atach volume v_485ea70c-618d-4373-a78f-a460914fc095 to target iqn.2010-10.org.openstack:485ea70c-618d-4373-a78f-a460914fc095 {{(pid=84696) attach_target_vol /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:576}} Apr 17 15:26:36 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-b239e302-6fec-456c-b14a-e56af141016e req-50246045-33bf-4669-b8b9-131dfb9a20fc tempest-TestVolumeSwap-1210043716 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:485ea70c-618d-4373-a78f-a460914fc095/luns {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:26:36 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-b239e302-6fec-456c-b14a-e56af141016e req-50246045-33bf-4669-b8b9-131dfb9a20fc tempest-TestVolumeSwap-1210043716 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:485ea70c-618d-4373-a78f-a460914fc095/luns {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:26:36 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-8aa52899-63af-4722-b9ff-f7493ed0a17e req-5e0be7d2-298e-46a4-9ff5-3a608e3ec33a tempest-TaggedAttachmentsTest-1309189231 None] Geting {'code': 204} from DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:3c05f4f0-17a8-4f4c-9987-1fcb502ffd3a/luns/v_3c05f4f0-17a8-4f4c-9987-1fcb502ffd3a {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:26:36 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-8aa52899-63af-4722-b9ff-f7493ed0a17e req-5e0be7d2-298e-46a4-9ff5-3a608e3ec33a tempest-TaggedAttachmentsTest-1309189231 None] delete target: iqn.2010-10.org.openstack:3c05f4f0-17a8-4f4c-9987-1fcb502ffd3a {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:930}} Apr 17 15:26:36 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [req-8aa52899-63af-4722-b9ff-f7493ed0a17e req-5e0be7d2-298e-46a4-9ff5-3a608e3ec33a tempest-TaggedAttachmentsTest-1309189231 None] delete target iqn.2010-10.org.openstack:3c05f4f0-17a8-4f4c-9987-1fcb502ffd3a {{(pid=84696) delete_target /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:442}} Apr 17 15:26:36 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-8aa52899-63af-4722-b9ff-f7493ed0a17e req-5e0be7d2-298e-46a4-9ff5-3a608e3ec33a tempest-TaggedAttachmentsTest-1309189231 None] Sending pool request DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:3c05f4f0-17a8-4f4c-9987-1fcb502ffd3a {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:26:36 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-8aa52899-63af-4722-b9ff-f7493ed0a17e req-5e0be7d2-298e-46a4-9ff5-3a608e3ec33a tempest-TaggedAttachmentsTest-1309189231 None] Sending DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:3c05f4f0-17a8-4f4c-9987-1fcb502ffd3a {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:26:36 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-709e8d1d-5a28-4240-8e7a-adfedc29ea50 req-559d7de6-cd8d-4397-90b5-72ff22badc88 tempest-AttachSCSIVolumeTestJSON-944917744 None] Geting {'code': 204} from DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:bf63d38e-00b8-46cb-acce-fda3fd9b6a9f/luns/v_bf63d38e-00b8-46cb-acce-fda3fd9b6a9f {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:26:36 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-709e8d1d-5a28-4240-8e7a-adfedc29ea50 req-559d7de6-cd8d-4397-90b5-72ff22badc88 tempest-AttachSCSIVolumeTestJSON-944917744 None] delete target: iqn.2010-10.org.openstack:bf63d38e-00b8-46cb-acce-fda3fd9b6a9f {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:930}} Apr 17 15:26:36 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [req-709e8d1d-5a28-4240-8e7a-adfedc29ea50 req-559d7de6-cd8d-4397-90b5-72ff22badc88 tempest-AttachSCSIVolumeTestJSON-944917744 None] delete target iqn.2010-10.org.openstack:bf63d38e-00b8-46cb-acce-fda3fd9b6a9f {{(pid=84696) delete_target /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:442}} Apr 17 15:26:36 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-709e8d1d-5a28-4240-8e7a-adfedc29ea50 req-559d7de6-cd8d-4397-90b5-72ff22badc88 tempest-AttachSCSIVolumeTestJSON-944917744 None] Sending pool request DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:bf63d38e-00b8-46cb-acce-fda3fd9b6a9f {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:26:36 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-709e8d1d-5a28-4240-8e7a-adfedc29ea50 req-559d7de6-cd8d-4397-90b5-72ff22badc88 tempest-AttachSCSIVolumeTestJSON-944917744 None] Sending DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:bf63d38e-00b8-46cb-acce-fda3fd9b6a9f {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:26:37 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-928c2c4b-8430-48c9-b964-08ab40043521 tempest-DeleteServersTestJSON-1458225442 None] Geting {'code': 404, 'error': {'class': 'opene.exceptions.ItemNotFoundError', 'message': 'Target iqn.2010-10.org.openstack:a5f9d059-9f1b-4cf5-ae89-92bcf6c50326 not exists.', 'url': 'http://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:a5f9d059-9f1b-4cf5-ae89-92bcf6c50326/luns/v_a5f9d059-9f1b-4cf5-ae89-92bcf6c50326'}, 'data': None} from DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:a5f9d059-9f1b-4cf5-ae89-92bcf6c50326/luns/v_a5f9d059-9f1b-4cf5-ae89-92bcf6c50326 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:26:37 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-928c2c4b-8430-48c9-b964-08ab40043521 tempest-DeleteServersTestJSON-1458225442 None] failed to remove resource iqn.2010-10.org.openstack:a5f9d059-9f1b-4cf5-ae89-92bcf6c50326 because of JDSS resource v_a5f9d059-9f1b-4cf5-ae89-92bcf6c50326 DNE. {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:920}} Apr 17 15:26:37 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-928c2c4b-8430-48c9-b964-08ab40043521 tempest-DeleteServersTestJSON-1458225442 None] delete target: iqn.2010-10.org.openstack:a5f9d059-9f1b-4cf5-ae89-92bcf6c50326 {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:930}} Apr 17 15:26:37 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-928c2c4b-8430-48c9-b964-08ab40043521 tempest-DeleteServersTestJSON-1458225442 None] delete target iqn.2010-10.org.openstack:a5f9d059-9f1b-4cf5-ae89-92bcf6c50326 {{(pid=84696) delete_target /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:442}} Apr 17 15:26:37 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-928c2c4b-8430-48c9-b964-08ab40043521 tempest-DeleteServersTestJSON-1458225442 None] Sending pool request DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:a5f9d059-9f1b-4cf5-ae89-92bcf6c50326 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:26:37 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-928c2c4b-8430-48c9-b964-08ab40043521 tempest-DeleteServersTestJSON-1458225442 None] Sending DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:a5f9d059-9f1b-4cf5-ae89-92bcf6c50326 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:26:37 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-8aa52899-63af-4722-b9ff-f7493ed0a17e req-5e0be7d2-298e-46a4-9ff5-3a608e3ec33a tempest-TaggedAttachmentsTest-1309189231 None] Geting {'code': 204} from DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:3c05f4f0-17a8-4f4c-9987-1fcb502ffd3a {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:26:37 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [req-8aa52899-63af-4722-b9ff-f7493ed0a17e req-5e0be7d2-298e-46a4-9ff5-3a608e3ec33a tempest-TaggedAttachmentsTest-1309189231 None] target iqn.2010-10.org.openstack:3c05f4f0-17a8-4f4c-9987-1fcb502ffd3a deleted {{(pid=84696) delete_target /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:447}} Apr 17 15:26:38 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-b239e302-6fec-456c-b14a-e56af141016e req-50246045-33bf-4669-b8b9-131dfb9a20fc tempest-TestVolumeSwap-1210043716 None] Geting {'code': 201, 'error': None, 'data': {'block_size': 512, 'device_handler': 'vdisk_blockio', 'eui': '6231366632393535', 'lun': 0, 'mode': 'wt', 'name': 'v_485ea70c-618d-4373-a78f-a460914fc095', 'prod_id': 'Storage', 'scsi_id': 'b16f29557ea06ac5'}} from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:485ea70c-618d-4373-a78f-a460914fc095/luns {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:26:38 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [req-b239e302-6fec-456c-b14a-e56af141016e req-50246045-33bf-4669-b8b9-131dfb9a20fc tempest-TestVolumeSwap-1210043716 None] add credentails to target iqn.2010-10.org.openstack:485ea70c-618d-4373-a78f-a460914fc095 {{(pid=84696) create_target_user /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:474}} Apr 17 15:26:38 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-b239e302-6fec-456c-b14a-e56af141016e req-50246045-33bf-4669-b8b9-131dfb9a20fc tempest-TestVolumeSwap-1210043716 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:485ea70c-618d-4373-a78f-a460914fc095/incoming-users {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:26:38 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-b239e302-6fec-456c-b14a-e56af141016e req-50246045-33bf-4669-b8b9-131dfb9a20fc tempest-TestVolumeSwap-1210043716 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:485ea70c-618d-4373-a78f-a460914fc095/incoming-users {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:26:38 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-709e8d1d-5a28-4240-8e7a-adfedc29ea50 req-559d7de6-cd8d-4397-90b5-72ff22badc88 tempest-AttachSCSIVolumeTestJSON-944917744 None] Geting {'code': 204} from DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:bf63d38e-00b8-46cb-acce-fda3fd9b6a9f {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:26:38 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [req-709e8d1d-5a28-4240-8e7a-adfedc29ea50 req-559d7de6-cd8d-4397-90b5-72ff22badc88 tempest-AttachSCSIVolumeTestJSON-944917744 None] target iqn.2010-10.org.openstack:bf63d38e-00b8-46cb-acce-fda3fd9b6a9f deleted {{(pid=84696) delete_target /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:447}} Apr 17 15:26:38 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-928c2c4b-8430-48c9-b964-08ab40043521 tempest-DeleteServersTestJSON-1458225442 None] Geting {'code': 404, 'error': {'class': 'opene.exceptions.ItemNotFoundError', 'message': 'Target iqn.2010-10.org.openstack:a5f9d059-9f1b-4cf5-ae89-92bcf6c50326 not exists.', 'url': 'http://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:a5f9d059-9f1b-4cf5-ae89-92bcf6c50326'}, 'data': None} from DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:a5f9d059-9f1b-4cf5-ae89-92bcf6c50326 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:26:38 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-928c2c4b-8430-48c9-b964-08ab40043521 tempest-DeleteServersTestJSON-1458225442 None] failed to remove resource iqn.2010-10.org.openstack:a5f9d059-9f1b-4cf5-ae89-92bcf6c50326 because of JDSS resource iqn.2010-10.org.openstack:a5f9d059-9f1b-4cf5-ae89-92bcf6c50326 DNE. {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:935}} Apr 17 15:26:38 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-928c2c4b-8430-48c9-b964-08ab40043521 tempest-DeleteServersTestJSON-1458225442 None] deleating volume v_a5f9d059-9f1b-4cf5-ae89-92bcf6c50326 {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 15:26:38 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-928c2c4b-8430-48c9-b964-08ab40043521 tempest-DeleteServersTestJSON-1458225442 None] get snapshots for volume v_a5f9d059-9f1b-4cf5-ae89-92bcf6c50326 {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 15:26:38 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-928c2c4b-8430-48c9-b964-08ab40043521 tempest-DeleteServersTestJSON-1458225442 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_a5f9d059-9f1b-4cf5-ae89-92bcf6c50326/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:26:38 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-928c2c4b-8430-48c9-b964-08ab40043521 tempest-DeleteServersTestJSON-1458225442 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_a5f9d059-9f1b-4cf5-ae89-92bcf6c50326/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:26:39 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-b239e302-6fec-456c-b14a-e56af141016e req-50246045-33bf-4669-b8b9-131dfb9a20fc tempest-TestVolumeSwap-1210043716 None] Geting {'code': 201, 'error': None, 'data': {'name': 'WBubQvJS'}} from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:485ea70c-618d-4373-a78f-a460914fc095/incoming-users {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:26:39 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=72,cinder:UPDATE=17 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:26:39 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-b239e302-6fec-456c-b14a-e56af141016e req-50246045-33bf-4669-b8b9-131dfb9a20fc tempest-TestVolumeSwap-1210043716 None] initialize_connection for 485ea70c-618d-4373-a78f-a460914fc095 10.0.0.210. {{(pid=84696) initialize_connection /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:1018}} Apr 17 15:26:39 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-b239e302-6fec-456c-b14a-e56af141016e req-50246045-33bf-4669-b8b9-131dfb9a20fc tempest-TestVolumeSwap-1210043716 None] Connection info returned from driver {'target_iqn': 'iqn.2010-10.org.openstack:485ea70c-618d-4373-a78f-a460914fc095', 'target_portal': '172.16.0.220:3260', 'target_discovered': False, 'auth_method': 'CHAP', 'auth_username': 'WBubQvJS', 'auth_password': '***', 'target_lun': 0, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False, 'driver_volume_type': 'iscsi', 'attachment_id': '9a8c6327-030a-41e4-98c6-a23b982c5ee9'} {{(pid=84696) _connection_create /opt/stack/cinder/cinder/volume/manager.py:4875}} Apr 17 15:26:39 user cinder-volume[84696]: INFO cinder.volume.manager [req-b239e302-6fec-456c-b14a-e56af141016e req-50246045-33bf-4669-b8b9-131dfb9a20fc tempest-TestVolumeSwap-1210043716 None] attachment_update completed successfully. Apr 17 15:26:39 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-928c2c4b-8430-48c9-b964-08ab40043521 tempest-DeleteServersTestJSON-1458225442 None] Geting {'code': 200, 'error': None, 'data': {'results': 0, 'entries': []}} from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_a5f9d059-9f1b-4cf5-ae89-92bcf6c50326/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:26:39 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-928c2c4b-8430-48c9-b964-08ab40043521 tempest-DeleteServersTestJSON-1458225442 None] get volume v_a5f9d059-9f1b-4cf5-ae89-92bcf6c50326 info {{(pid=84696) get_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:239}} Apr 17 15:26:39 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-928c2c4b-8430-48c9-b964-08ab40043521 tempest-DeleteServersTestJSON-1458225442 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_a5f9d059-9f1b-4cf5-ae89-92bcf6c50326 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:26:39 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-928c2c4b-8430-48c9-b964-08ab40043521 tempest-DeleteServersTestJSON-1458225442 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_a5f9d059-9f1b-4cf5-ae89-92bcf6c50326 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:26:39 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-928c2c4b-8430-48c9-b964-08ab40043521 tempest-DeleteServersTestJSON-1458225442 None] Geting {'code': 200, 'error': None, 'data': {'origin': None, 'relatime': None, 'acltype': None, 'vscan': None, 'full_name': 'Pool-0/v_a5f9d059-9f1b-4cf5-ae89-92bcf6c50326', 'userrefs': None, 'primarycache': 'all', 'logbias': 'latency', 'creation': '1681746528', 'sync': 'always', 'is_clone': False, 'dedup': 'off', 'sharenfs': None, 'receive_resume_token': None, 'volsize': '1073741824', 'referenced': '57344', 'sharesmb': None, 'createtxg': '188039', 'reservation': '0', 'scontext': None, 'mountpoint': None, 'casesensitivity': None, 'guid': '6826203426898781113', 'usedbyrefreservation': '0', 'dnodesize': None, 'written': '57344', 'logicalused': '28672', 'compressratio': '1.00', 'rootcontext': 'none', 'default_scsi_id': '9fa1f1cf9f3015b5', 'type': 'volume', 'compression': 'lz4', 'snapdir': None, 'overlay': None, 'encryption': 'off', 'xattr': None, 'volmode': 'default', 'copies': '1', 'snapshot_limit': '18446744073709551615', 'aclinherit': None, 'defcontext': 'none', 'readonly': 'off', 'version': None, 'recordsize': None, 'filesystem_limit': None, 'mounted': None, 'mlslabel': 'none', 'secondarycache': 'all', 'refreservation': '0', 'available': '113889226752', 'san:volume_id': '9fa1f1cf9f3015b5688aa834d50fd75e5cb9294f', 'encryptionroot': None, 'exec': None, 'refquota': None, 'refcompressratio': '1.00', 'quota': None, 'utf8only': None, 'keylocation': 'none', 'snapdev': 'hidden', 'snapshot_count': '18446744073709551615', 'fscontext': 'none', 'clones': None, 'canmount': None, 'keystatus': None, 'atime': None, 'usedbysnapshots': '0', 'normalization': None, 'usedbychildren': '0', 'volblocksize': '131072', 'usedbydataset': '57344', 'objsetid': '122560', 'name': 'v_a5f9d059-9f1b-4cf5-ae89-92bcf6c50326', 'defer_destroy': None, 'pbkdf2iters': '0', 'checksum': 'on', 'redundant_metadata': 'all', 'filesystem_count': None, 'devices': None, 'keyformat': 'none', 'setuid': None, 'used': '57344', 'logicalreferenced': '28672', 'context': 'none', 'zoned': None, 'nbmand': None}} from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_a5f9d059-9f1b-4cf5-ae89-92bcf6c50326 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:26:39 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-928c2c4b-8430-48c9-b964-08ab40043521 tempest-DeleteServersTestJSON-1458225442 None] delete volume:v_a5f9d059-9f1b-4cf5-ae89-92bcf6c50326 recursively children:{'force_umount': True} {{(pid=84696) delete_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:335}} Apr 17 15:26:39 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-928c2c4b-8430-48c9-b964-08ab40043521 tempest-DeleteServersTestJSON-1458225442 None] Sending pool request DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_a5f9d059-9f1b-4cf5-ae89-92bcf6c50326 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:26:39 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-928c2c4b-8430-48c9-b964-08ab40043521 tempest-DeleteServersTestJSON-1458225442 None] Sending DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_a5f9d059-9f1b-4cf5-ae89-92bcf6c50326 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:26:42 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-928c2c4b-8430-48c9-b964-08ab40043521 tempest-DeleteServersTestJSON-1458225442 None] Geting {'code': 204} from DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_a5f9d059-9f1b-4cf5-ae89-92bcf6c50326 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:26:42 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-928c2c4b-8430-48c9-b964-08ab40043521 tempest-DeleteServersTestJSON-1458225442 None] volume v_a5f9d059-9f1b-4cf5-ae89-92bcf6c50326 deleted {{(pid=84696) delete_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:346}} Apr 17 15:26:42 user cinder-volume[84696]: DEBUG cinder.quota [None req-928c2c4b-8430-48c9-b964-08ab40043521 tempest-DeleteServersTestJSON-1458225442 None] Created reservations ['02713f5b-11f0-48db-bae0-ccfb253066a7', '0bd5aff6-dc4c-452d-a275-4c6fd4587e5b', 'dc0dbf1f-30e0-4c72-a8e3-96f5c8ba4d4b', '3ebbe617-3c01-437f-9f66-9d6745cd3d52'] {{(pid=84696) reserve /opt/stack/cinder/cinder/quota.py:755}} Apr 17 15:26:42 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-928c2c4b-8430-48c9-b964-08ab40043521 tempest-DeleteServersTestJSON-1458225442 None] Updating volume stats {{(pid=84696) _update_volume_stats /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:724}} Apr 17 15:26:42 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-928c2c4b-8430-48c9-b964-08ab40043521 tempest-DeleteServersTestJSON-1458225442 None] Get pool Pool-0 fsprops {{(pid=84696) get_pool_stats /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:894}} Apr 17 15:26:42 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-928c2c4b-8430-48c9-b964-08ab40043521 tempest-DeleteServersTestJSON-1458225442 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:26:42 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-928c2c4b-8430-48c9-b964-08ab40043521 tempest-DeleteServersTestJSON-1458225442 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:26:43 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-928c2c4b-8430-48c9-b964-08ab40043521 tempest-DeleteServersTestJSON-1458225442 None] Geting {'code': 200, 'error': None, 'data': {'available': '113889308672', 'status': 30, 'import_status': {'import_steps': {'san_setup': True, 'vip_setup': True, 'zfs_import': True, 'mount_system_volume': True, 'nas_setup': True}, 'import_successful': True}, 'scan': None, 'encryption': {'enabled': False}, 'deduplication_rate': '1.00', 'sysvol_upgrade': 'not_needed', 'vdevs': [{'name': 'wwn-0x6000c296b793bf526a107a9d823e947a', 'iostats': {'read': '0', 'write': '0', 'chksum': '0'}, 'disks': [{'origin': 'local', 'slot': '', 'led': 'off', 'name': 'sdn', 'iostats': {'read': '0', 'write': '0', 'chksum': '0'}, 'alias': '', 'health': 'ONLINE', 'sn': '6000c296b793bf526a107a9d823e947a', 'trim_data': {'status': 'trim_unsupported', 'progress': None, 'trim_time': None}, 'path': 'pci-0000:03:00.0-scsi-0:0:14:0', 'model': 'Virtual disk', 'id': 'wwn-0x6000c296b793bf526a107a9d823e947a', 'size': 128849018880}], 'health': 'ONLINE', 'vdev_replacings': [], 'vdev_spares': [], 'type': ''}], 'id': '14181317334440137330', 'health': 'ONLINE', 'iostats': {'read': '0', 'write': '0', 'chksum': '0'}, 'operation': 'none', 'size': '127775277056', 'autotrim': False, 'name': 'Pool-0'}} from GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:26:43 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-928c2c4b-8430-48c9-b964-08ab40043521 tempest-DeleteServersTestJSON-1458225442 None] Total capacity: 119, Free 106. {{(pid=84696) _update_volume_stats /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:757}} Apr 17 15:26:43 user cinder-volume[84696]: DEBUG cinder.manager [None req-928c2c4b-8430-48c9-b964-08ab40043521 tempest-DeleteServersTestJSON-1458225442 None] Notifying Schedulers of capabilities ... {{(pid=84696) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 17 15:26:43 user cinder-volume[84696]: INFO cinder.volume.manager [None req-928c2c4b-8430-48c9-b964-08ab40043521 tempest-DeleteServersTestJSON-1458225442 None] Deleted volume successfully. Apr 17 15:26:43 user cinder-volume[84696]: DEBUG cinder.coordination [None req-928c2c4b-8430-48c9-b964-08ab40043521 tempest-DeleteServersTestJSON-1458225442 None] Lock "cinder-a5f9d059-9f1b-4cf5-ae89-92bcf6c50326-delete_volume" released by "delete_volume" :: held 9.098s {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:207}} Apr 17 15:26:45 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-b239e302-6fec-456c-b14a-e56af141016e req-bbf2c75a-7454-466b-b043-f4972e68c894 tempest-TestVolumeSwap-1210043716 None] terminate connection for 66000c93-eb12-4c23-8a69-71e1aa6b521f {{(pid=84696) terminate_connection /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:1032}} Apr 17 15:26:45 user cinder-volume[84696]: INFO cinder.volume.manager [req-b239e302-6fec-456c-b14a-e56af141016e req-bbf2c75a-7454-466b-b043-f4972e68c894 tempest-TestVolumeSwap-1210043716 None] Terminate volume connection completed successfully. Apr 17 15:26:45 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-b239e302-6fec-456c-b14a-e56af141016e req-bbf2c75a-7454-466b-b043-f4972e68c894 tempest-TestVolumeSwap-1210043716 None] Deleting attachment ed5eb659-cd00-475c-bec2-daa30716b18a. {{(pid=84696) attachment_delete /opt/stack/cinder/cinder/volume/manager.py:5006}} Apr 17 15:26:45 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-b239e302-6fec-456c-b14a-e56af141016e req-bbf2c75a-7454-466b-b043-f4972e68c894 tempest-TestVolumeSwap-1210043716 None] remove_export for volume: 66000c93-eb12-4c23-8a69-71e1aa6b521f. {{(pid=84696) remove_export /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:718}} Apr 17 15:26:45 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-b239e302-6fec-456c-b14a-e56af141016e req-bbf2c75a-7454-466b-b043-f4972e68c894 tempest-TestVolumeSwap-1210043716 None] remove export {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:912}} Apr 17 15:26:45 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-b239e302-6fec-456c-b14a-e56af141016e req-bbf2c75a-7454-466b-b043-f4972e68c894 tempest-TestVolumeSwap-1210043716 None] detach volume:Volume(_name_id=None,admin_metadata=,attach_status='attached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:24:08Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TestVolumeSwap-volume-1757059761',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=66000c93-eb12-4c23-8a69-71e1aa6b521f,launched_at=2023-04-17T15:24:10Z,metadata=,migration_status=None,multiattach=False,previous_status=None,project_id='00b574796d1942938ed2f348803fcf87',provider_auth='CHAP ygxXCJfw kRj6WPpS5zq2r1',provider_geometry=None,provider_id=None,provider_location='172.16.0.220:3260,1 iqn.2010-10.org.openstack:66000c93-eb12-4c23-8a69-71e1aa6b521f 0',replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:24:08Z,service_uuid='e628352e-c4cf-40eb-8389-cce40663026a',shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='detaching',terminated_at=None,updated_at=2023-04-17T15:26:33Z,use_quota=True,user_id='4c2bfcb4119e483ab1a41e3c598ee1b0',volume_attachment=,volume_type=,volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c) from target:iqn.2010-10.org.openstack:66000c93-eb12-4c23-8a69-71e1aa6b521f. {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:913}} Apr 17 15:26:45 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [req-b239e302-6fec-456c-b14a-e56af141016e req-bbf2c75a-7454-466b-b043-f4972e68c894 tempest-TestVolumeSwap-1210043716 None] detach volume v_66000c93-eb12-4c23-8a69-71e1aa6b521f from target iqn.2010-10.org.openstack:66000c93-eb12-4c23-8a69-71e1aa6b521f {{(pid=84696) detach_target_vol /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:606}} Apr 17 15:26:45 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-b239e302-6fec-456c-b14a-e56af141016e req-bbf2c75a-7454-466b-b043-f4972e68c894 tempest-TestVolumeSwap-1210043716 None] Sending pool request DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:66000c93-eb12-4c23-8a69-71e1aa6b521f/luns/v_66000c93-eb12-4c23-8a69-71e1aa6b521f {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:26:45 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-b239e302-6fec-456c-b14a-e56af141016e req-bbf2c75a-7454-466b-b043-f4972e68c894 tempest-TestVolumeSwap-1210043716 None] Sending DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:66000c93-eb12-4c23-8a69-71e1aa6b521f/luns/v_66000c93-eb12-4c23-8a69-71e1aa6b521f {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:26:46 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-b239e302-6fec-456c-b14a-e56af141016e req-bbf2c75a-7454-466b-b043-f4972e68c894 tempest-TestVolumeSwap-1210043716 None] Geting {'code': 204} from DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:66000c93-eb12-4c23-8a69-71e1aa6b521f/luns/v_66000c93-eb12-4c23-8a69-71e1aa6b521f {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:26:46 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-b239e302-6fec-456c-b14a-e56af141016e req-bbf2c75a-7454-466b-b043-f4972e68c894 tempest-TestVolumeSwap-1210043716 None] delete target: iqn.2010-10.org.openstack:66000c93-eb12-4c23-8a69-71e1aa6b521f {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:930}} Apr 17 15:26:46 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [req-b239e302-6fec-456c-b14a-e56af141016e req-bbf2c75a-7454-466b-b043-f4972e68c894 tempest-TestVolumeSwap-1210043716 None] delete target iqn.2010-10.org.openstack:66000c93-eb12-4c23-8a69-71e1aa6b521f {{(pid=84696) delete_target /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:442}} Apr 17 15:26:46 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-b239e302-6fec-456c-b14a-e56af141016e req-bbf2c75a-7454-466b-b043-f4972e68c894 tempest-TestVolumeSwap-1210043716 None] Sending pool request DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:66000c93-eb12-4c23-8a69-71e1aa6b521f {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:26:46 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-b239e302-6fec-456c-b14a-e56af141016e req-bbf2c75a-7454-466b-b043-f4972e68c894 tempest-TestVolumeSwap-1210043716 None] Sending DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:66000c93-eb12-4c23-8a69-71e1aa6b521f {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:26:46 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-b239e302-6fec-456c-b14a-e56af141016e req-bbf2c75a-7454-466b-b043-f4972e68c894 tempest-TestVolumeSwap-1210043716 None] Geting {'code': 204} from DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:66000c93-eb12-4c23-8a69-71e1aa6b521f {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:26:46 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [req-b239e302-6fec-456c-b14a-e56af141016e req-bbf2c75a-7454-466b-b043-f4972e68c894 tempest-TestVolumeSwap-1210043716 None] target iqn.2010-10.org.openstack:66000c93-eb12-4c23-8a69-71e1aa6b521f deleted {{(pid=84696) delete_target /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:447}} Apr 17 15:26:48 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-017c30e5-4553-487c-b935-9661e8bec910 req-840b23b6-a8ce-4397-bbb5-bfba15d04426 tempest-TestVolumeSwap-1210043716 None] create export for volume: 66000c93-eb12-4c23-8a69-71e1aa6b521f. {{(pid=84696) create_export /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:699}} Apr 17 15:26:48 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-017c30e5-4553-487c-b935-9661e8bec910 req-840b23b6-a8ce-4397-bbb5-bfba15d04426 tempest-TestVolumeSwap-1210043716 None] ensure volume 66000c93-eb12-4c23-8a69-71e1aa6b521f assigned to a proper target {{(pid=84696) _ensure_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:866}} Apr 17 15:26:48 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [req-017c30e5-4553-487c-b935-9661e8bec910 req-840b23b6-a8ce-4397-bbb5-bfba15d04426 tempest-TestVolumeSwap-1210043716 None] check if targe iqn.2010-10.org.openstack:66000c93-eb12-4c23-8a69-71e1aa6b521f exists {{(pid=84696) is_target /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:376}} Apr 17 15:26:48 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-017c30e5-4553-487c-b935-9661e8bec910 req-840b23b6-a8ce-4397-bbb5-bfba15d04426 tempest-TestVolumeSwap-1210043716 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:66000c93-eb12-4c23-8a69-71e1aa6b521f {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:26:48 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-017c30e5-4553-487c-b935-9661e8bec910 req-840b23b6-a8ce-4397-bbb5-bfba15d04426 tempest-TestVolumeSwap-1210043716 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:66000c93-eb12-4c23-8a69-71e1aa6b521f {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:26:48 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-017c30e5-4553-487c-b935-9661e8bec910 req-840b23b6-a8ce-4397-bbb5-bfba15d04426 tempest-TestVolumeSwap-1210043716 None] Geting {'code': 404, 'error': {'class': 'opene.exceptions.ItemNotFoundError', 'message': 'Target iqn.2010-10.org.openstack:66000c93-eb12-4c23-8a69-71e1aa6b521f not exists.', 'url': 'http://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:66000c93-eb12-4c23-8a69-71e1aa6b521f'}, 'data': None} from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:66000c93-eb12-4c23-8a69-71e1aa6b521f {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:26:48 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-017c30e5-4553-487c-b935-9661e8bec910 req-840b23b6-a8ce-4397-bbb5-bfba15d04426 tempest-TestVolumeSwap-1210043716 None] create target and attach volume 66000c93-eb12-4c23-8a69-71e1aa6b521f to it {{(pid=84696) _create_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:837}} Apr 17 15:26:48 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [req-017c30e5-4553-487c-b935-9661e8bec910 req-840b23b6-a8ce-4397-bbb5-bfba15d04426 tempest-TestVolumeSwap-1210043716 None] create target iqn.2010-10.org.openstack:66000c93-eb12-4c23-8a69-71e1aa6b521f {{(pid=84696) create_target /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:410}} Apr 17 15:26:48 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-017c30e5-4553-487c-b935-9661e8bec910 req-840b23b6-a8ce-4397-bbb5-bfba15d04426 tempest-TestVolumeSwap-1210043716 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:26:48 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-017c30e5-4553-487c-b935-9661e8bec910 req-840b23b6-a8ce-4397-bbb5-bfba15d04426 tempest-TestVolumeSwap-1210043716 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:26:51 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-7a4406d3-429f-4304-85ec-23b2da5176d0 tempest-BackendsCapabilitiesAdminTestsJSON-1595981469 None] Obtained capabilities list: {'vendor_name': 'Open-E', 'driver_version': '1.0.2', 'storage_protocol': 'iSCSI', 'total_capacity_gb': 119, 'free_capacity_gb': 106, 'reserved_percentage': 0, 'volume_backend_name': 'joviandss-2', 'QoS_support': False, 'location_info': 'JovianISCSIDriver:1:Pool-0', 'multiattach': True, 'properties': {'thin_provisioning': {'title': 'Thin Provisioning', 'description': 'Sets thin provisioning.', 'type': 'boolean'}, 'compression': {'title': 'Compression', 'description': 'Enables compression.', 'type': 'boolean'}, 'qos': {'title': 'QoS', 'description': 'Enables QoS.', 'type': 'boolean'}, 'replication_enabled': {'title': 'Replication', 'description': 'Enables replication.', 'type': 'boolean'}}}. {{(pid=84696) get_capabilities /opt/stack/cinder/cinder/volume/manager.py:4784}} Apr 17 15:26:51 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-a8cc175b-2e7c-49b2-8a1e-8bce1caa60c0 tempest-BackendsCapabilitiesAdminTestsJSON-1595981469 None] Obtained capabilities list: {'vendor_name': 'Open-E', 'driver_version': '1.0.2', 'storage_protocol': 'iSCSI', 'total_capacity_gb': 119, 'free_capacity_gb': 106, 'reserved_percentage': 0, 'volume_backend_name': 'joviandss-2', 'QoS_support': False, 'location_info': 'JovianISCSIDriver:1:Pool-0', 'multiattach': True, 'properties': {'thin_provisioning': {'title': 'Thin Provisioning', 'description': 'Sets thin provisioning.', 'type': 'boolean'}, 'compression': {'title': 'Compression', 'description': 'Enables compression.', 'type': 'boolean'}, 'qos': {'title': 'QoS', 'description': 'Enables QoS.', 'type': 'boolean'}, 'replication_enabled': {'title': 'Replication', 'description': 'Enables replication.', 'type': 'boolean'}}}. {{(pid=84696) get_capabilities /opt/stack/cinder/cinder/volume/manager.py:4784}} Apr 17 15:26:51 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-017c30e5-4553-487c-b935-9661e8bec910 req-840b23b6-a8ce-4397-bbb5-bfba15d04426 tempest-TestVolumeSwap-1210043716 None] Geting {'code': 201, 'error': None, 'data': {'active': True, 'allow_ip': [], 'conflicted': False, 'deny_ip': [], 'incoming_users_active': True, 'name': 'iqn.2010-10.org.openstack:66000c93-eb12-4c23-8a69-71e1aa6b521f', 'outgoing_user': None}} from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:26:51 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [req-017c30e5-4553-487c-b935-9661e8bec910 req-840b23b6-a8ce-4397-bbb5-bfba15d04426 tempest-TestVolumeSwap-1210043716 None] atach volume v_66000c93-eb12-4c23-8a69-71e1aa6b521f to target iqn.2010-10.org.openstack:66000c93-eb12-4c23-8a69-71e1aa6b521f {{(pid=84696) attach_target_vol /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:576}} Apr 17 15:26:51 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-017c30e5-4553-487c-b935-9661e8bec910 req-840b23b6-a8ce-4397-bbb5-bfba15d04426 tempest-TestVolumeSwap-1210043716 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:66000c93-eb12-4c23-8a69-71e1aa6b521f/luns {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:26:51 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-017c30e5-4553-487c-b935-9661e8bec910 req-840b23b6-a8ce-4397-bbb5-bfba15d04426 tempest-TestVolumeSwap-1210043716 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:66000c93-eb12-4c23-8a69-71e1aa6b521f/luns {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:26:53 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-017c30e5-4553-487c-b935-9661e8bec910 req-840b23b6-a8ce-4397-bbb5-bfba15d04426 tempest-TestVolumeSwap-1210043716 None] Geting {'code': 201, 'error': None, 'data': {'block_size': 512, 'device_handler': 'vdisk_blockio', 'eui': '3338626565376231', 'lun': 0, 'mode': 'wt', 'name': 'v_66000c93-eb12-4c23-8a69-71e1aa6b521f', 'prod_id': 'Storage', 'scsi_id': '38bee7b12b313f31'}} from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:66000c93-eb12-4c23-8a69-71e1aa6b521f/luns {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:26:53 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [req-017c30e5-4553-487c-b935-9661e8bec910 req-840b23b6-a8ce-4397-bbb5-bfba15d04426 tempest-TestVolumeSwap-1210043716 None] add credentails to target iqn.2010-10.org.openstack:66000c93-eb12-4c23-8a69-71e1aa6b521f {{(pid=84696) create_target_user /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:474}} Apr 17 15:26:53 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-017c30e5-4553-487c-b935-9661e8bec910 req-840b23b6-a8ce-4397-bbb5-bfba15d04426 tempest-TestVolumeSwap-1210043716 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:66000c93-eb12-4c23-8a69-71e1aa6b521f/incoming-users {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:26:53 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-017c30e5-4553-487c-b935-9661e8bec910 req-840b23b6-a8ce-4397-bbb5-bfba15d04426 tempest-TestVolumeSwap-1210043716 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:66000c93-eb12-4c23-8a69-71e1aa6b521f/incoming-users {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:26:53 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-017c30e5-4553-487c-b935-9661e8bec910 req-840b23b6-a8ce-4397-bbb5-bfba15d04426 tempest-TestVolumeSwap-1210043716 None] Geting {'code': 201, 'error': None, 'data': {'name': 'ygxXCJfw'}} from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:66000c93-eb12-4c23-8a69-71e1aa6b521f/incoming-users {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:26:53 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-017c30e5-4553-487c-b935-9661e8bec910 req-840b23b6-a8ce-4397-bbb5-bfba15d04426 tempest-TestVolumeSwap-1210043716 None] initialize_connection for 66000c93-eb12-4c23-8a69-71e1aa6b521f 10.0.0.210. {{(pid=84696) initialize_connection /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:1018}} Apr 17 15:26:53 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-017c30e5-4553-487c-b935-9661e8bec910 req-840b23b6-a8ce-4397-bbb5-bfba15d04426 tempest-TestVolumeSwap-1210043716 None] Connection info returned from driver {'target_iqn': 'iqn.2010-10.org.openstack:66000c93-eb12-4c23-8a69-71e1aa6b521f', 'target_portal': '172.16.0.220:3260', 'target_discovered': False, 'auth_method': 'CHAP', 'auth_username': 'ygxXCJfw', 'auth_password': '***', 'target_lun': 0, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False, 'driver_volume_type': 'iscsi', 'attachment_id': '69941f8a-0395-4c45-90b6-e2f0055de4d1'} {{(pid=84696) _connection_create /opt/stack/cinder/cinder/volume/manager.py:4875}} Apr 17 15:26:53 user cinder-volume[84696]: INFO cinder.volume.manager [req-017c30e5-4553-487c-b935-9661e8bec910 req-840b23b6-a8ce-4397-bbb5-bfba15d04426 tempest-TestVolumeSwap-1210043716 None] attachment_update completed successfully. Apr 17 15:26:54 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-e77294fa-8c23-4853-ad54-8d48216106ca req-23e9cf08-ae5b-4b94-a3c0-cffea9786996 tempest-TaggedBootDevicesTest_v242-1317742003 None] terminate connection for c74b7878-4936-4015-8b63-142923947310 {{(pid=84696) terminate_connection /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:1032}} Apr 17 15:26:54 user cinder-volume[84696]: INFO cinder.volume.manager [req-e77294fa-8c23-4853-ad54-8d48216106ca req-23e9cf08-ae5b-4b94-a3c0-cffea9786996 tempest-TaggedBootDevicesTest_v242-1317742003 None] Terminate volume connection completed successfully. Apr 17 15:26:54 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-e77294fa-8c23-4853-ad54-8d48216106ca req-23e9cf08-ae5b-4b94-a3c0-cffea9786996 tempest-TaggedBootDevicesTest_v242-1317742003 None] Deleting attachment e51e7201-04a3-43d8-b3a0-c7467cb02c04. {{(pid=84696) attachment_delete /opt/stack/cinder/cinder/volume/manager.py:5006}} Apr 17 15:26:54 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-e77294fa-8c23-4853-ad54-8d48216106ca req-23e9cf08-ae5b-4b94-a3c0-cffea9786996 tempest-TaggedBootDevicesTest_v242-1317742003 None] remove_export for volume: c74b7878-4936-4015-8b63-142923947310. {{(pid=84696) remove_export /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:718}} Apr 17 15:26:54 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-e77294fa-8c23-4853-ad54-8d48216106ca req-23e9cf08-ae5b-4b94-a3c0-cffea9786996 tempest-TaggedBootDevicesTest_v242-1317742003 None] remove export {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:912}} Apr 17 15:26:54 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-e77294fa-8c23-4853-ad54-8d48216106ca req-23e9cf08-ae5b-4b94-a3c0-cffea9786996 tempest-TaggedBootDevicesTest_v242-1317742003 None] detach volume:Volume(_name_id=None,admin_metadata=,attach_status='attached',availability_zone='nova',bootable=True,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:24:01Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TaggedBootDevicesTest_v242-volume-1367581919',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=c74b7878-4936-4015-8b63-142923947310,launched_at=2023-04-17T15:24:32Z,metadata=,migration_status=None,multiattach=False,previous_status=None,project_id='8f0c848b490b47629051e961c682c5a5',provider_auth='CHAP PUvXoCNO 1H8kFoBshXZL64',provider_geometry=None,provider_id=None,provider_location='172.16.0.220:3260,1 iqn.2010-10.org.openstack:c74b7878-4936-4015-8b63-142923947310 0',replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:24:01Z,service_uuid='e628352e-c4cf-40eb-8389-cce40663026a',shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='in-use',terminated_at=None,updated_at=2023-04-17T15:25:03Z,use_quota=True,user_id='3e4b069991954c57993051fbe8665d35',volume_attachment=,volume_type=,volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c) from target:iqn.2010-10.org.openstack:c74b7878-4936-4015-8b63-142923947310. {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:913}} Apr 17 15:26:54 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [req-e77294fa-8c23-4853-ad54-8d48216106ca req-23e9cf08-ae5b-4b94-a3c0-cffea9786996 tempest-TaggedBootDevicesTest_v242-1317742003 None] detach volume v_c74b7878-4936-4015-8b63-142923947310 from target iqn.2010-10.org.openstack:c74b7878-4936-4015-8b63-142923947310 {{(pid=84696) detach_target_vol /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:606}} Apr 17 15:26:54 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-e77294fa-8c23-4853-ad54-8d48216106ca req-23e9cf08-ae5b-4b94-a3c0-cffea9786996 tempest-TaggedBootDevicesTest_v242-1317742003 None] Sending pool request DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:c74b7878-4936-4015-8b63-142923947310/luns/v_c74b7878-4936-4015-8b63-142923947310 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:26:54 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-e77294fa-8c23-4853-ad54-8d48216106ca req-23e9cf08-ae5b-4b94-a3c0-cffea9786996 tempest-TaggedBootDevicesTest_v242-1317742003 None] Sending DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:c74b7878-4936-4015-8b63-142923947310/luns/v_c74b7878-4936-4015-8b63-142923947310 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:26:54 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-e77294fa-8c23-4853-ad54-8d48216106ca req-23e9cf08-ae5b-4b94-a3c0-cffea9786996 tempest-TaggedBootDevicesTest_v242-1317742003 None] Geting {'code': 204} from DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:c74b7878-4936-4015-8b63-142923947310/luns/v_c74b7878-4936-4015-8b63-142923947310 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:26:54 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-e77294fa-8c23-4853-ad54-8d48216106ca req-23e9cf08-ae5b-4b94-a3c0-cffea9786996 tempest-TaggedBootDevicesTest_v242-1317742003 None] delete target: iqn.2010-10.org.openstack:c74b7878-4936-4015-8b63-142923947310 {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:930}} Apr 17 15:26:54 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [req-e77294fa-8c23-4853-ad54-8d48216106ca req-23e9cf08-ae5b-4b94-a3c0-cffea9786996 tempest-TaggedBootDevicesTest_v242-1317742003 None] delete target iqn.2010-10.org.openstack:c74b7878-4936-4015-8b63-142923947310 {{(pid=84696) delete_target /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:442}} Apr 17 15:26:54 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-e77294fa-8c23-4853-ad54-8d48216106ca req-23e9cf08-ae5b-4b94-a3c0-cffea9786996 tempest-TaggedBootDevicesTest_v242-1317742003 None] Sending pool request DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:c74b7878-4936-4015-8b63-142923947310 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:26:54 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-e77294fa-8c23-4853-ad54-8d48216106ca req-23e9cf08-ae5b-4b94-a3c0-cffea9786996 tempest-TaggedBootDevicesTest_v242-1317742003 None] Sending DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:c74b7878-4936-4015-8b63-142923947310 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:26:55 user cinder-volume[84696]: INFO cinder.volume.manager [None req-25550f17-2d1f-4ce2-87bd-fbb40384387d tempest-GroupSnapshotsV319Test-531716155 None] Group tempest-GroupSnapshotsV319Test-Group-1996717408: creating Apr 17 15:26:55 user cinder-volume[84696]: INFO cinder.volume.manager [None req-25550f17-2d1f-4ce2-87bd-fbb40384387d tempest-GroupSnapshotsV319Test-531716155 None] Group tempest-GroupSnapshotsV319Test-Group-1996717408: created successfully Apr 17 15:26:55 user cinder-volume[84696]: INFO cinder.volume.manager [None req-25550f17-2d1f-4ce2-87bd-fbb40384387d tempest-GroupSnapshotsV319Test-531716155 None] Create group completed successfully. Apr 17 15:26:55 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-e77294fa-8c23-4853-ad54-8d48216106ca req-23e9cf08-ae5b-4b94-a3c0-cffea9786996 tempest-TaggedBootDevicesTest_v242-1317742003 None] Geting {'code': 204} from DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:c74b7878-4936-4015-8b63-142923947310 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:26:55 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [req-e77294fa-8c23-4853-ad54-8d48216106ca req-23e9cf08-ae5b-4b94-a3c0-cffea9786996 tempest-TaggedBootDevicesTest_v242-1317742003 None] target iqn.2010-10.org.openstack:c74b7878-4936-4015-8b63-142923947310 deleted {{(pid=84696) delete_target /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:447}} Apr 17 15:26:55 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-e77294fa-8c23-4853-ad54-8d48216106ca req-c4a9c14e-cfe5-424c-8ebc-4b0d01489f35 tempest-TaggedBootDevicesTest_v242-1317742003 None] terminate connection for 18475864-c202-4937-b18f-cc3e16c6d89f {{(pid=84696) terminate_connection /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:1032}} Apr 17 15:26:55 user cinder-volume[84696]: INFO cinder.volume.manager [req-e77294fa-8c23-4853-ad54-8d48216106ca req-c4a9c14e-cfe5-424c-8ebc-4b0d01489f35 tempest-TaggedBootDevicesTest_v242-1317742003 None] Terminate volume connection completed successfully. Apr 17 15:26:55 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-e77294fa-8c23-4853-ad54-8d48216106ca req-c4a9c14e-cfe5-424c-8ebc-4b0d01489f35 tempest-TaggedBootDevicesTest_v242-1317742003 None] Deleting attachment d21f5ce0-237b-4771-ac96-cb17b8159c18. {{(pid=84696) attachment_delete /opt/stack/cinder/cinder/volume/manager.py:5006}} Apr 17 15:26:55 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-e77294fa-8c23-4853-ad54-8d48216106ca req-c4a9c14e-cfe5-424c-8ebc-4b0d01489f35 tempest-TaggedBootDevicesTest_v242-1317742003 None] remove_export for volume: 18475864-c202-4937-b18f-cc3e16c6d89f. {{(pid=84696) remove_export /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:718}} Apr 17 15:26:55 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-e77294fa-8c23-4853-ad54-8d48216106ca req-c4a9c14e-cfe5-424c-8ebc-4b0d01489f35 tempest-TaggedBootDevicesTest_v242-1317742003 None] remove export {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:912}} Apr 17 15:26:55 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-e77294fa-8c23-4853-ad54-8d48216106ca req-c4a9c14e-cfe5-424c-8ebc-4b0d01489f35 tempest-TaggedBootDevicesTest_v242-1317742003 None] detach volume:Volume(_name_id=None,admin_metadata=,attach_status='attached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:24:33Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TaggedBootDevicesTest_v242-volume-1325806681',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=18475864-c202-4937-b18f-cc3e16c6d89f,launched_at=2023-04-17T15:24:35Z,metadata=,migration_status=None,multiattach=False,previous_status=None,project_id='8f0c848b490b47629051e961c682c5a5',provider_auth='CHAP pmlSchIr 83IVml0ZKYD5yo',provider_geometry=None,provider_id=None,provider_location='172.16.0.220:3260,1 iqn.2010-10.org.openstack:18475864-c202-4937-b18f-cc3e16c6d89f 0',replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:24:33Z,service_uuid='e628352e-c4cf-40eb-8389-cce40663026a',shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='in-use',terminated_at=None,updated_at=2023-04-17T15:25:10Z,use_quota=True,user_id='3e4b069991954c57993051fbe8665d35',volume_attachment=,volume_type=,volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c) from target:iqn.2010-10.org.openstack:18475864-c202-4937-b18f-cc3e16c6d89f. {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:913}} Apr 17 15:26:55 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [req-e77294fa-8c23-4853-ad54-8d48216106ca req-c4a9c14e-cfe5-424c-8ebc-4b0d01489f35 tempest-TaggedBootDevicesTest_v242-1317742003 None] detach volume v_18475864-c202-4937-b18f-cc3e16c6d89f from target iqn.2010-10.org.openstack:18475864-c202-4937-b18f-cc3e16c6d89f {{(pid=84696) detach_target_vol /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:606}} Apr 17 15:26:55 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-e77294fa-8c23-4853-ad54-8d48216106ca req-c4a9c14e-cfe5-424c-8ebc-4b0d01489f35 tempest-TaggedBootDevicesTest_v242-1317742003 None] Sending pool request DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:18475864-c202-4937-b18f-cc3e16c6d89f/luns/v_18475864-c202-4937-b18f-cc3e16c6d89f {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:26:55 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-e77294fa-8c23-4853-ad54-8d48216106ca req-c4a9c14e-cfe5-424c-8ebc-4b0d01489f35 tempest-TaggedBootDevicesTest_v242-1317742003 None] Sending DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:18475864-c202-4937-b18f-cc3e16c6d89f/luns/v_18475864-c202-4937-b18f-cc3e16c6d89f {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:26:56 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-e77294fa-8c23-4853-ad54-8d48216106ca req-c4a9c14e-cfe5-424c-8ebc-4b0d01489f35 tempest-TaggedBootDevicesTest_v242-1317742003 None] Geting {'code': 204} from DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:18475864-c202-4937-b18f-cc3e16c6d89f/luns/v_18475864-c202-4937-b18f-cc3e16c6d89f {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:26:56 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-e77294fa-8c23-4853-ad54-8d48216106ca req-c4a9c14e-cfe5-424c-8ebc-4b0d01489f35 tempest-TaggedBootDevicesTest_v242-1317742003 None] delete target: iqn.2010-10.org.openstack:18475864-c202-4937-b18f-cc3e16c6d89f {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:930}} Apr 17 15:26:56 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [req-e77294fa-8c23-4853-ad54-8d48216106ca req-c4a9c14e-cfe5-424c-8ebc-4b0d01489f35 tempest-TaggedBootDevicesTest_v242-1317742003 None] delete target iqn.2010-10.org.openstack:18475864-c202-4937-b18f-cc3e16c6d89f {{(pid=84696) delete_target /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:442}} Apr 17 15:26:56 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-e77294fa-8c23-4853-ad54-8d48216106ca req-c4a9c14e-cfe5-424c-8ebc-4b0d01489f35 tempest-TaggedBootDevicesTest_v242-1317742003 None] Sending pool request DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:18475864-c202-4937-b18f-cc3e16c6d89f {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:26:56 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-e77294fa-8c23-4853-ad54-8d48216106ca req-c4a9c14e-cfe5-424c-8ebc-4b0d01489f35 tempest-TaggedBootDevicesTest_v242-1317742003 None] Sending DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:18475864-c202-4937-b18f-cc3e16c6d89f {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:26:56 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-e77294fa-8c23-4853-ad54-8d48216106ca req-c4a9c14e-cfe5-424c-8ebc-4b0d01489f35 tempest-TaggedBootDevicesTest_v242-1317742003 None] Geting {'code': 204} from DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:18475864-c202-4937-b18f-cc3e16c6d89f {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:26:56 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [req-e77294fa-8c23-4853-ad54-8d48216106ca req-c4a9c14e-cfe5-424c-8ebc-4b0d01489f35 tempest-TaggedBootDevicesTest_v242-1317742003 None] target iqn.2010-10.org.openstack:18475864-c202-4937-b18f-cc3e16c6d89f deleted {{(pid=84696) delete_target /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:447}} Apr 17 15:26:56 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-e77294fa-8c23-4853-ad54-8d48216106ca req-a2f2e835-96d0-4773-95ee-0899b21b0f11 tempest-TaggedBootDevicesTest_v242-1317742003 None] terminate connection for 65a7c5eb-3af6-4afd-918f-ccb69a68641c {{(pid=84696) terminate_connection /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:1032}} Apr 17 15:26:56 user cinder-volume[84696]: INFO cinder.volume.manager [req-e77294fa-8c23-4853-ad54-8d48216106ca req-a2f2e835-96d0-4773-95ee-0899b21b0f11 tempest-TaggedBootDevicesTest_v242-1317742003 None] Terminate volume connection completed successfully. Apr 17 15:26:56 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-e77294fa-8c23-4853-ad54-8d48216106ca req-a2f2e835-96d0-4773-95ee-0899b21b0f11 tempest-TaggedBootDevicesTest_v242-1317742003 None] Deleting attachment c57372fa-5821-4a41-9048-fb252947e1d0. {{(pid=84696) attachment_delete /opt/stack/cinder/cinder/volume/manager.py:5006}} Apr 17 15:26:56 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-e77294fa-8c23-4853-ad54-8d48216106ca req-a2f2e835-96d0-4773-95ee-0899b21b0f11 tempest-TaggedBootDevicesTest_v242-1317742003 None] remove_export for volume: 65a7c5eb-3af6-4afd-918f-ccb69a68641c. {{(pid=84696) remove_export /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:718}} Apr 17 15:26:56 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-e77294fa-8c23-4853-ad54-8d48216106ca req-a2f2e835-96d0-4773-95ee-0899b21b0f11 tempest-TaggedBootDevicesTest_v242-1317742003 None] remove export {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:912}} Apr 17 15:26:56 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-e77294fa-8c23-4853-ad54-8d48216106ca req-a2f2e835-96d0-4773-95ee-0899b21b0f11 tempest-TaggedBootDevicesTest_v242-1317742003 None] detach volume:Volume(_name_id=None,admin_metadata=,attach_status='attached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:24:36Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TaggedBootDevicesTest_v242-volume-1252584131',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=65a7c5eb-3af6-4afd-918f-ccb69a68641c,launched_at=2023-04-17T15:24:38Z,metadata=,migration_status=None,multiattach=False,previous_status=None,project_id='8f0c848b490b47629051e961c682c5a5',provider_auth='CHAP wouATrvO yqXPcs1aB98YHb',provider_geometry=None,provider_id=None,provider_location='172.16.0.220:3260,1 iqn.2010-10.org.openstack:65a7c5eb-3af6-4afd-918f-ccb69a68641c 0',replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:24:36Z,service_uuid='e628352e-c4cf-40eb-8389-cce40663026a',shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='in-use',terminated_at=None,updated_at=2023-04-17T15:25:15Z,use_quota=True,user_id='3e4b069991954c57993051fbe8665d35',volume_attachment=,volume_type=,volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c) from target:iqn.2010-10.org.openstack:65a7c5eb-3af6-4afd-918f-ccb69a68641c. {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:913}} Apr 17 15:26:56 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [req-e77294fa-8c23-4853-ad54-8d48216106ca req-a2f2e835-96d0-4773-95ee-0899b21b0f11 tempest-TaggedBootDevicesTest_v242-1317742003 None] detach volume v_65a7c5eb-3af6-4afd-918f-ccb69a68641c from target iqn.2010-10.org.openstack:65a7c5eb-3af6-4afd-918f-ccb69a68641c {{(pid=84696) detach_target_vol /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:606}} Apr 17 15:26:56 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-e77294fa-8c23-4853-ad54-8d48216106ca req-a2f2e835-96d0-4773-95ee-0899b21b0f11 tempest-TaggedBootDevicesTest_v242-1317742003 None] Sending pool request DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:65a7c5eb-3af6-4afd-918f-ccb69a68641c/luns/v_65a7c5eb-3af6-4afd-918f-ccb69a68641c {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:26:56 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-e77294fa-8c23-4853-ad54-8d48216106ca req-a2f2e835-96d0-4773-95ee-0899b21b0f11 tempest-TaggedBootDevicesTest_v242-1317742003 None] Sending DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:65a7c5eb-3af6-4afd-918f-ccb69a68641c/luns/v_65a7c5eb-3af6-4afd-918f-ccb69a68641c {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:26:56 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-edcc7e4e-9e9b-4c4f-80de-91c579e36770 tempest-GroupSnapshotsV319Test-531716155 None] Volume reschedule parameters: True retry: {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 15:26:56 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-edcc7e4e-9e9b-4c4f-80de-91c579e36770 tempest-GroupSnapshotsV319Test-531716155 None] Flow 'volume_create_manager' (591ca05c-fcd3-4eec-9d8e-549934b55f67) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:26:56 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-edcc7e4e-9e9b-4c4f-80de-91c579e36770 tempest-GroupSnapshotsV319Test-531716155 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (891e6ccc-bd98-4422-aade-6b60bdb3e05b) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:26:56 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-edcc7e4e-9e9b-4c4f-80de-91c579e36770 tempest-GroupSnapshotsV319Test-531716155 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (891e6ccc-bd98-4422-aade-6b60bdb3e05b) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:26:57Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-volume-609793849',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=7cfa2150-9d68-4591-b1b3-bd3a13002e9e,host='user@joviandss-2#joviandss-2',id=d5a81ad1-94ca-4688-9d1d-2d953807886e,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='dd1792b234e44b5faf547b359e08667b',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:26:57Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:26:57Z,use_quota=True,user_id='bb9c1925c06c40c481c028493885ea57',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(3710bfe6-8491-42c0-8fc3-c1f34b0279ed),volume_type_id=3710bfe6-8491-42c0-8fc3-c1f34b0279ed)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:26:56 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-edcc7e4e-9e9b-4c4f-80de-91c579e36770 tempest-GroupSnapshotsV319Test-531716155 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (9a0a7d14-bc3d-49c6-a5b7-5089b38ae6ba) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:26:56 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-edcc7e4e-9e9b-4c4f-80de-91c579e36770 tempest-GroupSnapshotsV319Test-531716155 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (9a0a7d14-bc3d-49c6-a5b7-5089b38ae6ba) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:26:56 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-edcc7e4e-9e9b-4c4f-80de-91c579e36770 tempest-GroupSnapshotsV319Test-531716155 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (368e9ec0-39d2-4259-9283-5f9a315eb990) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:26:56 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-edcc7e4e-9e9b-4c4f-80de-91c579e36770 tempest-GroupSnapshotsV319Test-531716155 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (368e9ec0-39d2-4259-9283-5f9a315eb990) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': 'd5a81ad1-94ca-4688-9d1d-2d953807886e', 'volume_name': 'volume-d5a81ad1-94ca-4688-9d1d-2d953807886e', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:26:56 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-edcc7e4e-9e9b-4c4f-80de-91c579e36770 tempest-GroupSnapshotsV319Test-531716155 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (91de7a32-8bc3-429a-92b9-c6fc1dce8117) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:26:56 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-edcc7e4e-9e9b-4c4f-80de-91c579e36770 tempest-GroupSnapshotsV319Test-531716155 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (91de7a32-8bc3-429a-92b9-c6fc1dce8117) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:26:56 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-edcc7e4e-9e9b-4c4f-80de-91c579e36770 tempest-GroupSnapshotsV319Test-531716155 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (2a2e03a5-d5e1-4765-bc05-ddbd209ad587) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:26:56 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-edcc7e4e-9e9b-4c4f-80de-91c579e36770 tempest-GroupSnapshotsV319Test-531716155 None] Volume d5a81ad1-94ca-4688-9d1d-2d953807886e: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-d5a81ad1-94ca-4688-9d1d-2d953807886e', 'volume_size': 1} Apr 17 15:26:56 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-edcc7e4e-9e9b-4c4f-80de-91c579e36770 tempest-GroupSnapshotsV319Test-531716155 None] creating volume v_d5a81ad1-94ca-4688-9d1d-2d953807886e. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 15:26:56 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-edcc7e4e-9e9b-4c4f-80de-91c579e36770 tempest-GroupSnapshotsV319Test-531716155 None] create volume {'name': 'v_d5a81ad1-94ca-4688-9d1d-2d953807886e', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 15:26:56 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-edcc7e4e-9e9b-4c4f-80de-91c579e36770 tempest-GroupSnapshotsV319Test-531716155 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:26:56 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-edcc7e4e-9e9b-4c4f-80de-91c579e36770 tempest-GroupSnapshotsV319Test-531716155 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:26:57 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-e77294fa-8c23-4853-ad54-8d48216106ca req-a2f2e835-96d0-4773-95ee-0899b21b0f11 tempest-TaggedBootDevicesTest_v242-1317742003 None] Geting {'code': 204} from DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:65a7c5eb-3af6-4afd-918f-ccb69a68641c/luns/v_65a7c5eb-3af6-4afd-918f-ccb69a68641c {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:26:57 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-e77294fa-8c23-4853-ad54-8d48216106ca req-a2f2e835-96d0-4773-95ee-0899b21b0f11 tempest-TaggedBootDevicesTest_v242-1317742003 None] delete target: iqn.2010-10.org.openstack:65a7c5eb-3af6-4afd-918f-ccb69a68641c {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:930}} Apr 17 15:26:57 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [req-e77294fa-8c23-4853-ad54-8d48216106ca req-a2f2e835-96d0-4773-95ee-0899b21b0f11 tempest-TaggedBootDevicesTest_v242-1317742003 None] delete target iqn.2010-10.org.openstack:65a7c5eb-3af6-4afd-918f-ccb69a68641c {{(pid=84696) delete_target /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:442}} Apr 17 15:26:57 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-e77294fa-8c23-4853-ad54-8d48216106ca req-a2f2e835-96d0-4773-95ee-0899b21b0f11 tempest-TaggedBootDevicesTest_v242-1317742003 None] Sending pool request DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:65a7c5eb-3af6-4afd-918f-ccb69a68641c {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:26:57 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-e77294fa-8c23-4853-ad54-8d48216106ca req-a2f2e835-96d0-4773-95ee-0899b21b0f11 tempest-TaggedBootDevicesTest_v242-1317742003 None] Sending DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:65a7c5eb-3af6-4afd-918f-ccb69a68641c {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:26:58 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-edcc7e4e-9e9b-4c4f-80de-91c579e36770 tempest-GroupSnapshotsV319Test-531716155 None] Geting {'code': 201, 'error': None, 'data': {'full_name': 'Pool-0/v_d5a81ad1-94ca-4688-9d1d-2d953807886e', 'is_clone': False, 'name': 'v_d5a81ad1-94ca-4688-9d1d-2d953807886e'}} from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:26:58 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-edcc7e4e-9e9b-4c4f-80de-91c579e36770 tempest-GroupSnapshotsV319Test-531716155 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (2a2e03a5-d5e1-4765-bc05-ddbd209ad587) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'volume_name': 'volume-d5a81ad1-94ca-4688-9d1d-2d953807886e', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:26:58 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-edcc7e4e-9e9b-4c4f-80de-91c579e36770 tempest-GroupSnapshotsV319Test-531716155 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeOnFinishTask;volume:create, create.end' (c9c17ce5-e802-4870-a31d-88c471f1af96) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:26:59 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-edcc7e4e-9e9b-4c4f-80de-91c579e36770 tempest-GroupSnapshotsV319Test-531716155 None] Volume volume-d5a81ad1-94ca-4688-9d1d-2d953807886e (d5a81ad1-94ca-4688-9d1d-2d953807886e): created successfully Apr 17 15:26:59 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-edcc7e4e-9e9b-4c4f-80de-91c579e36770 tempest-GroupSnapshotsV319Test-531716155 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeOnFinishTask;volume:create, create.end' (c9c17ce5-e802-4870-a31d-88c471f1af96) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:26:59 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-edcc7e4e-9e9b-4c4f-80de-91c579e36770 tempest-GroupSnapshotsV319Test-531716155 None] Flow 'volume_create_manager' (591ca05c-fcd3-4eec-9d8e-549934b55f67) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:26:59 user cinder-volume[84696]: INFO cinder.volume.manager [None req-edcc7e4e-9e9b-4c4f-80de-91c579e36770 tempest-GroupSnapshotsV319Test-531716155 None] Created volume successfully. Apr 17 15:26:59 user cinder-volume[84696]: INFO cinder.volume.manager [None req-eae26192-6f8e-4d5e-9435-887ba02602dd tempest-GroupSnapshotsV319Test-531716155 None] GroupSnapshot f4dcb486-d239-4af8-9532-6ea5d66ec840: creating. Apr 17 15:26:59 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-eae26192-6f8e-4d5e-9435-887ba02602dd tempest-GroupSnapshotsV319Test-531716155 None] Group snapshot f4dcb486-d239-4af8-9532-6ea5d66ec840: creating. {{(pid=84696) create_group_snapshot /opt/stack/cinder/cinder/volume/manager.py:4129}} Apr 17 15:26:59 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-eae26192-6f8e-4d5e-9435-887ba02602dd tempest-GroupSnapshotsV319Test-531716155 None] create snapshot 1761e65f-0800-4c2d-85cb-ce19f5f68b7a for volume d5a81ad1-94ca-4688-9d1d-2d953807886e {{(pid=84696) create_snapshot /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:625}} Apr 17 15:26:59 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-eae26192-6f8e-4d5e-9435-887ba02602dd tempest-GroupSnapshotsV319Test-531716155 None] cloning v_d5a81ad1-94ca-4688-9d1d-2d953807886e to s_1761e65f-0800-4c2d-85cb-ce19f5f68b7a {{(pid=84696) _clone_object /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:456}} Apr 17 15:26:59 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-eae26192-6f8e-4d5e-9435-887ba02602dd tempest-GroupSnapshotsV319Test-531716155 None] create snapshot s_1761e65f-0800-4c2d-85cb-ce19f5f68b7a {{(pid=84696) create_snapshot /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:635}} Apr 17 15:26:59 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-eae26192-6f8e-4d5e-9435-887ba02602dd tempest-GroupSnapshotsV319Test-531716155 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_d5a81ad1-94ca-4688-9d1d-2d953807886e/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:26:59 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-eae26192-6f8e-4d5e-9435-887ba02602dd tempest-GroupSnapshotsV319Test-531716155 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_d5a81ad1-94ca-4688-9d1d-2d953807886e/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:26:59 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-e77294fa-8c23-4853-ad54-8d48216106ca req-a2f2e835-96d0-4773-95ee-0899b21b0f11 tempest-TaggedBootDevicesTest_v242-1317742003 None] Geting {'code': 204} from DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:65a7c5eb-3af6-4afd-918f-ccb69a68641c {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:26:59 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [req-e77294fa-8c23-4853-ad54-8d48216106ca req-a2f2e835-96d0-4773-95ee-0899b21b0f11 tempest-TaggedBootDevicesTest_v242-1317742003 None] target iqn.2010-10.org.openstack:65a7c5eb-3af6-4afd-918f-ccb69a68641c deleted {{(pid=84696) delete_target /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:447}} Apr 17 15:27:00 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-017c30e5-4553-487c-b935-9661e8bec910 req-3b7b90ca-cd4b-49c9-8f82-8a6198aeba83 tempest-TestVolumeSwap-1210043716 None] terminate connection for 485ea70c-618d-4373-a78f-a460914fc095 {{(pid=84696) terminate_connection /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:1032}} Apr 17 15:27:00 user cinder-volume[84696]: INFO cinder.volume.manager [req-017c30e5-4553-487c-b935-9661e8bec910 req-3b7b90ca-cd4b-49c9-8f82-8a6198aeba83 tempest-TestVolumeSwap-1210043716 None] Terminate volume connection completed successfully. Apr 17 15:27:00 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-017c30e5-4553-487c-b935-9661e8bec910 req-3b7b90ca-cd4b-49c9-8f82-8a6198aeba83 tempest-TestVolumeSwap-1210043716 None] Deleting attachment 9a8c6327-030a-41e4-98c6-a23b982c5ee9. {{(pid=84696) attachment_delete /opt/stack/cinder/cinder/volume/manager.py:5006}} Apr 17 15:27:00 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-017c30e5-4553-487c-b935-9661e8bec910 req-3b7b90ca-cd4b-49c9-8f82-8a6198aeba83 tempest-TestVolumeSwap-1210043716 None] remove_export for volume: 485ea70c-618d-4373-a78f-a460914fc095. {{(pid=84696) remove_export /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:718}} Apr 17 15:27:00 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-017c30e5-4553-487c-b935-9661e8bec910 req-3b7b90ca-cd4b-49c9-8f82-8a6198aeba83 tempest-TestVolumeSwap-1210043716 None] remove export {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:912}} Apr 17 15:27:00 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-017c30e5-4553-487c-b935-9661e8bec910 req-3b7b90ca-cd4b-49c9-8f82-8a6198aeba83 tempest-TestVolumeSwap-1210043716 None] detach volume:Volume(_name_id=None,admin_metadata=,attach_status='attached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:24:11Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TestVolumeSwap-volume-2088850205',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=485ea70c-618d-4373-a78f-a460914fc095,launched_at=2023-04-17T15:24:14Z,metadata=,migration_status=None,multiattach=False,previous_status=None,project_id='00b574796d1942938ed2f348803fcf87',provider_auth='CHAP WBubQvJS XBNtnWHrPfMu3U',provider_geometry=None,provider_id=None,provider_location='172.16.0.220:3260,1 iqn.2010-10.org.openstack:485ea70c-618d-4373-a78f-a460914fc095 0',replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:24:12Z,service_uuid='e628352e-c4cf-40eb-8389-cce40663026a',shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='detaching',terminated_at=None,updated_at=2023-04-17T15:26:47Z,use_quota=True,user_id='4c2bfcb4119e483ab1a41e3c598ee1b0',volume_attachment=,volume_type=,volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c) from target:iqn.2010-10.org.openstack:485ea70c-618d-4373-a78f-a460914fc095. {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:913}} Apr 17 15:27:00 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [req-017c30e5-4553-487c-b935-9661e8bec910 req-3b7b90ca-cd4b-49c9-8f82-8a6198aeba83 tempest-TestVolumeSwap-1210043716 None] detach volume v_485ea70c-618d-4373-a78f-a460914fc095 from target iqn.2010-10.org.openstack:485ea70c-618d-4373-a78f-a460914fc095 {{(pid=84696) detach_target_vol /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:606}} Apr 17 15:27:00 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-017c30e5-4553-487c-b935-9661e8bec910 req-3b7b90ca-cd4b-49c9-8f82-8a6198aeba83 tempest-TestVolumeSwap-1210043716 None] Sending pool request DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:485ea70c-618d-4373-a78f-a460914fc095/luns/v_485ea70c-618d-4373-a78f-a460914fc095 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:27:00 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-017c30e5-4553-487c-b935-9661e8bec910 req-3b7b90ca-cd4b-49c9-8f82-8a6198aeba83 tempest-TestVolumeSwap-1210043716 None] Sending DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:485ea70c-618d-4373-a78f-a460914fc095/luns/v_485ea70c-618d-4373-a78f-a460914fc095 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:27:00 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-eae26192-6f8e-4d5e-9435-887ba02602dd tempest-GroupSnapshotsV319Test-531716155 None] Geting {'code': 200, 'error': None, 'data': {'name': 's_1761e65f-0800-4c2d-85cb-ce19f5f68b7a'}} from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_d5a81ad1-94ca-4688-9d1d-2d953807886e/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:27:00 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-eae26192-6f8e-4d5e-9435-887ba02602dd tempest-GroupSnapshotsV319Test-531716155 None] create volume s_1761e65f-0800-4c2d-85cb-ce19f5f68b7a from snapshot s_1761e65f-0800-4c2d-85cb-ce19f5f68b7a {{(pid=84696) create_volume_from_snapshot /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:674}} Apr 17 15:27:00 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-eae26192-6f8e-4d5e-9435-887ba02602dd tempest-GroupSnapshotsV319Test-531716155 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_d5a81ad1-94ca-4688-9d1d-2d953807886e/clone {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:27:00 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-eae26192-6f8e-4d5e-9435-887ba02602dd tempest-GroupSnapshotsV319Test-531716155 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_d5a81ad1-94ca-4688-9d1d-2d953807886e/clone {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:27:01 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-017c30e5-4553-487c-b935-9661e8bec910 req-3b7b90ca-cd4b-49c9-8f82-8a6198aeba83 tempest-TestVolumeSwap-1210043716 None] Geting {'code': 204} from DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:485ea70c-618d-4373-a78f-a460914fc095/luns/v_485ea70c-618d-4373-a78f-a460914fc095 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:27:01 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-017c30e5-4553-487c-b935-9661e8bec910 req-3b7b90ca-cd4b-49c9-8f82-8a6198aeba83 tempest-TestVolumeSwap-1210043716 None] delete target: iqn.2010-10.org.openstack:485ea70c-618d-4373-a78f-a460914fc095 {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:930}} Apr 17 15:27:01 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [req-017c30e5-4553-487c-b935-9661e8bec910 req-3b7b90ca-cd4b-49c9-8f82-8a6198aeba83 tempest-TestVolumeSwap-1210043716 None] delete target iqn.2010-10.org.openstack:485ea70c-618d-4373-a78f-a460914fc095 {{(pid=84696) delete_target /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:442}} Apr 17 15:27:01 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-017c30e5-4553-487c-b935-9661e8bec910 req-3b7b90ca-cd4b-49c9-8f82-8a6198aeba83 tempest-TestVolumeSwap-1210043716 None] Sending pool request DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:485ea70c-618d-4373-a78f-a460914fc095 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:27:01 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-017c30e5-4553-487c-b935-9661e8bec910 req-3b7b90ca-cd4b-49c9-8f82-8a6198aeba83 tempest-TestVolumeSwap-1210043716 None] Sending DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:485ea70c-618d-4373-a78f-a460914fc095 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:27:02 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-eae26192-6f8e-4d5e-9435-887ba02602dd tempest-GroupSnapshotsV319Test-531716155 None] Geting {'code': 200, 'error': None, 'data': {'origin': 'Pool-0/v_d5a81ad1-94ca-4688-9d1d-2d953807886e@s_1761e65f-0800-4c2d-85cb-ce19f5f68b7a', 'is_clone': True, 'full_name': 'Pool-0/s_1761e65f-0800-4c2d-85cb-ce19f5f68b7a', 'name': 's_1761e65f-0800-4c2d-85cb-ce19f5f68b7a'}} from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_d5a81ad1-94ca-4688-9d1d-2d953807886e/clone {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:27:02 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-eae26192-6f8e-4d5e-9435-887ba02602dd tempest-GroupSnapshotsV319Test-531716155 None] Sending pool request PUT to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/s_1761e65f-0800-4c2d-85cb-ce19f5f68b7a/properties {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:27:02 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-eae26192-6f8e-4d5e-9435-887ba02602dd tempest-GroupSnapshotsV319Test-531716155 None] Sending PUT to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/s_1761e65f-0800-4c2d-85cb-ce19f5f68b7a/properties {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:27:02 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-017c30e5-4553-487c-b935-9661e8bec910 req-3b7b90ca-cd4b-49c9-8f82-8a6198aeba83 tempest-TestVolumeSwap-1210043716 None] Geting {'code': 204} from DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:485ea70c-618d-4373-a78f-a460914fc095 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:27:02 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [req-017c30e5-4553-487c-b935-9661e8bec910 req-3b7b90ca-cd4b-49c9-8f82-8a6198aeba83 tempest-TestVolumeSwap-1210043716 None] target iqn.2010-10.org.openstack:485ea70c-618d-4373-a78f-a460914fc095 deleted {{(pid=84696) delete_target /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:447}} Apr 17 15:27:03 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-eae26192-6f8e-4d5e-9435-887ba02602dd tempest-GroupSnapshotsV319Test-531716155 None] Geting {'code': 201, 'error': None, 'data': None} from PUT to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/s_1761e65f-0800-4c2d-85cb-ce19f5f68b7a/properties {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:27:03 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-eae26192-6f8e-4d5e-9435-887ba02602dd tempest-GroupSnapshotsV319Test-531716155 None] volume s_1761e65f-0800-4c2d-85cb-ce19f5f68b7a properties updated {{(pid=84696) modify_property_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:301}} Apr 17 15:27:03 user cinder-volume[84696]: INFO cinder.volume.manager [None req-eae26192-6f8e-4d5e-9435-887ba02602dd tempest-GroupSnapshotsV319Test-531716155 None] group_snapshot f4dcb486-d239-4af8-9532-6ea5d66ec840: created successfully Apr 17 15:27:03 user cinder-volume[84696]: INFO cinder.volume.manager [None req-dc265cc2-2c37-4e0d-a41d-71d4ac546ea7 tempest-GroupSnapshotsV319Test-531716155 None] group_snapshot f4dcb486-d239-4af8-9532-6ea5d66ec840: deleting Apr 17 15:27:03 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=77,cinder:UPDATE=19,cinder:INSERT=2,cinder:DELETE=1 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:27:03 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-dc265cc2-2c37-4e0d-a41d-71d4ac546ea7 tempest-GroupSnapshotsV319Test-531716155 None] group_snapshot f4dcb486-d239-4af8-9532-6ea5d66ec840: deleting {{(pid=84696) delete_group_snapshot /opt/stack/cinder/cinder/volume/manager.py:4304}} Apr 17 15:27:03 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-dc265cc2-2c37-4e0d-a41d-71d4ac546ea7 tempest-GroupSnapshotsV319Test-531716155 None] deleating snapshot s_1761e65f-0800-4c2d-85cb-ce19f5f68b7a. {{(pid=84696) delete_snapshot /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:649}} Apr 17 15:27:03 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-dc265cc2-2c37-4e0d-a41d-71d4ac546ea7 tempest-GroupSnapshotsV319Test-531716155 None] get snapshots for volume s_1761e65f-0800-4c2d-85cb-ce19f5f68b7a {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 15:27:03 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-dc265cc2-2c37-4e0d-a41d-71d4ac546ea7 tempest-GroupSnapshotsV319Test-531716155 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/s_1761e65f-0800-4c2d-85cb-ce19f5f68b7a/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:27:03 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-dc265cc2-2c37-4e0d-a41d-71d4ac546ea7 tempest-GroupSnapshotsV319Test-531716155 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/s_1761e65f-0800-4c2d-85cb-ce19f5f68b7a/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:27:04 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-6dca4c82-5919-4549-9ef9-2dd9c5ab78c8 req-4e9cfff8-5b13-4801-9a21-6a871ccbf656 tempest-TestVolumeSwap-1714109745 None] terminate connection for 66000c93-eb12-4c23-8a69-71e1aa6b521f {{(pid=84696) terminate_connection /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:1032}} Apr 17 15:27:04 user cinder-volume[84696]: INFO cinder.volume.manager [req-6dca4c82-5919-4549-9ef9-2dd9c5ab78c8 req-4e9cfff8-5b13-4801-9a21-6a871ccbf656 tempest-TestVolumeSwap-1714109745 None] Terminate volume connection completed successfully. Apr 17 15:27:04 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-6dca4c82-5919-4549-9ef9-2dd9c5ab78c8 req-4e9cfff8-5b13-4801-9a21-6a871ccbf656 tempest-TestVolumeSwap-1714109745 None] Deleting attachment 69941f8a-0395-4c45-90b6-e2f0055de4d1. {{(pid=84696) attachment_delete /opt/stack/cinder/cinder/volume/manager.py:5006}} Apr 17 15:27:04 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-6dca4c82-5919-4549-9ef9-2dd9c5ab78c8 req-4e9cfff8-5b13-4801-9a21-6a871ccbf656 tempest-TestVolumeSwap-1714109745 None] remove_export for volume: 66000c93-eb12-4c23-8a69-71e1aa6b521f. {{(pid=84696) remove_export /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:718}} Apr 17 15:27:04 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-6dca4c82-5919-4549-9ef9-2dd9c5ab78c8 req-4e9cfff8-5b13-4801-9a21-6a871ccbf656 tempest-TestVolumeSwap-1714109745 None] remove export {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:912}} Apr 17 15:27:04 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-6dca4c82-5919-4549-9ef9-2dd9c5ab78c8 req-4e9cfff8-5b13-4801-9a21-6a871ccbf656 tempest-TestVolumeSwap-1714109745 None] detach volume:Volume(_name_id=None,admin_metadata=,attach_status='attached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:24:08Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TestVolumeSwap-volume-1757059761',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=66000c93-eb12-4c23-8a69-71e1aa6b521f,launched_at=2023-04-17T15:24:10Z,metadata=,migration_status=None,multiattach=False,previous_status=None,project_id='00b574796d1942938ed2f348803fcf87',provider_auth='CHAP ygxXCJfw kRj6WPpS5zq2r1',provider_geometry=None,provider_id=None,provider_location='172.16.0.220:3260,1 iqn.2010-10.org.openstack:66000c93-eb12-4c23-8a69-71e1aa6b521f 0',replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:24:08Z,service_uuid='e628352e-c4cf-40eb-8389-cce40663026a',shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='detaching',terminated_at=None,updated_at=2023-04-17T15:27:04Z,use_quota=True,user_id='4c2bfcb4119e483ab1a41e3c598ee1b0',volume_attachment=,volume_type=,volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c) from target:iqn.2010-10.org.openstack:66000c93-eb12-4c23-8a69-71e1aa6b521f. {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:913}} Apr 17 15:27:04 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [req-6dca4c82-5919-4549-9ef9-2dd9c5ab78c8 req-4e9cfff8-5b13-4801-9a21-6a871ccbf656 tempest-TestVolumeSwap-1714109745 None] detach volume v_66000c93-eb12-4c23-8a69-71e1aa6b521f from target iqn.2010-10.org.openstack:66000c93-eb12-4c23-8a69-71e1aa6b521f {{(pid=84696) detach_target_vol /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:606}} Apr 17 15:27:04 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-6dca4c82-5919-4549-9ef9-2dd9c5ab78c8 req-4e9cfff8-5b13-4801-9a21-6a871ccbf656 tempest-TestVolumeSwap-1714109745 None] Sending pool request DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:66000c93-eb12-4c23-8a69-71e1aa6b521f/luns/v_66000c93-eb12-4c23-8a69-71e1aa6b521f {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:27:04 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-6dca4c82-5919-4549-9ef9-2dd9c5ab78c8 req-4e9cfff8-5b13-4801-9a21-6a871ccbf656 tempest-TestVolumeSwap-1714109745 None] Sending DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:66000c93-eb12-4c23-8a69-71e1aa6b521f/luns/v_66000c93-eb12-4c23-8a69-71e1aa6b521f {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:27:04 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-dc265cc2-2c37-4e0d-a41d-71d4ac546ea7 tempest-GroupSnapshotsV319Test-531716155 None] Geting {'code': 200, 'error': None, 'data': {'results': 0, 'entries': []}} from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/s_1761e65f-0800-4c2d-85cb-ce19f5f68b7a/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:27:04 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-dc265cc2-2c37-4e0d-a41d-71d4ac546ea7 tempest-GroupSnapshotsV319Test-531716155 None] get volume s_1761e65f-0800-4c2d-85cb-ce19f5f68b7a info {{(pid=84696) get_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:239}} Apr 17 15:27:04 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-dc265cc2-2c37-4e0d-a41d-71d4ac546ea7 tempest-GroupSnapshotsV319Test-531716155 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/s_1761e65f-0800-4c2d-85cb-ce19f5f68b7a {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:27:04 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-dc265cc2-2c37-4e0d-a41d-71d4ac546ea7 tempest-GroupSnapshotsV319Test-531716155 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/s_1761e65f-0800-4c2d-85cb-ce19f5f68b7a {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:27:05 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-6dca4c82-5919-4549-9ef9-2dd9c5ab78c8 req-4e9cfff8-5b13-4801-9a21-6a871ccbf656 tempest-TestVolumeSwap-1714109745 None] Geting {'code': 204} from DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:66000c93-eb12-4c23-8a69-71e1aa6b521f/luns/v_66000c93-eb12-4c23-8a69-71e1aa6b521f {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:27:05 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-6dca4c82-5919-4549-9ef9-2dd9c5ab78c8 req-4e9cfff8-5b13-4801-9a21-6a871ccbf656 tempest-TestVolumeSwap-1714109745 None] delete target: iqn.2010-10.org.openstack:66000c93-eb12-4c23-8a69-71e1aa6b521f {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:930}} Apr 17 15:27:05 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [req-6dca4c82-5919-4549-9ef9-2dd9c5ab78c8 req-4e9cfff8-5b13-4801-9a21-6a871ccbf656 tempest-TestVolumeSwap-1714109745 None] delete target iqn.2010-10.org.openstack:66000c93-eb12-4c23-8a69-71e1aa6b521f {{(pid=84696) delete_target /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:442}} Apr 17 15:27:05 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-6dca4c82-5919-4549-9ef9-2dd9c5ab78c8 req-4e9cfff8-5b13-4801-9a21-6a871ccbf656 tempest-TestVolumeSwap-1714109745 None] Sending pool request DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:66000c93-eb12-4c23-8a69-71e1aa6b521f {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:27:05 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-6dca4c82-5919-4549-9ef9-2dd9c5ab78c8 req-4e9cfff8-5b13-4801-9a21-6a871ccbf656 tempest-TestVolumeSwap-1714109745 None] Sending DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:66000c93-eb12-4c23-8a69-71e1aa6b521f {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:27:05 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-dc265cc2-2c37-4e0d-a41d-71d4ac546ea7 tempest-GroupSnapshotsV319Test-531716155 None] Geting {'code': 200, 'error': None, 'data': {'origin': 'Pool-0/v_d5a81ad1-94ca-4688-9d1d-2d953807886e@s_1761e65f-0800-4c2d-85cb-ce19f5f68b7a', 'relatime': None, 'acltype': None, 'vscan': None, 'full_name': 'Pool-0/s_1761e65f-0800-4c2d-85cb-ce19f5f68b7a', 'userrefs': None, 'primarycache': 'all', 'logbias': 'latency', 'creation': '1681746615', 'sync': 'always', 'is_clone': True, 'dedup': 'off', 'sharenfs': None, 'receive_resume_token': None, 'volsize': '1073741824', 'referenced': '57344', 'sharesmb': None, 'createtxg': '188122', 'reservation': '0', 'scontext': None, 'mountpoint': None, 'casesensitivity': None, 'guid': '7255129556551640345', 'usedbyrefreservation': '0', 'dnodesize': None, 'written': '0', 'logicalused': '0', 'compressratio': '1.00', 'rootcontext': 'none', 'default_scsi_id': '6bd3a8dd988105fe', 'type': 'volume', 'compression': 'lz4', 'snapdir': None, 'overlay': None, 'encryption': 'off', 'xattr': None, 'volmode': 'default', 'copies': '1', 'snapshot_limit': '18446744073709551615', 'aclinherit': None, 'defcontext': 'none', 'readonly': 'on', 'version': None, 'recordsize': None, 'filesystem_limit': None, 'mounted': None, 'mlslabel': 'none', 'secondarycache': 'all', 'refreservation': '0', 'available': '113889202176', 'used': '0', 'encryptionroot': None, 'exec': None, 'refquota': None, 'refcompressratio': '1.00', 'quota': None, 'utf8only': None, 'keylocation': 'none', 'snapdev': 'hidden', 'snapshot_count': '18446744073709551615', 'fscontext': 'none', 'clones': None, 'canmount': None, 'keystatus': None, 'atime': None, 'usedbysnapshots': '0', 'normalization': None, 'usedbychildren': '0', 'volblocksize': '131072', 'usedbydataset': '0', 'objsetid': '123306', 'name': 's_1761e65f-0800-4c2d-85cb-ce19f5f68b7a', 'defer_destroy': None, 'pbkdf2iters': '0', 'checksum': 'on', 'redundant_metadata': 'all', 'filesystem_count': None, 'devices': None, 'keyformat': 'none', 'setuid': None, 'logicalreferenced': '28672', 'context': 'none', 'zoned': None, 'nbmand': None}} from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/s_1761e65f-0800-4c2d-85cb-ce19f5f68b7a {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:27:05 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-dc265cc2-2c37-4e0d-a41d-71d4ac546ea7 tempest-GroupSnapshotsV319Test-531716155 None] delete snapshot s_1761e65f-0800-4c2d-85cb-ce19f5f68b7a of volume v_d5a81ad1-94ca-4688-9d1d-2d953807886e {{(pid=84696) delete_snapshot /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:751}} Apr 17 15:27:05 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-dc265cc2-2c37-4e0d-a41d-71d4ac546ea7 tempest-GroupSnapshotsV319Test-531716155 None] Sending pool request DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_d5a81ad1-94ca-4688-9d1d-2d953807886e/snapshots/s_1761e65f-0800-4c2d-85cb-ce19f5f68b7a {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:27:05 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-dc265cc2-2c37-4e0d-a41d-71d4ac546ea7 tempest-GroupSnapshotsV319Test-531716155 None] Sending DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_d5a81ad1-94ca-4688-9d1d-2d953807886e/snapshots/s_1761e65f-0800-4c2d-85cb-ce19f5f68b7a {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:27:05 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-6dca4c82-5919-4549-9ef9-2dd9c5ab78c8 req-4e9cfff8-5b13-4801-9a21-6a871ccbf656 tempest-TestVolumeSwap-1714109745 None] Geting {'code': 204} from DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:66000c93-eb12-4c23-8a69-71e1aa6b521f {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:27:05 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [req-6dca4c82-5919-4549-9ef9-2dd9c5ab78c8 req-4e9cfff8-5b13-4801-9a21-6a871ccbf656 tempest-TestVolumeSwap-1714109745 None] target iqn.2010-10.org.openstack:66000c93-eb12-4c23-8a69-71e1aa6b521f deleted {{(pid=84696) delete_target /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:447}} Apr 17 15:27:07 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-dc265cc2-2c37-4e0d-a41d-71d4ac546ea7 tempest-GroupSnapshotsV319Test-531716155 None] Geting {'code': 204} from DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_d5a81ad1-94ca-4688-9d1d-2d953807886e/snapshots/s_1761e65f-0800-4c2d-85cb-ce19f5f68b7a {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:27:07 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-dc265cc2-2c37-4e0d-a41d-71d4ac546ea7 tempest-GroupSnapshotsV319Test-531716155 None] snapshot s_1761e65f-0800-4c2d-85cb-ce19f5f68b7a deleted {{(pid=84696) delete_snapshot /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:772}} Apr 17 15:27:07 user cinder-volume[84696]: WARNING cinder.quota [None req-dc265cc2-2c37-4e0d-a41d-71d4ac546ea7 tempest-GroupSnapshotsV319Test-531716155 None] Deprecated: Default quota for resource: gigabytes_tempest-GroupSnapshotsV319Test-volume-type-880093158 is set by the default quota flag: quota_gigabytes_tempest-GroupSnapshotsV319Test-volume-type-880093158, it is now deprecated. Please use the default quota class for default quota. Apr 17 15:27:07 user cinder-volume[84696]: WARNING cinder.quota [None req-dc265cc2-2c37-4e0d-a41d-71d4ac546ea7 tempest-GroupSnapshotsV319Test-531716155 None] Deprecated: Default quota for resource: snapshots_tempest-GroupSnapshotsV319Test-volume-type-880093158 is set by the default quota flag: quota_snapshots_tempest-GroupSnapshotsV319Test-volume-type-880093158, it is now deprecated. Please use the default quota class for default quota. Apr 17 15:27:07 user cinder-volume[84696]: DEBUG cinder.quota [None req-dc265cc2-2c37-4e0d-a41d-71d4ac546ea7 tempest-GroupSnapshotsV319Test-531716155 None] Created reservations ['632ca44e-fe92-43f0-b3f1-d7964b9beec3', '9c7229f6-9fa7-402c-b821-a3e9306dc3e4', 'eaa1fd07-f783-417d-b3e3-1b7215793f35', '57e55355-39af-434b-b3d5-69afdc11e06b'] {{(pid=84696) reserve /opt/stack/cinder/cinder/quota.py:755}} Apr 17 15:27:07 user cinder-volume[84696]: INFO cinder.volume.manager [None req-dc265cc2-2c37-4e0d-a41d-71d4ac546ea7 tempest-GroupSnapshotsV319Test-531716155 None] group_snapshot f4dcb486-d239-4af8-9532-6ea5d66ec840: deleted successfully Apr 17 15:27:08 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=70,cinder:UPDATE=25,cinder:INSERT=4,cinder:DELETE=1 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:27:08 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-805f6e6b-96bf-4188-8fef-61d9e93617c2 tempest-GroupSnapshotsV319Test-531716155 None] remove_export for volume: d5a81ad1-94ca-4688-9d1d-2d953807886e. {{(pid=84696) remove_export /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:718}} Apr 17 15:27:08 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-805f6e6b-96bf-4188-8fef-61d9e93617c2 tempest-GroupSnapshotsV319Test-531716155 None] remove export {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:912}} Apr 17 15:27:08 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-805f6e6b-96bf-4188-8fef-61d9e93617c2 tempest-GroupSnapshotsV319Test-531716155 None] detach volume:Volume(_name_id=None,admin_metadata={},attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:26:57Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-volume-609793849',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=7cfa2150-9d68-4591-b1b3-bd3a13002e9e,host='user@joviandss-2#joviandss-2',id=d5a81ad1-94ca-4688-9d1d-2d953807886e,launched_at=2023-04-17T15:26:59Z,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='dd1792b234e44b5faf547b359e08667b',provider_auth='CHAP HwCJXFgi jCWUHxoltz0kO2',provider_geometry=None,provider_id=None,provider_location='172.16.0.220:3260,1 iqn.2010-10.org.openstack:d5a81ad1-94ca-4688-9d1d-2d953807886e 0',replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:26:57Z,service_uuid='e628352e-c4cf-40eb-8389-cce40663026a',shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='deleting',terminated_at=None,updated_at=2023-04-17T15:27:09Z,use_quota=True,user_id='bb9c1925c06c40c481c028493885ea57',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(3710bfe6-8491-42c0-8fc3-c1f34b0279ed),volume_type_id=3710bfe6-8491-42c0-8fc3-c1f34b0279ed) from target:iqn.2010-10.org.openstack:d5a81ad1-94ca-4688-9d1d-2d953807886e. {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:913}} Apr 17 15:27:08 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-805f6e6b-96bf-4188-8fef-61d9e93617c2 tempest-GroupSnapshotsV319Test-531716155 None] detach volume v_d5a81ad1-94ca-4688-9d1d-2d953807886e from target iqn.2010-10.org.openstack:d5a81ad1-94ca-4688-9d1d-2d953807886e {{(pid=84696) detach_target_vol /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:606}} Apr 17 15:27:08 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-805f6e6b-96bf-4188-8fef-61d9e93617c2 tempest-GroupSnapshotsV319Test-531716155 None] Sending pool request DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:d5a81ad1-94ca-4688-9d1d-2d953807886e/luns/v_d5a81ad1-94ca-4688-9d1d-2d953807886e {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:27:08 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-805f6e6b-96bf-4188-8fef-61d9e93617c2 tempest-GroupSnapshotsV319Test-531716155 None] Sending DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:d5a81ad1-94ca-4688-9d1d-2d953807886e/luns/v_d5a81ad1-94ca-4688-9d1d-2d953807886e {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:27:08 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-805f6e6b-96bf-4188-8fef-61d9e93617c2 tempest-GroupSnapshotsV319Test-531716155 None] Geting {'code': 404, 'error': {'class': 'opene.exceptions.ItemNotFoundError', 'message': 'Target iqn.2010-10.org.openstack:d5a81ad1-94ca-4688-9d1d-2d953807886e not exists.', 'url': 'http://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:d5a81ad1-94ca-4688-9d1d-2d953807886e/luns/v_d5a81ad1-94ca-4688-9d1d-2d953807886e'}, 'data': None} from DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:d5a81ad1-94ca-4688-9d1d-2d953807886e/luns/v_d5a81ad1-94ca-4688-9d1d-2d953807886e {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:27:08 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-805f6e6b-96bf-4188-8fef-61d9e93617c2 tempest-GroupSnapshotsV319Test-531716155 None] failed to remove resource iqn.2010-10.org.openstack:d5a81ad1-94ca-4688-9d1d-2d953807886e because of JDSS resource v_d5a81ad1-94ca-4688-9d1d-2d953807886e DNE. {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:920}} Apr 17 15:27:08 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-805f6e6b-96bf-4188-8fef-61d9e93617c2 tempest-GroupSnapshotsV319Test-531716155 None] delete target: iqn.2010-10.org.openstack:d5a81ad1-94ca-4688-9d1d-2d953807886e {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:930}} Apr 17 15:27:08 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-805f6e6b-96bf-4188-8fef-61d9e93617c2 tempest-GroupSnapshotsV319Test-531716155 None] delete target iqn.2010-10.org.openstack:d5a81ad1-94ca-4688-9d1d-2d953807886e {{(pid=84696) delete_target /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:442}} Apr 17 15:27:08 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-805f6e6b-96bf-4188-8fef-61d9e93617c2 tempest-GroupSnapshotsV319Test-531716155 None] Sending pool request DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:d5a81ad1-94ca-4688-9d1d-2d953807886e {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:27:08 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-805f6e6b-96bf-4188-8fef-61d9e93617c2 tempest-GroupSnapshotsV319Test-531716155 None] Sending DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:d5a81ad1-94ca-4688-9d1d-2d953807886e {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:27:08 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-805f6e6b-96bf-4188-8fef-61d9e93617c2 tempest-GroupSnapshotsV319Test-531716155 None] Geting {'code': 404, 'error': {'class': 'opene.exceptions.ItemNotFoundError', 'message': 'Target iqn.2010-10.org.openstack:d5a81ad1-94ca-4688-9d1d-2d953807886e not exists.', 'url': 'http://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:d5a81ad1-94ca-4688-9d1d-2d953807886e'}, 'data': None} from DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:d5a81ad1-94ca-4688-9d1d-2d953807886e {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:27:08 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-805f6e6b-96bf-4188-8fef-61d9e93617c2 tempest-GroupSnapshotsV319Test-531716155 None] failed to remove resource iqn.2010-10.org.openstack:d5a81ad1-94ca-4688-9d1d-2d953807886e because of JDSS resource iqn.2010-10.org.openstack:d5a81ad1-94ca-4688-9d1d-2d953807886e DNE. {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:935}} Apr 17 15:27:08 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-805f6e6b-96bf-4188-8fef-61d9e93617c2 tempest-GroupSnapshotsV319Test-531716155 None] deleating volume v_d5a81ad1-94ca-4688-9d1d-2d953807886e {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 15:27:08 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-805f6e6b-96bf-4188-8fef-61d9e93617c2 tempest-GroupSnapshotsV319Test-531716155 None] get snapshots for volume v_d5a81ad1-94ca-4688-9d1d-2d953807886e {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 15:27:08 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-805f6e6b-96bf-4188-8fef-61d9e93617c2 tempest-GroupSnapshotsV319Test-531716155 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_d5a81ad1-94ca-4688-9d1d-2d953807886e/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:27:08 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-805f6e6b-96bf-4188-8fef-61d9e93617c2 tempest-GroupSnapshotsV319Test-531716155 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_d5a81ad1-94ca-4688-9d1d-2d953807886e/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:27:09 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-805f6e6b-96bf-4188-8fef-61d9e93617c2 tempest-GroupSnapshotsV319Test-531716155 None] Geting {'code': 200, 'error': None, 'data': {'results': 0, 'entries': []}} from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_d5a81ad1-94ca-4688-9d1d-2d953807886e/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:27:09 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-805f6e6b-96bf-4188-8fef-61d9e93617c2 tempest-GroupSnapshotsV319Test-531716155 None] get volume v_d5a81ad1-94ca-4688-9d1d-2d953807886e info {{(pid=84696) get_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:239}} Apr 17 15:27:09 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-805f6e6b-96bf-4188-8fef-61d9e93617c2 tempest-GroupSnapshotsV319Test-531716155 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_d5a81ad1-94ca-4688-9d1d-2d953807886e {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:27:09 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-805f6e6b-96bf-4188-8fef-61d9e93617c2 tempest-GroupSnapshotsV319Test-531716155 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_d5a81ad1-94ca-4688-9d1d-2d953807886e {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:27:09 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-805f6e6b-96bf-4188-8fef-61d9e93617c2 tempest-GroupSnapshotsV319Test-531716155 None] Geting {'code': 200, 'error': None, 'data': {'origin': None, 'relatime': None, 'acltype': None, 'vscan': None, 'full_name': 'Pool-0/v_d5a81ad1-94ca-4688-9d1d-2d953807886e', 'userrefs': None, 'primarycache': 'all', 'logbias': 'latency', 'creation': '1681746611', 'sync': 'always', 'is_clone': False, 'dedup': 'off', 'sharenfs': None, 'receive_resume_token': None, 'volsize': '1073741824', 'referenced': '57344', 'sharesmb': None, 'createtxg': '188111', 'reservation': '0', 'scontext': None, 'mountpoint': None, 'casesensitivity': None, 'guid': '12701657509359287122', 'usedbyrefreservation': '0', 'dnodesize': None, 'written': '57344', 'logicalused': '28672', 'compressratio': '1.00', 'rootcontext': 'none', 'default_scsi_id': 'dd83f04abb4db76f', 'type': 'volume', 'compression': 'lz4', 'snapdir': None, 'overlay': None, 'encryption': 'off', 'xattr': None, 'volmode': 'default', 'copies': '1', 'snapshot_limit': '18446744073709551615', 'aclinherit': None, 'defcontext': 'none', 'readonly': 'off', 'version': None, 'recordsize': None, 'filesystem_limit': None, 'mounted': None, 'mlslabel': 'none', 'secondarycache': 'all', 'refreservation': '0', 'available': '113889214464', 'san:volume_id': 'dd83f04abb4db76f86efc1e1add2ed6f6ddf1189', 'encryptionroot': None, 'exec': None, 'refquota': None, 'refcompressratio': '1.00', 'quota': None, 'utf8only': None, 'keylocation': 'none', 'snapdev': 'hidden', 'snapshot_count': '18446744073709551615', 'fscontext': 'none', 'clones': None, 'canmount': None, 'keystatus': None, 'atime': None, 'usedbysnapshots': '0', 'normalization': None, 'usedbychildren': '0', 'volblocksize': '131072', 'usedbydataset': '57344', 'objsetid': '123293', 'name': 'v_d5a81ad1-94ca-4688-9d1d-2d953807886e', 'defer_destroy': None, 'pbkdf2iters': '0', 'checksum': 'on', 'redundant_metadata': 'all', 'filesystem_count': None, 'devices': None, 'keyformat': 'none', 'setuid': None, 'used': '57344', 'logicalreferenced': '28672', 'context': 'none', 'zoned': None, 'nbmand': None}} from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_d5a81ad1-94ca-4688-9d1d-2d953807886e {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:27:09 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-805f6e6b-96bf-4188-8fef-61d9e93617c2 tempest-GroupSnapshotsV319Test-531716155 None] delete volume:v_d5a81ad1-94ca-4688-9d1d-2d953807886e recursively children:{'force_umount': True} {{(pid=84696) delete_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:335}} Apr 17 15:27:09 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-805f6e6b-96bf-4188-8fef-61d9e93617c2 tempest-GroupSnapshotsV319Test-531716155 None] Sending pool request DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_d5a81ad1-94ca-4688-9d1d-2d953807886e {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:27:09 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-805f6e6b-96bf-4188-8fef-61d9e93617c2 tempest-GroupSnapshotsV319Test-531716155 None] Sending DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_d5a81ad1-94ca-4688-9d1d-2d953807886e {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:27:09 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-6a324b90-a845-4cbf-a525-b6a8ba5fe285 tempest-VolumesActionsTest-707453245 None] Volume reschedule parameters: True retry: {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 15:27:09 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-6a324b90-a845-4cbf-a525-b6a8ba5fe285 tempest-VolumesActionsTest-707453245 None] Flow 'volume_create_manager' (5a7f5ac5-f0e3-4524-bede-2e1d64fc3185) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:27:09 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-6a324b90-a845-4cbf-a525-b6a8ba5fe285 tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (e7683ce4-7c73-402c-9e16-773983400974) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:27:09 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-6a324b90-a845-4cbf-a525-b6a8ba5fe285 tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (e7683ce4-7c73-402c-9e16-773983400974) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:27:09Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesActionsTest-Volume-1049939874',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=7d13a2d4-818a-4fbe-961d-1cd3c7a4cc75,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='a8b97ff6ed6047f0a33e74684a1c162b',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:27:09Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:27:09Z,use_quota=True,user_id='c993ceeb4a5c49eea14057e2125518ad',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:27:09 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-6a324b90-a845-4cbf-a525-b6a8ba5fe285 tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (2a2e5822-df96-489b-b478-c6b66f5f26d9) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:27:09 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-6a324b90-a845-4cbf-a525-b6a8ba5fe285 tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (2a2e5822-df96-489b-b478-c6b66f5f26d9) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:27:09 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-6a324b90-a845-4cbf-a525-b6a8ba5fe285 tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (7fb76576-9523-4cba-8b86-a3d80d810d2b) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:27:09 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-6a324b90-a845-4cbf-a525-b6a8ba5fe285 tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (7fb76576-9523-4cba-8b86-a3d80d810d2b) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': '7d13a2d4-818a-4fbe-961d-1cd3c7a4cc75', 'volume_name': 'volume-7d13a2d4-818a-4fbe-961d-1cd3c7a4cc75', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:27:09 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-6a324b90-a845-4cbf-a525-b6a8ba5fe285 tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (0ae666af-9982-4ccf-bb0d-56601f4a8032) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:27:09 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-6a324b90-a845-4cbf-a525-b6a8ba5fe285 tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (0ae666af-9982-4ccf-bb0d-56601f4a8032) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:27:09 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-6a324b90-a845-4cbf-a525-b6a8ba5fe285 tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (fb074c06-9445-45a0-88d1-bdefb35ce300) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:27:09 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-6a324b90-a845-4cbf-a525-b6a8ba5fe285 tempest-VolumesActionsTest-707453245 None] Volume 7d13a2d4-818a-4fbe-961d-1cd3c7a4cc75: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-7d13a2d4-818a-4fbe-961d-1cd3c7a4cc75', 'volume_size': 1} Apr 17 15:27:09 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-6a324b90-a845-4cbf-a525-b6a8ba5fe285 tempest-VolumesActionsTest-707453245 None] creating volume v_7d13a2d4-818a-4fbe-961d-1cd3c7a4cc75. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 15:27:09 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-6a324b90-a845-4cbf-a525-b6a8ba5fe285 tempest-VolumesActionsTest-707453245 None] create volume {'name': 'v_7d13a2d4-818a-4fbe-961d-1cd3c7a4cc75', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 15:27:09 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-6a324b90-a845-4cbf-a525-b6a8ba5fe285 tempest-VolumesActionsTest-707453245 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:27:09 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-6a324b90-a845-4cbf-a525-b6a8ba5fe285 tempest-VolumesActionsTest-707453245 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:27:11 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-805f6e6b-96bf-4188-8fef-61d9e93617c2 tempest-GroupSnapshotsV319Test-531716155 None] Geting {'code': 204} from DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_d5a81ad1-94ca-4688-9d1d-2d953807886e {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:27:11 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-805f6e6b-96bf-4188-8fef-61d9e93617c2 tempest-GroupSnapshotsV319Test-531716155 None] volume v_d5a81ad1-94ca-4688-9d1d-2d953807886e deleted {{(pid=84696) delete_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:346}} Apr 17 15:27:11 user cinder-volume[84696]: DEBUG cinder.quota [None req-805f6e6b-96bf-4188-8fef-61d9e93617c2 tempest-GroupSnapshotsV319Test-531716155 None] Created reservations ['abec90c8-00df-442f-ba46-c0fde94365b0'] {{(pid=84696) reserve /opt/stack/cinder/cinder/quota.py:755}} Apr 17 15:27:11 user cinder-volume[84696]: WARNING cinder.quota [None req-805f6e6b-96bf-4188-8fef-61d9e93617c2 tempest-GroupSnapshotsV319Test-531716155 None] Deprecated: Default quota for resource: volumes_tempest-GroupSnapshotsV319Test-volume-type-880093158 is set by the default quota flag: quota_volumes_tempest-GroupSnapshotsV319Test-volume-type-880093158, it is now deprecated. Please use the default quota class for default quota. Apr 17 15:27:11 user cinder-volume[84696]: DEBUG cinder.quota [None req-805f6e6b-96bf-4188-8fef-61d9e93617c2 tempest-GroupSnapshotsV319Test-531716155 None] Created reservations ['486dd429-34dc-40ee-ae18-030986a9966c', '4b777cb6-1dfd-45b2-822c-9444add57526', '3dd6d024-03c5-44b4-85ce-60d338c2f2db', 'aec3fd00-c675-4865-8b00-2cf9de50dd6c'] {{(pid=84696) reserve /opt/stack/cinder/cinder/quota.py:755}} Apr 17 15:27:11 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-805f6e6b-96bf-4188-8fef-61d9e93617c2 tempest-GroupSnapshotsV319Test-531716155 None] Updating volume stats {{(pid=84696) _update_volume_stats /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:724}} Apr 17 15:27:11 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-805f6e6b-96bf-4188-8fef-61d9e93617c2 tempest-GroupSnapshotsV319Test-531716155 None] Get pool Pool-0 fsprops {{(pid=84696) get_pool_stats /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:894}} Apr 17 15:27:11 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-805f6e6b-96bf-4188-8fef-61d9e93617c2 tempest-GroupSnapshotsV319Test-531716155 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:27:11 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-805f6e6b-96bf-4188-8fef-61d9e93617c2 tempest-GroupSnapshotsV319Test-531716155 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:27:12 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-6a324b90-a845-4cbf-a525-b6a8ba5fe285 tempest-VolumesActionsTest-707453245 None] Geting {'code': 201, 'error': None, 'data': {'full_name': 'Pool-0/v_7d13a2d4-818a-4fbe-961d-1cd3c7a4cc75', 'is_clone': False, 'name': 'v_7d13a2d4-818a-4fbe-961d-1cd3c7a4cc75'}} from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:27:12 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-6a324b90-a845-4cbf-a525-b6a8ba5fe285 tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (fb074c06-9445-45a0-88d1-bdefb35ce300) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'volume_name': 'volume-7d13a2d4-818a-4fbe-961d-1cd3c7a4cc75', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:27:12 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-6a324b90-a845-4cbf-a525-b6a8ba5fe285 tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeOnFinishTask;volume:create, create.end' (efc9fcec-bf33-426b-993b-c81fdf01b567) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:27:12 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-6a324b90-a845-4cbf-a525-b6a8ba5fe285 tempest-VolumesActionsTest-707453245 None] Volume volume-7d13a2d4-818a-4fbe-961d-1cd3c7a4cc75 (7d13a2d4-818a-4fbe-961d-1cd3c7a4cc75): created successfully Apr 17 15:27:12 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-6a324b90-a845-4cbf-a525-b6a8ba5fe285 tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeOnFinishTask;volume:create, create.end' (efc9fcec-bf33-426b-993b-c81fdf01b567) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:27:12 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-6a324b90-a845-4cbf-a525-b6a8ba5fe285 tempest-VolumesActionsTest-707453245 None] Flow 'volume_create_manager' (5a7f5ac5-f0e3-4524-bede-2e1d64fc3185) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:27:12 user cinder-volume[84696]: INFO cinder.volume.manager [None req-6a324b90-a845-4cbf-a525-b6a8ba5fe285 tempest-VolumesActionsTest-707453245 None] Created volume successfully. Apr 17 15:27:13 user cinder-volume[84696]: DEBUG cinder.coordination [None req-9db5dd95-4a7f-4604-bd3b-f9f7addec122 tempest-VolumesActionsTest-707453245 None] Acquiring lock "cinder-7d13a2d4-818a-4fbe-961d-1cd3c7a4cc75" by "attach_volume" {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:192}} Apr 17 15:27:13 user cinder-volume[84696]: DEBUG cinder.coordination [None req-9db5dd95-4a7f-4604-bd3b-f9f7addec122 tempest-VolumesActionsTest-707453245 None] Lock "cinder-7d13a2d4-818a-4fbe-961d-1cd3c7a4cc75" acquired by "attach_volume" :: waited 0.022s {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:196}} Apr 17 15:27:13 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-805f6e6b-96bf-4188-8fef-61d9e93617c2 tempest-GroupSnapshotsV319Test-531716155 None] Geting {'code': 200, 'error': None, 'data': {'available': '113889226752', 'status': 30, 'import_status': {'import_steps': {'san_setup': True, 'vip_setup': True, 'zfs_import': True, 'mount_system_volume': True, 'nas_setup': True}, 'import_successful': True}, 'scan': None, 'encryption': {'enabled': False}, 'deduplication_rate': '1.00', 'sysvol_upgrade': 'not_needed', 'vdevs': [{'name': 'wwn-0x6000c296b793bf526a107a9d823e947a', 'iostats': {'read': '0', 'write': '0', 'chksum': '0'}, 'disks': [{'origin': 'local', 'slot': '', 'led': 'off', 'name': 'sdn', 'iostats': {'read': '0', 'write': '0', 'chksum': '0'}, 'alias': '', 'health': 'ONLINE', 'sn': '6000c296b793bf526a107a9d823e947a', 'trim_data': {'status': 'trim_unsupported', 'progress': None, 'trim_time': None}, 'path': 'pci-0000:03:00.0-scsi-0:0:14:0', 'model': 'Virtual disk', 'id': 'wwn-0x6000c296b793bf526a107a9d823e947a', 'size': 128849018880}], 'health': 'ONLINE', 'vdev_replacings': [], 'vdev_spares': [], 'type': ''}], 'id': '14181317334440137330', 'health': 'ONLINE', 'iostats': {'read': '0', 'write': '0', 'chksum': '0'}, 'operation': 'none', 'size': '127775277056', 'autotrim': False, 'name': 'Pool-0'}} from GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:27:13 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-805f6e6b-96bf-4188-8fef-61d9e93617c2 tempest-GroupSnapshotsV319Test-531716155 None] Total capacity: 119, Free 106. {{(pid=84696) _update_volume_stats /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:757}} Apr 17 15:27:13 user cinder-volume[84696]: DEBUG cinder.manager [None req-805f6e6b-96bf-4188-8fef-61d9e93617c2 tempest-GroupSnapshotsV319Test-531716155 None] Notifying Schedulers of capabilities ... {{(pid=84696) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 17 15:27:13 user cinder-volume[84696]: INFO cinder.volume.manager [None req-805f6e6b-96bf-4188-8fef-61d9e93617c2 tempest-GroupSnapshotsV319Test-531716155 None] Delete group completed successfully. Apr 17 15:27:13 user cinder-volume[84696]: INFO cinder.volume.manager [None req-9db5dd95-4a7f-4604-bd3b-f9f7addec122 tempest-VolumesActionsTest-707453245 None] Attaching volume 7d13a2d4-818a-4fbe-961d-1cd3c7a4cc75 to instance ddc8a602-70e9-4697-b75b-c602ad4c2447 at mountpoint /dev/vdb on host None. Apr 17 15:27:13 user cinder-volume[84696]: INFO cinder.volume.manager [None req-9db5dd95-4a7f-4604-bd3b-f9f7addec122 tempest-VolumesActionsTest-707453245 None] Attach volume completed successfully. Apr 17 15:27:13 user cinder-volume[84696]: DEBUG cinder.coordination [None req-9db5dd95-4a7f-4604-bd3b-f9f7addec122 tempest-VolumesActionsTest-707453245 None] Lock "cinder-7d13a2d4-818a-4fbe-961d-1cd3c7a4cc75" released by "attach_volume" :: held 0.114s {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:207}} Apr 17 15:27:14 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-4286f46d-10bb-4270-b53b-04a7d15c62f1 tempest-VolumesActionsTest-649676615 None] terminate connection for 7d13a2d4-818a-4fbe-961d-1cd3c7a4cc75 {{(pid=84696) terminate_connection /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:1032}} Apr 17 15:27:14 user cinder-volume[84696]: INFO cinder.volume.manager [None req-4286f46d-10bb-4270-b53b-04a7d15c62f1 tempest-VolumesActionsTest-649676615 None] Terminate volume connection completed successfully. Apr 17 15:27:14 user cinder-volume[84696]: DEBUG cinder.coordination [None req-4286f46d-10bb-4270-b53b-04a7d15c62f1 tempest-VolumesActionsTest-649676615 None] Acquiring lock "cinder-7d13a2d4-818a-4fbe-961d-1cd3c7a4cc75-detach_volume" by "detach_volume" {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:192}} Apr 17 15:27:14 user cinder-volume[84696]: DEBUG cinder.coordination [None req-4286f46d-10bb-4270-b53b-04a7d15c62f1 tempest-VolumesActionsTest-649676615 None] Lock "cinder-7d13a2d4-818a-4fbe-961d-1cd3c7a4cc75-detach_volume" acquired by "detach_volume" :: waited 0.011s {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:196}} Apr 17 15:27:14 user cinder-volume[84696]: INFO cinder.volume.manager [None req-4286f46d-10bb-4270-b53b-04a7d15c62f1 tempest-VolumesActionsTest-649676615 None] Detaching volume 7d13a2d4-818a-4fbe-961d-1cd3c7a4cc75 from instance ddc8a602-70e9-4697-b75b-c602ad4c2447. Apr 17 15:27:14 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-4286f46d-10bb-4270-b53b-04a7d15c62f1 tempest-VolumesActionsTest-649676615 None] remove_export for volume: 7d13a2d4-818a-4fbe-961d-1cd3c7a4cc75. {{(pid=84696) remove_export /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:718}} Apr 17 15:27:14 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-4286f46d-10bb-4270-b53b-04a7d15c62f1 tempest-VolumesActionsTest-649676615 None] remove export {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:912}} Apr 17 15:27:14 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-4286f46d-10bb-4270-b53b-04a7d15c62f1 tempest-VolumesActionsTest-649676615 None] detach volume:Volume(_name_id=None,admin_metadata={attached_mode='rw',readonly='False'},attach_status='attached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:27:09Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesActionsTest-Volume-1049939874',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=7d13a2d4-818a-4fbe-961d-1cd3c7a4cc75,launched_at=2023-04-17T15:27:13Z,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='a8b97ff6ed6047f0a33e74684a1c162b',provider_auth='CHAP yUbaimus Jw6GUhogNIKkFa',provider_geometry=None,provider_id=None,provider_location='172.16.0.220:3260,1 iqn.2010-10.org.openstack:7d13a2d4-818a-4fbe-961d-1cd3c7a4cc75 0',replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:27:09Z,service_uuid='e628352e-c4cf-40eb-8389-cce40663026a',shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='error',terminated_at=None,updated_at=2023-04-17T15:27:14Z,use_quota=True,user_id='c993ceeb4a5c49eea14057e2125518ad',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c) from target:iqn.2010-10.org.openstack:7d13a2d4-818a-4fbe-961d-1cd3c7a4cc75. {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:913}} Apr 17 15:27:14 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-4286f46d-10bb-4270-b53b-04a7d15c62f1 tempest-VolumesActionsTest-649676615 None] detach volume v_7d13a2d4-818a-4fbe-961d-1cd3c7a4cc75 from target iqn.2010-10.org.openstack:7d13a2d4-818a-4fbe-961d-1cd3c7a4cc75 {{(pid=84696) detach_target_vol /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:606}} Apr 17 15:27:14 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-4286f46d-10bb-4270-b53b-04a7d15c62f1 tempest-VolumesActionsTest-649676615 None] Sending pool request DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:7d13a2d4-818a-4fbe-961d-1cd3c7a4cc75/luns/v_7d13a2d4-818a-4fbe-961d-1cd3c7a4cc75 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:27:14 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-4286f46d-10bb-4270-b53b-04a7d15c62f1 tempest-VolumesActionsTest-649676615 None] Sending DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:7d13a2d4-818a-4fbe-961d-1cd3c7a4cc75/luns/v_7d13a2d4-818a-4fbe-961d-1cd3c7a4cc75 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:27:14 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-4286f46d-10bb-4270-b53b-04a7d15c62f1 tempest-VolumesActionsTest-649676615 None] Geting {'code': 404, 'error': {'class': 'opene.exceptions.ItemNotFoundError', 'message': 'Target iqn.2010-10.org.openstack:7d13a2d4-818a-4fbe-961d-1cd3c7a4cc75 not exists.', 'url': 'http://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:7d13a2d4-818a-4fbe-961d-1cd3c7a4cc75/luns/v_7d13a2d4-818a-4fbe-961d-1cd3c7a4cc75'}, 'data': None} from DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:7d13a2d4-818a-4fbe-961d-1cd3c7a4cc75/luns/v_7d13a2d4-818a-4fbe-961d-1cd3c7a4cc75 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:27:14 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-4286f46d-10bb-4270-b53b-04a7d15c62f1 tempest-VolumesActionsTest-649676615 None] failed to remove resource iqn.2010-10.org.openstack:7d13a2d4-818a-4fbe-961d-1cd3c7a4cc75 because of JDSS resource v_7d13a2d4-818a-4fbe-961d-1cd3c7a4cc75 DNE. {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:920}} Apr 17 15:27:14 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-4286f46d-10bb-4270-b53b-04a7d15c62f1 tempest-VolumesActionsTest-649676615 None] delete target: iqn.2010-10.org.openstack:7d13a2d4-818a-4fbe-961d-1cd3c7a4cc75 {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:930}} Apr 17 15:27:14 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-4286f46d-10bb-4270-b53b-04a7d15c62f1 tempest-VolumesActionsTest-649676615 None] delete target iqn.2010-10.org.openstack:7d13a2d4-818a-4fbe-961d-1cd3c7a4cc75 {{(pid=84696) delete_target /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:442}} Apr 17 15:27:14 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-4286f46d-10bb-4270-b53b-04a7d15c62f1 tempest-VolumesActionsTest-649676615 None] Sending pool request DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:7d13a2d4-818a-4fbe-961d-1cd3c7a4cc75 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:27:14 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-4286f46d-10bb-4270-b53b-04a7d15c62f1 tempest-VolumesActionsTest-649676615 None] Sending DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:7d13a2d4-818a-4fbe-961d-1cd3c7a4cc75 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:27:14 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-4286f46d-10bb-4270-b53b-04a7d15c62f1 tempest-VolumesActionsTest-649676615 None] Geting {'code': 404, 'error': {'class': 'opene.exceptions.ItemNotFoundError', 'message': 'Target iqn.2010-10.org.openstack:7d13a2d4-818a-4fbe-961d-1cd3c7a4cc75 not exists.', 'url': 'http://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:7d13a2d4-818a-4fbe-961d-1cd3c7a4cc75'}, 'data': None} from DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:7d13a2d4-818a-4fbe-961d-1cd3c7a4cc75 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:27:14 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-4286f46d-10bb-4270-b53b-04a7d15c62f1 tempest-VolumesActionsTest-649676615 None] failed to remove resource iqn.2010-10.org.openstack:7d13a2d4-818a-4fbe-961d-1cd3c7a4cc75 because of JDSS resource iqn.2010-10.org.openstack:7d13a2d4-818a-4fbe-961d-1cd3c7a4cc75 DNE. {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:935}} Apr 17 15:27:14 user cinder-volume[84696]: INFO cinder.volume.manager [None req-4286f46d-10bb-4270-b53b-04a7d15c62f1 tempest-VolumesActionsTest-649676615 None] Detach volume completed successfully. Apr 17 15:27:14 user cinder-volume[84696]: DEBUG cinder.coordination [None req-4286f46d-10bb-4270-b53b-04a7d15c62f1 tempest-VolumesActionsTest-649676615 None] Lock "cinder-7d13a2d4-818a-4fbe-961d-1cd3c7a4cc75-detach_volume" released by "detach_volume" :: held 0.104s {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:207}} Apr 17 15:27:14 user cinder-volume[84696]: DEBUG cinder.coordination [None req-91f05be3-e6a5-44b9-95e8-2cfdf58aca19 tempest-VolumesActionsTest-707453245 None] Acquiring lock "cinder-7d13a2d4-818a-4fbe-961d-1cd3c7a4cc75-detach_volume" by "detach_volume" {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:192}} Apr 17 15:27:14 user cinder-volume[84696]: DEBUG cinder.coordination [None req-91f05be3-e6a5-44b9-95e8-2cfdf58aca19 tempest-VolumesActionsTest-707453245 None] Lock "cinder-7d13a2d4-818a-4fbe-961d-1cd3c7a4cc75-detach_volume" acquired by "detach_volume" :: waited 0.011s {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:196}} Apr 17 15:27:14 user cinder-volume[84696]: INFO cinder.volume.manager [None req-91f05be3-e6a5-44b9-95e8-2cfdf58aca19 tempest-VolumesActionsTest-707453245 None] Volume detach called, but volume not attached. Apr 17 15:27:14 user cinder-volume[84696]: DEBUG cinder.coordination [None req-91f05be3-e6a5-44b9-95e8-2cfdf58aca19 tempest-VolumesActionsTest-707453245 None] Lock "cinder-7d13a2d4-818a-4fbe-961d-1cd3c7a4cc75-detach_volume" released by "detach_volume" :: held 0.016s {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:207}} Apr 17 15:27:14 user cinder-volume[84696]: DEBUG cinder.coordination [None req-7985a04f-f20c-4e04-8b5b-ceee9484c226 tempest-VolumesActionsTest-707453245 None] Acquiring lock "cinder-7d13a2d4-818a-4fbe-961d-1cd3c7a4cc75-delete_volume" by "delete_volume" {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:192}} Apr 17 15:27:14 user cinder-volume[84696]: DEBUG cinder.coordination [None req-7985a04f-f20c-4e04-8b5b-ceee9484c226 tempest-VolumesActionsTest-707453245 None] Lock "cinder-7d13a2d4-818a-4fbe-961d-1cd3c7a4cc75-delete_volume" acquired by "delete_volume" :: waited 0.009s {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:196}} Apr 17 15:27:14 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-7985a04f-f20c-4e04-8b5b-ceee9484c226 tempest-VolumesActionsTest-707453245 None] remove_export for volume: 7d13a2d4-818a-4fbe-961d-1cd3c7a4cc75. {{(pid=84696) remove_export /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:718}} Apr 17 15:27:14 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-7985a04f-f20c-4e04-8b5b-ceee9484c226 tempest-VolumesActionsTest-707453245 None] remove export {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:912}} Apr 17 15:27:14 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-7985a04f-f20c-4e04-8b5b-ceee9484c226 tempest-VolumesActionsTest-707453245 None] detach volume:Volume(_name_id=None,admin_metadata={readonly='False'},attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:27:09Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesActionsTest-Volume-1049939874',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=7d13a2d4-818a-4fbe-961d-1cd3c7a4cc75,launched_at=2023-04-17T15:27:13Z,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='a8b97ff6ed6047f0a33e74684a1c162b',provider_auth='CHAP yUbaimus Jw6GUhogNIKkFa',provider_geometry=None,provider_id=None,provider_location='172.16.0.220:3260,1 iqn.2010-10.org.openstack:7d13a2d4-818a-4fbe-961d-1cd3c7a4cc75 0',replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:27:09Z,service_uuid='e628352e-c4cf-40eb-8389-cce40663026a',shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='deleting',terminated_at=2023-04-17T15:27:15Z,updated_at=2023-04-17T15:27:15Z,use_quota=True,user_id='c993ceeb4a5c49eea14057e2125518ad',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c) from target:iqn.2010-10.org.openstack:7d13a2d4-818a-4fbe-961d-1cd3c7a4cc75. {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:913}} Apr 17 15:27:14 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-7985a04f-f20c-4e04-8b5b-ceee9484c226 tempest-VolumesActionsTest-707453245 None] detach volume v_7d13a2d4-818a-4fbe-961d-1cd3c7a4cc75 from target iqn.2010-10.org.openstack:7d13a2d4-818a-4fbe-961d-1cd3c7a4cc75 {{(pid=84696) detach_target_vol /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:606}} Apr 17 15:27:14 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-7985a04f-f20c-4e04-8b5b-ceee9484c226 tempest-VolumesActionsTest-707453245 None] Sending pool request DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:7d13a2d4-818a-4fbe-961d-1cd3c7a4cc75/luns/v_7d13a2d4-818a-4fbe-961d-1cd3c7a4cc75 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:27:14 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-7985a04f-f20c-4e04-8b5b-ceee9484c226 tempest-VolumesActionsTest-707453245 None] Sending DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:7d13a2d4-818a-4fbe-961d-1cd3c7a4cc75/luns/v_7d13a2d4-818a-4fbe-961d-1cd3c7a4cc75 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:27:14 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-7985a04f-f20c-4e04-8b5b-ceee9484c226 tempest-VolumesActionsTest-707453245 None] Geting {'code': 404, 'error': {'class': 'opene.exceptions.ItemNotFoundError', 'message': 'Target iqn.2010-10.org.openstack:7d13a2d4-818a-4fbe-961d-1cd3c7a4cc75 not exists.', 'url': 'http://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:7d13a2d4-818a-4fbe-961d-1cd3c7a4cc75/luns/v_7d13a2d4-818a-4fbe-961d-1cd3c7a4cc75'}, 'data': None} from DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:7d13a2d4-818a-4fbe-961d-1cd3c7a4cc75/luns/v_7d13a2d4-818a-4fbe-961d-1cd3c7a4cc75 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:27:14 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-7985a04f-f20c-4e04-8b5b-ceee9484c226 tempest-VolumesActionsTest-707453245 None] failed to remove resource iqn.2010-10.org.openstack:7d13a2d4-818a-4fbe-961d-1cd3c7a4cc75 because of JDSS resource v_7d13a2d4-818a-4fbe-961d-1cd3c7a4cc75 DNE. {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:920}} Apr 17 15:27:14 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-7985a04f-f20c-4e04-8b5b-ceee9484c226 tempest-VolumesActionsTest-707453245 None] delete target: iqn.2010-10.org.openstack:7d13a2d4-818a-4fbe-961d-1cd3c7a4cc75 {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:930}} Apr 17 15:27:14 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-7985a04f-f20c-4e04-8b5b-ceee9484c226 tempest-VolumesActionsTest-707453245 None] delete target iqn.2010-10.org.openstack:7d13a2d4-818a-4fbe-961d-1cd3c7a4cc75 {{(pid=84696) delete_target /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:442}} Apr 17 15:27:14 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-7985a04f-f20c-4e04-8b5b-ceee9484c226 tempest-VolumesActionsTest-707453245 None] Sending pool request DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:7d13a2d4-818a-4fbe-961d-1cd3c7a4cc75 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:27:14 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-7985a04f-f20c-4e04-8b5b-ceee9484c226 tempest-VolumesActionsTest-707453245 None] Sending DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:7d13a2d4-818a-4fbe-961d-1cd3c7a4cc75 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:27:15 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-7985a04f-f20c-4e04-8b5b-ceee9484c226 tempest-VolumesActionsTest-707453245 None] Geting {'code': 404, 'error': {'class': 'opene.exceptions.ItemNotFoundError', 'message': 'Target iqn.2010-10.org.openstack:7d13a2d4-818a-4fbe-961d-1cd3c7a4cc75 not exists.', 'url': 'http://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:7d13a2d4-818a-4fbe-961d-1cd3c7a4cc75'}, 'data': None} from DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:7d13a2d4-818a-4fbe-961d-1cd3c7a4cc75 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:27:15 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-7985a04f-f20c-4e04-8b5b-ceee9484c226 tempest-VolumesActionsTest-707453245 None] failed to remove resource iqn.2010-10.org.openstack:7d13a2d4-818a-4fbe-961d-1cd3c7a4cc75 because of JDSS resource iqn.2010-10.org.openstack:7d13a2d4-818a-4fbe-961d-1cd3c7a4cc75 DNE. {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:935}} Apr 17 15:27:15 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-7985a04f-f20c-4e04-8b5b-ceee9484c226 tempest-VolumesActionsTest-707453245 None] deleating volume v_7d13a2d4-818a-4fbe-961d-1cd3c7a4cc75 {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 15:27:15 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-7985a04f-f20c-4e04-8b5b-ceee9484c226 tempest-VolumesActionsTest-707453245 None] get snapshots for volume v_7d13a2d4-818a-4fbe-961d-1cd3c7a4cc75 {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 15:27:15 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-7985a04f-f20c-4e04-8b5b-ceee9484c226 tempest-VolumesActionsTest-707453245 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_7d13a2d4-818a-4fbe-961d-1cd3c7a4cc75/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:27:15 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-7985a04f-f20c-4e04-8b5b-ceee9484c226 tempest-VolumesActionsTest-707453245 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_7d13a2d4-818a-4fbe-961d-1cd3c7a4cc75/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:27:15 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-1204bfec-a970-49c4-83e9-d36f3df48c1f tempest-VolumesImageMetadata-1836508373 None] Volume reschedule parameters: True retry: {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 15:27:15 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-1204bfec-a970-49c4-83e9-d36f3df48c1f tempest-VolumesImageMetadata-1836508373 None] Flow 'volume_create_manager' (9ceb59b0-e814-48b5-b197-92429bc0c943) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:27:15 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-1204bfec-a970-49c4-83e9-d36f3df48c1f tempest-VolumesImageMetadata-1836508373 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (66f20464-16ef-4580-8c48-17297ee26b9b) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:27:15 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-1204bfec-a970-49c4-83e9-d36f3df48c1f tempest-VolumesImageMetadata-1836508373 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (66f20464-16ef-4580-8c48-17297ee26b9b) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:27:15Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesImageMetadata-Volume-274804235',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=11b50ce4-5220-4297-91e0-e1c6dae2f734,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='ca45c8abc2554ccf8f5a4d7cd5393011',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:27:15Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:27:15Z,use_quota=True,user_id='1be02a47c4774ab5bd8f29eb63a5df06',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:27:15 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-1204bfec-a970-49c4-83e9-d36f3df48c1f tempest-VolumesImageMetadata-1836508373 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (631c9e3b-5c02-4909-8588-fe3c903e548d) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:27:15 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-1204bfec-a970-49c4-83e9-d36f3df48c1f tempest-VolumesImageMetadata-1836508373 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (631c9e3b-5c02-4909-8588-fe3c903e548d) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:27:15 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-1204bfec-a970-49c4-83e9-d36f3df48c1f tempest-VolumesImageMetadata-1836508373 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (27243e2d-c17a-4ff0-b63c-64d0722c6ed2) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:27:15 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-1204bfec-a970-49c4-83e9-d36f3df48c1f tempest-VolumesImageMetadata-1836508373 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (27243e2d-c17a-4ff0-b63c-64d0722c6ed2) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'image', 'volume_id': '11b50ce4-5220-4297-91e0-e1c6dae2f734', 'volume_name': 'volume-11b50ce4-5220-4297-91e0-e1c6dae2f734', 'volume_size': 1, 'image_id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'image_location': (None, None), 'image_meta': {'name': 'cirros-0.5.2-x86_64-disk', 'disk_format': 'qcow2', 'container_format': 'bare', 'visibility': 'public', 'size': 16300544, 'virtual_size': 117440512, 'status': 'active', 'checksum': 'b874c39491a2377b8490f5f1e89761a4', 'protected': False, 'min_ram': 0, 'min_disk': 0, 'owner': 'a43c211b4ed542538799349489b1d884', 'os_hidden': False, 'os_hash_algo': 'sha512', 'os_hash_value': '6b813aa46bb90b4da216a4d19376593fa3f4fc7e617f03a92b7fe11e9a3981cbe8f0959dbebe36225e5f53dc4492341a4863cac4ed1ee0909f3fc78ef9c3e869', 'id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'created_at': datetime.datetime(2023, 4, 17, 15, 19, 53, tzinfo=datetime.timezone.utc), 'updated_at': datetime.datetime(2023, 4, 17, 15, 19, 55, tzinfo=datetime.timezone.utc), 'tags': [], 'file': '/v2/images/b3f0dec1-7758-44a8-82e0-76ada8837428/file', 'properties': {'hw_rng_model': 'virtio', 'owner_specified.openstack.md5': '', 'owner_specified.openstack.object': 'images/cirros-0.5.2-x86_64-disk', 'owner_specified.openstack.sha256': ''}}, 'image_service': }' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:27:15 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-1204bfec-a970-49c4-83e9-d36f3df48c1f tempest-VolumesImageMetadata-1836508373 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (58913ff1-376f-4d35-9ae3-11e7eda49139) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:27:15 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-1204bfec-a970-49c4-83e9-d36f3df48c1f tempest-VolumesImageMetadata-1836508373 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (58913ff1-376f-4d35-9ae3-11e7eda49139) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:27:15 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-1204bfec-a970-49c4-83e9-d36f3df48c1f tempest-VolumesImageMetadata-1836508373 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (07b156b8-d980-4a46-a14e-14b7285cf1ee) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:27:15 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-1204bfec-a970-49c4-83e9-d36f3df48c1f tempest-VolumesImageMetadata-1836508373 None] Volume 11b50ce4-5220-4297-91e0-e1c6dae2f734: being created as image with specification: {'status': 'creating', 'volume_name': 'volume-11b50ce4-5220-4297-91e0-e1c6dae2f734', 'volume_size': 1, 'image_id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'image_location': (None, None), 'image_meta': {'name': 'cirros-0.5.2-x86_64-disk', 'disk_format': 'qcow2', 'container_format': 'bare', 'visibility': 'public', 'size': 16300544, 'virtual_size': 117440512, 'status': 'active', 'checksum': 'b874c39491a2377b8490f5f1e89761a4', 'protected': False, 'min_ram': 0, 'min_disk': 0, 'owner': 'a43c211b4ed542538799349489b1d884', 'os_hidden': False, 'os_hash_algo': 'sha512', 'os_hash_value': '6b813aa46bb90b4da216a4d19376593fa3f4fc7e617f03a92b7fe11e9a3981cbe8f0959dbebe36225e5f53dc4492341a4863cac4ed1ee0909f3fc78ef9c3e869', 'id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'created_at': datetime.datetime(2023, 4, 17, 15, 19, 53, tzinfo=datetime.timezone.utc), 'updated_at': datetime.datetime(2023, 4, 17, 15, 19, 55, tzinfo=datetime.timezone.utc), 'tags': [], 'file': '/v2/images/b3f0dec1-7758-44a8-82e0-76ada8837428/file', 'properties': {'hw_rng_model': 'virtio', 'owner_specified.openstack.md5': '', 'owner_specified.openstack.object': 'images/cirros-0.5.2-x86_64-disk', 'owner_specified.openstack.sha256': ''}}, 'image_service': } Apr 17 15:27:15 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-1204bfec-a970-49c4-83e9-d36f3df48c1f tempest-VolumesImageMetadata-1836508373 None] Cloning 11b50ce4-5220-4297-91e0-e1c6dae2f734 from image b3f0dec1-7758-44a8-82e0-76ada8837428 at location (None, None). {{(pid=84696) _create_from_image /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1071}} Apr 17 15:27:15 user cinder-volume[84696]: DEBUG cinder.coordination [None req-1204bfec-a970-49c4-83e9-d36f3df48c1f tempest-VolumesImageMetadata-1836508373 None] Acquiring lock "cinder-b3f0dec1-7758-44a8-82e0-76ada8837428" by "_prepare_image_cache_entry" {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:192}} Apr 17 15:27:15 user cinder-volume[84696]: DEBUG cinder.coordination [None req-1204bfec-a970-49c4-83e9-d36f3df48c1f tempest-VolumesImageMetadata-1836508373 None] Lock "cinder-b3f0dec1-7758-44a8-82e0-76ada8837428" acquired by "_prepare_image_cache_entry" :: waited 0.008s {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:196}} Apr 17 15:27:15 user cinder-volume[84696]: DEBUG cinder.image.cache [None req-1204bfec-a970-49c4-83e9-d36f3df48c1f tempest-VolumesImageMetadata-1836508373 None] Found image-volume cache entry: {'id': 1, 'image_id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'volume_id': '006f20ae-5306-44fe-afdf-e433f79284e8', 'host': 'user@joviandss-2#joviandss-2', 'size': 1, 'image_updated_at': datetime.datetime(2023, 4, 17, 15, 19, 55), 'last_used': datetime.datetime(2023, 4, 17, 15, 27, 15, 255638)}. {{(pid=84696) get_entry /opt/stack/cinder/cinder/image/cache.py:76}} Apr 17 15:27:15 user cinder-volume[84696]: DEBUG cinder.image.cache [None req-1204bfec-a970-49c4-83e9-d36f3df48c1f tempest-VolumesImageMetadata-1836508373 None] Image-volume cache entry image_update_at = 2023-04-17 15:19:55+00:00, requested image updated_at = 2023-04-17 15:19:55+00:00. {{(pid=84696) _should_update_entry /opt/stack/cinder/cinder/image/cache.py:259}} Apr 17 15:27:15 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-1204bfec-a970-49c4-83e9-d36f3df48c1f tempest-VolumesImageMetadata-1836508373 None] Found cache entry for image = b3f0dec1-7758-44a8-82e0-76ada8837428 on host user@joviandss-2#joviandss-2. {{(pid=84696) _prepare_image_cache_entry /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:886}} Apr 17 15:27:15 user cinder-volume[84696]: DEBUG cinder.coordination [None req-1204bfec-a970-49c4-83e9-d36f3df48c1f tempest-VolumesImageMetadata-1836508373 None] Lock "cinder-b3f0dec1-7758-44a8-82e0-76ada8837428" released by "_prepare_image_cache_entry" :: held 0.015s {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:207}} Apr 17 15:27:15 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-1204bfec-a970-49c4-83e9-d36f3df48c1f tempest-VolumesImageMetadata-1836508373 None] Attempting to retrieve cache entry for image = b3f0dec1-7758-44a8-82e0-76ada8837428 on host user@joviandss-2#joviandss-2. {{(pid=84696) _create_from_image_cache /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:827}} Apr 17 15:27:15 user cinder-volume[84696]: DEBUG cinder.image.cache [None req-1204bfec-a970-49c4-83e9-d36f3df48c1f tempest-VolumesImageMetadata-1836508373 None] Found image-volume cache entry: {'id': 1, 'image_id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'volume_id': '006f20ae-5306-44fe-afdf-e433f79284e8', 'host': 'user@joviandss-2#joviandss-2', 'size': 1, 'image_updated_at': datetime.datetime(2023, 4, 17, 15, 19, 55), 'last_used': datetime.datetime(2023, 4, 17, 15, 27, 15, 269743)}. {{(pid=84696) get_entry /opt/stack/cinder/cinder/image/cache.py:76}} Apr 17 15:27:15 user cinder-volume[84696]: DEBUG cinder.image.cache [None req-1204bfec-a970-49c4-83e9-d36f3df48c1f tempest-VolumesImageMetadata-1836508373 None] Image-volume cache entry image_update_at = 2023-04-17 15:19:55+00:00, requested image updated_at = 2023-04-17 15:19:55+00:00. {{(pid=84696) _should_update_entry /opt/stack/cinder/cinder/image/cache.py:259}} Apr 17 15:27:15 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-1204bfec-a970-49c4-83e9-d36f3df48c1f tempest-VolumesImageMetadata-1836508373 None] Creating from source image-volume 006f20ae-5306-44fe-afdf-e433f79284e8 {{(pid=84696) _create_from_image_cache /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:841}} Apr 17 15:27:15 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-1204bfec-a970-49c4-83e9-d36f3df48c1f tempest-VolumesImageMetadata-1836508373 None] cloned volume 006f20ae-5306-44fe-afdf-e433f79284e8 to 11b50ce4-5220-4297-91e0-e1c6dae2f734 {{(pid=84696) create_cloned_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:521}} Apr 17 15:27:15 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-1204bfec-a970-49c4-83e9-d36f3df48c1f tempest-VolumesImageMetadata-1836508373 None] cloning v_006f20ae-5306-44fe-afdf-e433f79284e8 to v_11b50ce4-5220-4297-91e0-e1c6dae2f734 {{(pid=84696) _clone_object /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:456}} Apr 17 15:27:15 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-1204bfec-a970-49c4-83e9-d36f3df48c1f tempest-VolumesImageMetadata-1836508373 None] create snapshot v_11b50ce4-5220-4297-91e0-e1c6dae2f734 {{(pid=84696) create_snapshot /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:635}} Apr 17 15:27:15 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-1204bfec-a970-49c4-83e9-d36f3df48c1f tempest-VolumesImageMetadata-1836508373 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:27:15 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-1204bfec-a970-49c4-83e9-d36f3df48c1f tempest-VolumesImageMetadata-1836508373 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:27:15 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-7985a04f-f20c-4e04-8b5b-ceee9484c226 tempest-VolumesActionsTest-707453245 None] Geting {'code': 200, 'error': None, 'data': {'results': 0, 'entries': []}} from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_7d13a2d4-818a-4fbe-961d-1cd3c7a4cc75/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:27:15 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-7985a04f-f20c-4e04-8b5b-ceee9484c226 tempest-VolumesActionsTest-707453245 None] get volume v_7d13a2d4-818a-4fbe-961d-1cd3c7a4cc75 info {{(pid=84696) get_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:239}} Apr 17 15:27:15 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-7985a04f-f20c-4e04-8b5b-ceee9484c226 tempest-VolumesActionsTest-707453245 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_7d13a2d4-818a-4fbe-961d-1cd3c7a4cc75 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:27:15 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-7985a04f-f20c-4e04-8b5b-ceee9484c226 tempest-VolumesActionsTest-707453245 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_7d13a2d4-818a-4fbe-961d-1cd3c7a4cc75 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:27:15 user cinder-volume[84696]: DEBUG oslo_service.periodic_task [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Running periodic task VolumeManager.publish_service_capabilities {{(pid=84696) run_periodic_tasks /usr/local/lib/python3.10/dist-packages/oslo_service/periodic_task.py:210}} Apr 17 15:27:15 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Updating volume stats {{(pid=84696) _update_volume_stats /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:724}} Apr 17 15:27:15 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Get pool Pool-0 fsprops {{(pid=84696) get_pool_stats /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:894}} Apr 17 15:27:15 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:27:15 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:27:16 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Geting {'code': 200, 'error': None, 'data': {'available': '113889214464', 'status': 30, 'import_status': {'import_steps': {'san_setup': True, 'vip_setup': True, 'zfs_import': True, 'mount_system_volume': True, 'nas_setup': True}, 'import_successful': True}, 'scan': None, 'encryption': {'enabled': False}, 'deduplication_rate': '1.00', 'sysvol_upgrade': 'not_needed', 'vdevs': [{'name': 'wwn-0x6000c296b793bf526a107a9d823e947a', 'iostats': {'read': '0', 'write': '0', 'chksum': '0'}, 'disks': [{'origin': 'local', 'slot': '', 'led': 'off', 'name': 'sdn', 'iostats': {'read': '0', 'write': '0', 'chksum': '0'}, 'alias': '', 'health': 'ONLINE', 'sn': '6000c296b793bf526a107a9d823e947a', 'trim_data': {'status': 'trim_unsupported', 'progress': None, 'trim_time': None}, 'path': 'pci-0000:03:00.0-scsi-0:0:14:0', 'model': 'Virtual disk', 'id': 'wwn-0x6000c296b793bf526a107a9d823e947a', 'size': 128849018880}], 'health': 'ONLINE', 'vdev_replacings': [], 'vdev_spares': [], 'type': ''}], 'id': '14181317334440137330', 'health': 'ONLINE', 'iostats': {'read': '0', 'write': '0', 'chksum': '0'}, 'operation': 'none', 'size': '127775277056', 'autotrim': False, 'name': 'Pool-0'}} from GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:27:16 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Total capacity: 119, Free 106. {{(pid=84696) _update_volume_stats /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:757}} Apr 17 15:27:16 user cinder-volume[84696]: DEBUG cinder.manager [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Notifying Schedulers of capabilities ... {{(pid=84696) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 17 15:27:17 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-1204bfec-a970-49c4-83e9-d36f3df48c1f tempest-VolumesImageMetadata-1836508373 None] Geting {'code': 200, 'error': None, 'data': {'name': 'v_11b50ce4-5220-4297-91e0-e1c6dae2f734'}} from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:27:17 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-1204bfec-a970-49c4-83e9-d36f3df48c1f tempest-VolumesImageMetadata-1836508373 None] create volume v_11b50ce4-5220-4297-91e0-e1c6dae2f734 from snapshot v_11b50ce4-5220-4297-91e0-e1c6dae2f734 {{(pid=84696) create_volume_from_snapshot /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:674}} Apr 17 15:27:17 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-1204bfec-a970-49c4-83e9-d36f3df48c1f tempest-VolumesImageMetadata-1836508373 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/clone {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:27:17 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-1204bfec-a970-49c4-83e9-d36f3df48c1f tempest-VolumesImageMetadata-1836508373 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/clone {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:27:17 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-7985a04f-f20c-4e04-8b5b-ceee9484c226 tempest-VolumesActionsTest-707453245 None] Geting {'code': 200, 'error': None, 'data': {'origin': None, 'relatime': None, 'acltype': None, 'vscan': None, 'full_name': 'Pool-0/v_7d13a2d4-818a-4fbe-961d-1cd3c7a4cc75', 'userrefs': None, 'primarycache': 'all', 'logbias': 'latency', 'creation': '1681746625', 'sync': 'always', 'is_clone': False, 'dedup': 'off', 'sharenfs': None, 'receive_resume_token': None, 'volsize': '1073741824', 'referenced': '57344', 'sharesmb': None, 'createtxg': '188135', 'reservation': '0', 'scontext': None, 'mountpoint': None, 'casesensitivity': None, 'guid': '12560871980629688112', 'usedbyrefreservation': '0', 'dnodesize': None, 'written': '57344', 'logicalused': '28672', 'compressratio': '1.00', 'rootcontext': 'none', 'default_scsi_id': '91b8bc3617476894', 'type': 'volume', 'compression': 'lz4', 'snapdir': None, 'overlay': None, 'encryption': 'off', 'xattr': None, 'volmode': 'default', 'copies': '1', 'snapshot_limit': '18446744073709551615', 'aclinherit': None, 'defcontext': 'none', 'readonly': 'off', 'version': None, 'recordsize': None, 'filesystem_limit': None, 'mounted': None, 'mlslabel': 'none', 'secondarycache': 'all', 'refreservation': '0', 'available': '113889202176', 'san:volume_id': '91b8bc3617476894c3a85848d83510ef7d3aa2dc', 'encryptionroot': None, 'exec': None, 'refquota': None, 'refcompressratio': '1.00', 'quota': None, 'utf8only': None, 'keylocation': 'none', 'snapdev': 'hidden', 'snapshot_count': '18446744073709551615', 'fscontext': 'none', 'clones': None, 'canmount': None, 'keystatus': None, 'atime': None, 'usedbysnapshots': '0', 'normalization': None, 'usedbychildren': '0', 'volblocksize': '131072', 'usedbydataset': '57344', 'objsetid': '122710', 'name': 'v_7d13a2d4-818a-4fbe-961d-1cd3c7a4cc75', 'defer_destroy': None, 'pbkdf2iters': '0', 'checksum': 'on', 'redundant_metadata': 'all', 'filesystem_count': None, 'devices': None, 'keyformat': 'none', 'setuid': None, 'used': '57344', 'logicalreferenced': '28672', 'context': 'none', 'zoned': None, 'nbmand': None}} from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_7d13a2d4-818a-4fbe-961d-1cd3c7a4cc75 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:27:17 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-7985a04f-f20c-4e04-8b5b-ceee9484c226 tempest-VolumesActionsTest-707453245 None] delete volume:v_7d13a2d4-818a-4fbe-961d-1cd3c7a4cc75 recursively children:{'force_umount': True} {{(pid=84696) delete_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:335}} Apr 17 15:27:17 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-7985a04f-f20c-4e04-8b5b-ceee9484c226 tempest-VolumesActionsTest-707453245 None] Sending pool request DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_7d13a2d4-818a-4fbe-961d-1cd3c7a4cc75 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:27:17 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-7985a04f-f20c-4e04-8b5b-ceee9484c226 tempest-VolumesActionsTest-707453245 None] Sending DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_7d13a2d4-818a-4fbe-961d-1cd3c7a4cc75 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:27:18 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-1204bfec-a970-49c4-83e9-d36f3df48c1f tempest-VolumesImageMetadata-1836508373 None] Geting {'code': 200, 'error': None, 'data': {'origin': 'Pool-0/v_006f20ae-5306-44fe-afdf-e433f79284e8@v_11b50ce4-5220-4297-91e0-e1c6dae2f734', 'is_clone': True, 'full_name': 'Pool-0/v_11b50ce4-5220-4297-91e0-e1c6dae2f734', 'name': 'v_11b50ce4-5220-4297-91e0-e1c6dae2f734'}} from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/clone {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:27:18 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-1204bfec-a970-49c4-83e9-d36f3df48c1f tempest-VolumesImageMetadata-1836508373 None] get volume v_11b50ce4-5220-4297-91e0-e1c6dae2f734 info {{(pid=84696) get_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:239}} Apr 17 15:27:18 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-1204bfec-a970-49c4-83e9-d36f3df48c1f tempest-VolumesImageMetadata-1836508373 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_11b50ce4-5220-4297-91e0-e1c6dae2f734 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:27:18 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-1204bfec-a970-49c4-83e9-d36f3df48c1f tempest-VolumesImageMetadata-1836508373 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_11b50ce4-5220-4297-91e0-e1c6dae2f734 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:27:20 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-7985a04f-f20c-4e04-8b5b-ceee9484c226 tempest-VolumesActionsTest-707453245 None] Geting {'code': 204} from DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_7d13a2d4-818a-4fbe-961d-1cd3c7a4cc75 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:27:20 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-7985a04f-f20c-4e04-8b5b-ceee9484c226 tempest-VolumesActionsTest-707453245 None] volume v_7d13a2d4-818a-4fbe-961d-1cd3c7a4cc75 deleted {{(pid=84696) delete_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:346}} Apr 17 15:27:20 user cinder-volume[84696]: DEBUG cinder.quota [None req-7985a04f-f20c-4e04-8b5b-ceee9484c226 tempest-VolumesActionsTest-707453245 None] Created reservations ['45da0371-80a6-4458-b138-8d360a730f2c', '63a03b52-517d-4e89-b682-b803f327c770', '98925794-c0e4-424c-a51c-75287f02e267', 'd16cd8ff-b310-474a-b346-3b947635ee26'] {{(pid=84696) reserve /opt/stack/cinder/cinder/quota.py:755}} Apr 17 15:27:20 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=66,cinder:INSERT=7,cinder:UPDATE=27 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:27:20 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-1204bfec-a970-49c4-83e9-d36f3df48c1f tempest-VolumesImageMetadata-1836508373 None] Geting {'code': 200, 'error': None, 'data': {'origin': 'Pool-0/v_006f20ae-5306-44fe-afdf-e433f79284e8@v_11b50ce4-5220-4297-91e0-e1c6dae2f734', 'relatime': None, 'acltype': None, 'vscan': None, 'full_name': 'Pool-0/v_11b50ce4-5220-4297-91e0-e1c6dae2f734', 'userrefs': None, 'primarycache': 'all', 'logbias': 'latency', 'creation': '1681746630', 'sync': 'always', 'is_clone': True, 'dedup': 'off', 'sharenfs': None, 'receive_resume_token': None, 'volsize': '1073741824', 'referenced': '16285696', 'sharesmb': None, 'createtxg': '188141', 'reservation': '0', 'scontext': None, 'mountpoint': None, 'casesensitivity': None, 'guid': '7603708577091853313', 'usedbyrefreservation': '0', 'dnodesize': None, 'written': '0', 'logicalused': '0', 'compressratio': '1.00', 'rootcontext': 'none', 'default_scsi_id': '0f47559fd62af371', 'type': 'volume', 'compression': 'lz4', 'snapdir': None, 'overlay': None, 'encryption': 'off', 'xattr': None, 'volmode': 'default', 'copies': '1', 'snapshot_limit': '18446744073709551615', 'aclinherit': None, 'defcontext': 'none', 'readonly': 'off', 'version': None, 'recordsize': None, 'filesystem_limit': None, 'mounted': None, 'mlslabel': 'none', 'secondarycache': 'all', 'refreservation': '0', 'available': '113889259520', 'used': '0', 'encryptionroot': None, 'exec': None, 'refquota': None, 'refcompressratio': '1.17', 'quota': None, 'utf8only': None, 'keylocation': 'none', 'snapdev': 'hidden', 'snapshot_count': '18446744073709551615', 'fscontext': 'none', 'clones': None, 'canmount': None, 'keystatus': None, 'atime': None, 'usedbysnapshots': '0', 'normalization': None, 'usedbychildren': '0', 'volblocksize': '131072', 'usedbydataset': '0', 'objsetid': '122914', 'name': 'v_11b50ce4-5220-4297-91e0-e1c6dae2f734', 'defer_destroy': None, 'pbkdf2iters': '0', 'checksum': 'on', 'redundant_metadata': 'all', 'filesystem_count': None, 'devices': None, 'keyformat': 'none', 'setuid': None, 'logicalreferenced': '19042304', 'context': 'none', 'zoned': None, 'nbmand': None}} from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_11b50ce4-5220-4297-91e0-e1c6dae2f734 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:27:20 user cinder-volume[84696]: DEBUG cinder.volume.volume_utils [None req-1204bfec-a970-49c4-83e9-d36f3df48c1f tempest-VolumesImageMetadata-1836508373 None] Marking volume 11b50ce4-5220-4297-91e0-e1c6dae2f734 as bootable. {{(pid=84696) enable_bootable_flag /opt/stack/cinder/cinder/volume/volume_utils.py:1140}} Apr 17 15:27:20 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-1204bfec-a970-49c4-83e9-d36f3df48c1f tempest-VolumesImageMetadata-1836508373 None] Copying metadata from image b3f0dec1-7758-44a8-82e0-76ada8837428 to 11b50ce4-5220-4297-91e0-e1c6dae2f734. {{(pid=84696) _handle_bootable_volume_glance_meta /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:452}} Apr 17 15:27:20 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-1204bfec-a970-49c4-83e9-d36f3df48c1f tempest-VolumesImageMetadata-1836508373 None] Creating volume glance metadata for volume 11b50ce4-5220-4297-91e0-e1c6dae2f734 backed by image b3f0dec1-7758-44a8-82e0-76ada8837428 with: {'hw_rng_model': 'virtio', 'owner_specified.openstack.md5': '', 'owner_specified.openstack.object': 'images/cirros-0.5.2-x86_64-disk', 'owner_specified.openstack.sha256': '', 'image_id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'image_name': 'cirros-0.5.2-x86_64-disk', 'checksum': 'b874c39491a2377b8490f5f1e89761a4', 'container_format': 'bare', 'disk_format': 'qcow2', 'min_disk': 0, 'min_ram': 0, 'size': 16300544}. {{(pid=84696) _capture_volume_image_metadata /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:683}} Apr 17 15:27:20 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=66,cinder:INSERT=8,cinder:UPDATE=25,cinder:DELETE=1 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:27:20 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-7985a04f-f20c-4e04-8b5b-ceee9484c226 tempest-VolumesActionsTest-707453245 None] Updating volume stats {{(pid=84696) _update_volume_stats /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:724}} Apr 17 15:27:20 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-7985a04f-f20c-4e04-8b5b-ceee9484c226 tempest-VolumesActionsTest-707453245 None] Get pool Pool-0 fsprops {{(pid=84696) get_pool_stats /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:894}} Apr 17 15:27:20 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-7985a04f-f20c-4e04-8b5b-ceee9484c226 tempest-VolumesActionsTest-707453245 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:27:20 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-7985a04f-f20c-4e04-8b5b-ceee9484c226 tempest-VolumesActionsTest-707453245 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:27:20 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-1204bfec-a970-49c4-83e9-d36f3df48c1f tempest-VolumesImageMetadata-1836508373 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (07b156b8-d980-4a46-a14e-14b7285cf1ee) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'volume_name': 'volume-11b50ce4-5220-4297-91e0-e1c6dae2f734', 'volume_size': 1, 'image_id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'image_location': (None, None), 'image_meta': {'name': 'cirros-0.5.2-x86_64-disk', 'disk_format': 'qcow2', 'container_format': 'bare', 'visibility': 'public', 'size': 16300544, 'virtual_size': 117440512, 'status': 'active', 'checksum': 'b874c39491a2377b8490f5f1e89761a4', 'protected': False, 'min_ram': 0, 'min_disk': 0, 'owner': 'a43c211b4ed542538799349489b1d884', 'os_hidden': False, 'os_hash_algo': 'sha512', 'os_hash_value': '6b813aa46bb90b4da216a4d19376593fa3f4fc7e617f03a92b7fe11e9a3981cbe8f0959dbebe36225e5f53dc4492341a4863cac4ed1ee0909f3fc78ef9c3e869', 'id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'created_at': datetime.datetime(2023, 4, 17, 15, 19, 53, tzinfo=datetime.timezone.utc), 'updated_at': datetime.datetime(2023, 4, 17, 15, 19, 55, tzinfo=datetime.timezone.utc), 'tags': [], 'file': '/v2/images/b3f0dec1-7758-44a8-82e0-76ada8837428/file', 'properties': {'hw_rng_model': 'virtio', 'owner_specified.openstack.md5': '', 'owner_specified.openstack.object': 'images/cirros-0.5.2-x86_64-disk', 'owner_specified.openstack.sha256': ''}}, 'image_service': }' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:27:20 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-1204bfec-a970-49c4-83e9-d36f3df48c1f tempest-VolumesImageMetadata-1836508373 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeOnFinishTask;volume:create, create.end' (19a2f3e9-9047-4766-805d-77fa1dfe5f4f) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:27:20 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-1204bfec-a970-49c4-83e9-d36f3df48c1f tempest-VolumesImageMetadata-1836508373 None] Volume volume-11b50ce4-5220-4297-91e0-e1c6dae2f734 (11b50ce4-5220-4297-91e0-e1c6dae2f734): created successfully Apr 17 15:27:20 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-1204bfec-a970-49c4-83e9-d36f3df48c1f tempest-VolumesImageMetadata-1836508373 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeOnFinishTask;volume:create, create.end' (19a2f3e9-9047-4766-805d-77fa1dfe5f4f) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:27:20 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-1204bfec-a970-49c4-83e9-d36f3df48c1f tempest-VolumesImageMetadata-1836508373 None] Flow 'volume_create_manager' (9ceb59b0-e814-48b5-b197-92429bc0c943) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:27:20 user cinder-volume[84696]: INFO cinder.volume.manager [None req-1204bfec-a970-49c4-83e9-d36f3df48c1f tempest-VolumesImageMetadata-1836508373 None] Created volume successfully. Apr 17 15:27:21 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-7985a04f-f20c-4e04-8b5b-ceee9484c226 tempest-VolumesActionsTest-707453245 None] Geting {'code': 200, 'error': None, 'data': {'available': '113889247232', 'status': 30, 'import_status': {'import_steps': {'san_setup': True, 'vip_setup': True, 'zfs_import': True, 'mount_system_volume': True, 'nas_setup': True}, 'import_successful': True}, 'scan': None, 'encryption': {'enabled': False}, 'deduplication_rate': '1.00', 'sysvol_upgrade': 'not_needed', 'vdevs': [{'name': 'wwn-0x6000c296b793bf526a107a9d823e947a', 'iostats': {'read': '0', 'write': '0', 'chksum': '0'}, 'disks': [{'origin': 'local', 'slot': '', 'led': 'off', 'name': 'sdn', 'iostats': {'read': '0', 'write': '0', 'chksum': '0'}, 'alias': '', 'health': 'ONLINE', 'sn': '6000c296b793bf526a107a9d823e947a', 'trim_data': {'status': 'trim_unsupported', 'progress': None, 'trim_time': None}, 'path': 'pci-0000:03:00.0-scsi-0:0:14:0', 'model': 'Virtual disk', 'id': 'wwn-0x6000c296b793bf526a107a9d823e947a', 'size': 128849018880}], 'health': 'ONLINE', 'vdev_replacings': [], 'vdev_spares': [], 'type': ''}], 'id': '14181317334440137330', 'health': 'ONLINE', 'iostats': {'read': '0', 'write': '0', 'chksum': '0'}, 'operation': 'none', 'size': '127775277056', 'autotrim': False, 'name': 'Pool-0'}} from GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:27:21 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-7985a04f-f20c-4e04-8b5b-ceee9484c226 tempest-VolumesActionsTest-707453245 None] Total capacity: 119, Free 106. {{(pid=84696) _update_volume_stats /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:757}} Apr 17 15:27:21 user cinder-volume[84696]: DEBUG cinder.manager [None req-7985a04f-f20c-4e04-8b5b-ceee9484c226 tempest-VolumesActionsTest-707453245 None] Notifying Schedulers of capabilities ... {{(pid=84696) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 17 15:27:21 user cinder-volume[84696]: INFO cinder.volume.manager [None req-7985a04f-f20c-4e04-8b5b-ceee9484c226 tempest-VolumesActionsTest-707453245 None] Deleted volume successfully. Apr 17 15:27:21 user cinder-volume[84696]: DEBUG cinder.coordination [None req-7985a04f-f20c-4e04-8b5b-ceee9484c226 tempest-VolumesActionsTest-707453245 None] Lock "cinder-7d13a2d4-818a-4fbe-961d-1cd3c7a4cc75-delete_volume" released by "delete_volume" :: held 6.685s {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:207}} Apr 17 15:27:21 user cinder-volume[84696]: DEBUG cinder.coordination [None req-abb58d60-b907-4eab-8061-b333e35be167 tempest-VolumesImageMetadata-1836508373 None] Acquiring lock "cinder-11b50ce4-5220-4297-91e0-e1c6dae2f734-delete_volume" by "delete_volume" {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:192}} Apr 17 15:27:21 user cinder-volume[84696]: DEBUG cinder.coordination [None req-abb58d60-b907-4eab-8061-b333e35be167 tempest-VolumesImageMetadata-1836508373 None] Lock "cinder-11b50ce4-5220-4297-91e0-e1c6dae2f734-delete_volume" acquired by "delete_volume" :: waited 0.009s {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:196}} Apr 17 15:27:22 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-abb58d60-b907-4eab-8061-b333e35be167 tempest-VolumesImageMetadata-1836508373 None] remove_export for volume: 11b50ce4-5220-4297-91e0-e1c6dae2f734. {{(pid=84696) remove_export /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:718}} Apr 17 15:27:22 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-abb58d60-b907-4eab-8061-b333e35be167 tempest-VolumesImageMetadata-1836508373 None] remove export {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:912}} Apr 17 15:27:22 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-abb58d60-b907-4eab-8061-b333e35be167 tempest-VolumesImageMetadata-1836508373 None] detach volume:Volume(_name_id=None,admin_metadata={},attach_status='detached',availability_zone='nova',bootable=True,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:27:15Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesImageMetadata-Volume-274804235',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=11b50ce4-5220-4297-91e0-e1c6dae2f734,launched_at=2023-04-17T15:27:21Z,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='ca45c8abc2554ccf8f5a4d7cd5393011',provider_auth='CHAP UCMoOlKI jnCzAUuaM2gNHi',provider_geometry=None,provider_id=None,provider_location='172.16.0.220:3260,1 iqn.2010-10.org.openstack:11b50ce4-5220-4297-91e0-e1c6dae2f734 0',replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:27:15Z,service_uuid='e628352e-c4cf-40eb-8389-cce40663026a',shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='deleting',terminated_at=2023-04-17T15:27:22Z,updated_at=2023-04-17T15:27:22Z,use_quota=True,user_id='1be02a47c4774ab5bd8f29eb63a5df06',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c) from target:iqn.2010-10.org.openstack:11b50ce4-5220-4297-91e0-e1c6dae2f734. {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:913}} Apr 17 15:27:22 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-abb58d60-b907-4eab-8061-b333e35be167 tempest-VolumesImageMetadata-1836508373 None] detach volume v_11b50ce4-5220-4297-91e0-e1c6dae2f734 from target iqn.2010-10.org.openstack:11b50ce4-5220-4297-91e0-e1c6dae2f734 {{(pid=84696) detach_target_vol /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:606}} Apr 17 15:27:22 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-abb58d60-b907-4eab-8061-b333e35be167 tempest-VolumesImageMetadata-1836508373 None] Sending pool request DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:11b50ce4-5220-4297-91e0-e1c6dae2f734/luns/v_11b50ce4-5220-4297-91e0-e1c6dae2f734 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:27:22 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-abb58d60-b907-4eab-8061-b333e35be167 tempest-VolumesImageMetadata-1836508373 None] Sending DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:11b50ce4-5220-4297-91e0-e1c6dae2f734/luns/v_11b50ce4-5220-4297-91e0-e1c6dae2f734 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:27:22 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-abb58d60-b907-4eab-8061-b333e35be167 tempest-VolumesImageMetadata-1836508373 None] Geting {'code': 404, 'error': {'class': 'opene.exceptions.ItemNotFoundError', 'message': 'Target iqn.2010-10.org.openstack:11b50ce4-5220-4297-91e0-e1c6dae2f734 not exists.', 'url': 'http://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:11b50ce4-5220-4297-91e0-e1c6dae2f734/luns/v_11b50ce4-5220-4297-91e0-e1c6dae2f734'}, 'data': None} from DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:11b50ce4-5220-4297-91e0-e1c6dae2f734/luns/v_11b50ce4-5220-4297-91e0-e1c6dae2f734 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:27:22 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-abb58d60-b907-4eab-8061-b333e35be167 tempest-VolumesImageMetadata-1836508373 None] failed to remove resource iqn.2010-10.org.openstack:11b50ce4-5220-4297-91e0-e1c6dae2f734 because of JDSS resource v_11b50ce4-5220-4297-91e0-e1c6dae2f734 DNE. {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:920}} Apr 17 15:27:22 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-abb58d60-b907-4eab-8061-b333e35be167 tempest-VolumesImageMetadata-1836508373 None] delete target: iqn.2010-10.org.openstack:11b50ce4-5220-4297-91e0-e1c6dae2f734 {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:930}} Apr 17 15:27:22 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-abb58d60-b907-4eab-8061-b333e35be167 tempest-VolumesImageMetadata-1836508373 None] delete target iqn.2010-10.org.openstack:11b50ce4-5220-4297-91e0-e1c6dae2f734 {{(pid=84696) delete_target /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:442}} Apr 17 15:27:22 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-abb58d60-b907-4eab-8061-b333e35be167 tempest-VolumesImageMetadata-1836508373 None] Sending pool request DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:11b50ce4-5220-4297-91e0-e1c6dae2f734 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:27:22 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-abb58d60-b907-4eab-8061-b333e35be167 tempest-VolumesImageMetadata-1836508373 None] Sending DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:11b50ce4-5220-4297-91e0-e1c6dae2f734 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:27:22 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-abb58d60-b907-4eab-8061-b333e35be167 tempest-VolumesImageMetadata-1836508373 None] Geting {'code': 404, 'error': {'class': 'opene.exceptions.ItemNotFoundError', 'message': 'Target iqn.2010-10.org.openstack:11b50ce4-5220-4297-91e0-e1c6dae2f734 not exists.', 'url': 'http://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:11b50ce4-5220-4297-91e0-e1c6dae2f734'}, 'data': None} from DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:11b50ce4-5220-4297-91e0-e1c6dae2f734 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:27:22 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-abb58d60-b907-4eab-8061-b333e35be167 tempest-VolumesImageMetadata-1836508373 None] failed to remove resource iqn.2010-10.org.openstack:11b50ce4-5220-4297-91e0-e1c6dae2f734 because of JDSS resource iqn.2010-10.org.openstack:11b50ce4-5220-4297-91e0-e1c6dae2f734 DNE. {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:935}} Apr 17 15:27:22 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-abb58d60-b907-4eab-8061-b333e35be167 tempest-VolumesImageMetadata-1836508373 None] deleating volume v_11b50ce4-5220-4297-91e0-e1c6dae2f734 {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 15:27:22 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-abb58d60-b907-4eab-8061-b333e35be167 tempest-VolumesImageMetadata-1836508373 None] get snapshots for volume v_11b50ce4-5220-4297-91e0-e1c6dae2f734 {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 15:27:22 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-abb58d60-b907-4eab-8061-b333e35be167 tempest-VolumesImageMetadata-1836508373 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_11b50ce4-5220-4297-91e0-e1c6dae2f734/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:27:22 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-abb58d60-b907-4eab-8061-b333e35be167 tempest-VolumesImageMetadata-1836508373 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_11b50ce4-5220-4297-91e0-e1c6dae2f734/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:27:22 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-abb58d60-b907-4eab-8061-b333e35be167 tempest-VolumesImageMetadata-1836508373 None] Geting {'code': 200, 'error': None, 'data': {'results': 0, 'entries': []}} from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_11b50ce4-5220-4297-91e0-e1c6dae2f734/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:27:22 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-abb58d60-b907-4eab-8061-b333e35be167 tempest-VolumesImageMetadata-1836508373 None] get volume v_11b50ce4-5220-4297-91e0-e1c6dae2f734 info {{(pid=84696) get_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:239}} Apr 17 15:27:22 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-abb58d60-b907-4eab-8061-b333e35be167 tempest-VolumesImageMetadata-1836508373 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_11b50ce4-5220-4297-91e0-e1c6dae2f734 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:27:22 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-abb58d60-b907-4eab-8061-b333e35be167 tempest-VolumesImageMetadata-1836508373 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_11b50ce4-5220-4297-91e0-e1c6dae2f734 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:27:22 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-abb58d60-b907-4eab-8061-b333e35be167 tempest-VolumesImageMetadata-1836508373 None] Geting {'code': 200, 'error': None, 'data': {'origin': 'Pool-0/v_006f20ae-5306-44fe-afdf-e433f79284e8@v_11b50ce4-5220-4297-91e0-e1c6dae2f734', 'relatime': None, 'acltype': None, 'vscan': None, 'full_name': 'Pool-0/v_11b50ce4-5220-4297-91e0-e1c6dae2f734', 'userrefs': None, 'primarycache': 'all', 'logbias': 'latency', 'creation': '1681746630', 'sync': 'always', 'is_clone': True, 'dedup': 'off', 'sharenfs': None, 'receive_resume_token': None, 'volsize': '1073741824', 'referenced': '16285696', 'sharesmb': None, 'createtxg': '188141', 'reservation': '0', 'scontext': None, 'mountpoint': None, 'casesensitivity': None, 'guid': '7603708577091853313', 'usedbyrefreservation': '0', 'dnodesize': None, 'written': '0', 'logicalused': '0', 'compressratio': '1.00', 'rootcontext': 'none', 'default_scsi_id': '0f47559fd62af371', 'type': 'volume', 'compression': 'lz4', 'snapdir': None, 'overlay': None, 'encryption': 'off', 'xattr': None, 'volmode': 'default', 'copies': '1', 'snapshot_limit': '18446744073709551615', 'aclinherit': None, 'defcontext': 'none', 'readonly': 'off', 'version': None, 'recordsize': None, 'filesystem_limit': None, 'mounted': None, 'mlslabel': 'none', 'secondarycache': 'all', 'refreservation': '0', 'available': '113889247232', 'san:volume_id': '0f47559fd62af3713769cb1092ac323ba40767ff', 'encryptionroot': None, 'exec': None, 'refquota': None, 'refcompressratio': '1.17', 'quota': None, 'utf8only': None, 'keylocation': 'none', 'snapdev': 'hidden', 'snapshot_count': '18446744073709551615', 'fscontext': 'none', 'clones': None, 'canmount': None, 'keystatus': None, 'atime': None, 'usedbysnapshots': '0', 'normalization': None, 'usedbychildren': '0', 'volblocksize': '131072', 'usedbydataset': '0', 'objsetid': '122914', 'name': 'v_11b50ce4-5220-4297-91e0-e1c6dae2f734', 'defer_destroy': None, 'pbkdf2iters': '0', 'checksum': 'on', 'redundant_metadata': 'all', 'filesystem_count': None, 'devices': None, 'keyformat': 'none', 'setuid': None, 'used': '0', 'logicalreferenced': '19042304', 'context': 'none', 'zoned': None, 'nbmand': None}} from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_11b50ce4-5220-4297-91e0-e1c6dae2f734 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:27:22 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-abb58d60-b907-4eab-8061-b333e35be167 tempest-VolumesImageMetadata-1836508373 None] delete snapshot v_11b50ce4-5220-4297-91e0-e1c6dae2f734 of volume v_006f20ae-5306-44fe-afdf-e433f79284e8 {{(pid=84696) delete_snapshot /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:751}} Apr 17 15:27:22 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-abb58d60-b907-4eab-8061-b333e35be167 tempest-VolumesImageMetadata-1836508373 None] Sending pool request DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots/v_11b50ce4-5220-4297-91e0-e1c6dae2f734 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:27:22 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-abb58d60-b907-4eab-8061-b333e35be167 tempest-VolumesImageMetadata-1836508373 None] Sending DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots/v_11b50ce4-5220-4297-91e0-e1c6dae2f734 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:27:25 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-abb58d60-b907-4eab-8061-b333e35be167 tempest-VolumesImageMetadata-1836508373 None] Geting {'code': 204} from DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots/v_11b50ce4-5220-4297-91e0-e1c6dae2f734 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:27:25 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-abb58d60-b907-4eab-8061-b333e35be167 tempest-VolumesImageMetadata-1836508373 None] snapshot v_11b50ce4-5220-4297-91e0-e1c6dae2f734 deleted {{(pid=84696) delete_snapshot /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:772}} Apr 17 15:27:25 user cinder-volume[84696]: DEBUG cinder.quota [None req-abb58d60-b907-4eab-8061-b333e35be167 tempest-VolumesImageMetadata-1836508373 None] Created reservations ['d9a6d290-4776-404d-9414-86cc47fbaa3e', '06d6ea0f-e11d-4fed-b6e5-d4564873707e', '4b348179-2465-42d2-bff8-0fb3b02785d4', '3661c95f-74e6-458e-a71d-45e950e579dd'] {{(pid=84696) reserve /opt/stack/cinder/cinder/quota.py:755}} Apr 17 15:27:25 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-abb58d60-b907-4eab-8061-b333e35be167 tempest-VolumesImageMetadata-1836508373 None] Updating volume stats {{(pid=84696) _update_volume_stats /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:724}} Apr 17 15:27:25 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-abb58d60-b907-4eab-8061-b333e35be167 tempest-VolumesImageMetadata-1836508373 None] Get pool Pool-0 fsprops {{(pid=84696) get_pool_stats /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:894}} Apr 17 15:27:25 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-abb58d60-b907-4eab-8061-b333e35be167 tempest-VolumesImageMetadata-1836508373 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:27:25 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-abb58d60-b907-4eab-8061-b333e35be167 tempest-VolumesImageMetadata-1836508373 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:27:25 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-abb58d60-b907-4eab-8061-b333e35be167 tempest-VolumesImageMetadata-1836508373 None] Geting {'code': 200, 'error': None, 'data': {'available': '113889280000', 'status': 30, 'import_status': {'import_steps': {'san_setup': True, 'vip_setup': True, 'zfs_import': True, 'mount_system_volume': True, 'nas_setup': True}, 'import_successful': True}, 'scan': None, 'encryption': {'enabled': False}, 'deduplication_rate': '1.00', 'sysvol_upgrade': 'not_needed', 'vdevs': [{'name': 'wwn-0x6000c296b793bf526a107a9d823e947a', 'iostats': {'read': '0', 'write': '0', 'chksum': '0'}, 'disks': [{'origin': 'local', 'slot': '', 'led': 'off', 'name': 'sdn', 'iostats': {'read': '0', 'write': '0', 'chksum': '0'}, 'alias': '', 'health': 'ONLINE', 'sn': '6000c296b793bf526a107a9d823e947a', 'trim_data': {'status': 'trim_unsupported', 'progress': None, 'trim_time': None}, 'path': 'pci-0000:03:00.0-scsi-0:0:14:0', 'model': 'Virtual disk', 'id': 'wwn-0x6000c296b793bf526a107a9d823e947a', 'size': 128849018880}], 'health': 'ONLINE', 'vdev_replacings': [], 'vdev_spares': [], 'type': ''}], 'id': '14181317334440137330', 'health': 'ONLINE', 'iostats': {'read': '0', 'write': '0', 'chksum': '0'}, 'operation': 'none', 'size': '127775277056', 'autotrim': False, 'name': 'Pool-0'}} from GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:27:25 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-abb58d60-b907-4eab-8061-b333e35be167 tempest-VolumesImageMetadata-1836508373 None] Total capacity: 119, Free 106. {{(pid=84696) _update_volume_stats /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:757}} Apr 17 15:27:25 user cinder-volume[84696]: DEBUG cinder.manager [None req-abb58d60-b907-4eab-8061-b333e35be167 tempest-VolumesImageMetadata-1836508373 None] Notifying Schedulers of capabilities ... {{(pid=84696) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 17 15:27:26 user cinder-volume[84696]: INFO cinder.volume.manager [None req-abb58d60-b907-4eab-8061-b333e35be167 tempest-VolumesImageMetadata-1836508373 None] Deleted volume successfully. Apr 17 15:27:26 user cinder-volume[84696]: DEBUG cinder.coordination [None req-abb58d60-b907-4eab-8061-b333e35be167 tempest-VolumesImageMetadata-1836508373 None] Lock "cinder-11b50ce4-5220-4297-91e0-e1c6dae2f734-delete_volume" released by "delete_volume" :: held 4.054s {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:207}} Apr 17 15:27:27 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-1f4b549a-26d0-4230-a266-eaf4b89d5f6c req-52fa7a0d-376e-4848-b93c-0843e364cab6 tempest-TestShelveInstance-1683612675 None] terminate connection for d895d11d-6880-4418-a9ad-feca813ed5d6 {{(pid=84696) terminate_connection /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:1032}} Apr 17 15:27:27 user cinder-volume[84696]: INFO cinder.volume.manager [req-1f4b549a-26d0-4230-a266-eaf4b89d5f6c req-52fa7a0d-376e-4848-b93c-0843e364cab6 tempest-TestShelveInstance-1683612675 None] Terminate volume connection completed successfully. Apr 17 15:27:27 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-1f4b549a-26d0-4230-a266-eaf4b89d5f6c req-52fa7a0d-376e-4848-b93c-0843e364cab6 tempest-TestShelveInstance-1683612675 None] Deleting attachment c0eb6e83-df81-4c2b-920c-8280d74fcde6. {{(pid=84696) attachment_delete /opt/stack/cinder/cinder/volume/manager.py:5006}} Apr 17 15:27:27 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-1f4b549a-26d0-4230-a266-eaf4b89d5f6c req-52fa7a0d-376e-4848-b93c-0843e364cab6 tempest-TestShelveInstance-1683612675 None] remove_export for volume: d895d11d-6880-4418-a9ad-feca813ed5d6. {{(pid=84696) remove_export /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:718}} Apr 17 15:27:27 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-1f4b549a-26d0-4230-a266-eaf4b89d5f6c req-52fa7a0d-376e-4848-b93c-0843e364cab6 tempest-TestShelveInstance-1683612675 None] remove export {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:912}} Apr 17 15:27:27 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-1f4b549a-26d0-4230-a266-eaf4b89d5f6c req-52fa7a0d-376e-4848-b93c-0843e364cab6 tempest-TestShelveInstance-1683612675 None] detach volume:Volume(_name_id=None,admin_metadata=,attach_status='attached',availability_zone='nova',bootable=True,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:25:25Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-tempest.common.compute-volume-1444682483',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=d895d11d-6880-4418-a9ad-feca813ed5d6,launched_at=2023-04-17T15:25:29Z,metadata=,migration_status=None,multiattach=False,previous_status=None,project_id='a2df343eb0d64207a5f9f91aa0810999',provider_auth='CHAP FVRqMWdJ 6d3UKemu7AN8VG',provider_geometry=None,provider_id=None,provider_location='172.16.0.220:3260,1 iqn.2010-10.org.openstack:d895d11d-6880-4418-a9ad-feca813ed5d6 0',replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:25:25Z,service_uuid='e628352e-c4cf-40eb-8389-cce40663026a',shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='in-use',terminated_at=None,updated_at=2023-04-17T15:25:36Z,use_quota=True,user_id='909c09f2587f42f895352916d3f8636a',volume_attachment=,volume_type=,volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c) from target:iqn.2010-10.org.openstack:d895d11d-6880-4418-a9ad-feca813ed5d6. {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:913}} Apr 17 15:27:27 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [req-1f4b549a-26d0-4230-a266-eaf4b89d5f6c req-52fa7a0d-376e-4848-b93c-0843e364cab6 tempest-TestShelveInstance-1683612675 None] detach volume v_d895d11d-6880-4418-a9ad-feca813ed5d6 from target iqn.2010-10.org.openstack:d895d11d-6880-4418-a9ad-feca813ed5d6 {{(pid=84696) detach_target_vol /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:606}} Apr 17 15:27:27 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-1f4b549a-26d0-4230-a266-eaf4b89d5f6c req-52fa7a0d-376e-4848-b93c-0843e364cab6 tempest-TestShelveInstance-1683612675 None] Sending pool request DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:d895d11d-6880-4418-a9ad-feca813ed5d6/luns/v_d895d11d-6880-4418-a9ad-feca813ed5d6 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:27:27 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-1f4b549a-26d0-4230-a266-eaf4b89d5f6c req-52fa7a0d-376e-4848-b93c-0843e364cab6 tempest-TestShelveInstance-1683612675 None] Sending DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:d895d11d-6880-4418-a9ad-feca813ed5d6/luns/v_d895d11d-6880-4418-a9ad-feca813ed5d6 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:27:27 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-1f4b549a-26d0-4230-a266-eaf4b89d5f6c req-52fa7a0d-376e-4848-b93c-0843e364cab6 tempest-TestShelveInstance-1683612675 None] Geting {'code': 204} from DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:d895d11d-6880-4418-a9ad-feca813ed5d6/luns/v_d895d11d-6880-4418-a9ad-feca813ed5d6 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:27:27 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-1f4b549a-26d0-4230-a266-eaf4b89d5f6c req-52fa7a0d-376e-4848-b93c-0843e364cab6 tempest-TestShelveInstance-1683612675 None] delete target: iqn.2010-10.org.openstack:d895d11d-6880-4418-a9ad-feca813ed5d6 {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:930}} Apr 17 15:27:27 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [req-1f4b549a-26d0-4230-a266-eaf4b89d5f6c req-52fa7a0d-376e-4848-b93c-0843e364cab6 tempest-TestShelveInstance-1683612675 None] delete target iqn.2010-10.org.openstack:d895d11d-6880-4418-a9ad-feca813ed5d6 {{(pid=84696) delete_target /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:442}} Apr 17 15:27:27 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-1f4b549a-26d0-4230-a266-eaf4b89d5f6c req-52fa7a0d-376e-4848-b93c-0843e364cab6 tempest-TestShelveInstance-1683612675 None] Sending pool request DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:d895d11d-6880-4418-a9ad-feca813ed5d6 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:27:27 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-1f4b549a-26d0-4230-a266-eaf4b89d5f6c req-52fa7a0d-376e-4848-b93c-0843e364cab6 tempest-TestShelveInstance-1683612675 None] Sending DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:d895d11d-6880-4418-a9ad-feca813ed5d6 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:27:27 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-1f4b549a-26d0-4230-a266-eaf4b89d5f6c req-52fa7a0d-376e-4848-b93c-0843e364cab6 tempest-TestShelveInstance-1683612675 None] Geting {'code': 204} from DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:d895d11d-6880-4418-a9ad-feca813ed5d6 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:27:27 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [req-1f4b549a-26d0-4230-a266-eaf4b89d5f6c req-52fa7a0d-376e-4848-b93c-0843e364cab6 tempest-TestShelveInstance-1683612675 None] target iqn.2010-10.org.openstack:d895d11d-6880-4418-a9ad-feca813ed5d6 deleted {{(pid=84696) delete_target /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:447}} Apr 17 15:27:31 user cinder-volume[84696]: DEBUG cinder.coordination [None req-72a330be-4562-4f8d-a314-3a6d1e24e684 tempest-AttachSCSIVolumeTestJSON-944917744 None] Acquiring lock "cinder-bf63d38e-00b8-46cb-acce-fda3fd9b6a9f-delete_volume" by "delete_volume" {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:192}} Apr 17 15:27:31 user cinder-volume[84696]: DEBUG cinder.coordination [None req-72a330be-4562-4f8d-a314-3a6d1e24e684 tempest-AttachSCSIVolumeTestJSON-944917744 None] Lock "cinder-bf63d38e-00b8-46cb-acce-fda3fd9b6a9f-delete_volume" acquired by "delete_volume" :: waited 0.009s {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:196}} Apr 17 15:27:31 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-72a330be-4562-4f8d-a314-3a6d1e24e684 tempest-AttachSCSIVolumeTestJSON-944917744 None] remove_export for volume: bf63d38e-00b8-46cb-acce-fda3fd9b6a9f. {{(pid=84696) remove_export /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:718}} Apr 17 15:27:31 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-72a330be-4562-4f8d-a314-3a6d1e24e684 tempest-AttachSCSIVolumeTestJSON-944917744 None] remove export {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:912}} Apr 17 15:27:31 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-72a330be-4562-4f8d-a314-3a6d1e24e684 tempest-AttachSCSIVolumeTestJSON-944917744 None] detach volume:Volume(_name_id=None,admin_metadata={},attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:26:14Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-AttachSCSIVolumeTestJSON-volume-1764995674',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=bf63d38e-00b8-46cb-acce-fda3fd9b6a9f,launched_at=2023-04-17T15:26:17Z,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='0fd4ceec29874bc38e8da456cab2048d',provider_auth='CHAP xSVisnfg 8KuvVtUOwfxz5X',provider_geometry=None,provider_id=None,provider_location='172.16.0.220:3260,1 iqn.2010-10.org.openstack:bf63d38e-00b8-46cb-acce-fda3fd9b6a9f 0',replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:26:14Z,service_uuid='e628352e-c4cf-40eb-8389-cce40663026a',shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='deleting',terminated_at=2023-04-17T15:27:31Z,updated_at=2023-04-17T15:27:31Z,use_quota=True,user_id='b1734894909e44118bd60e233dc501a0',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c) from target:iqn.2010-10.org.openstack:bf63d38e-00b8-46cb-acce-fda3fd9b6a9f. {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:913}} Apr 17 15:27:31 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-72a330be-4562-4f8d-a314-3a6d1e24e684 tempest-AttachSCSIVolumeTestJSON-944917744 None] detach volume v_bf63d38e-00b8-46cb-acce-fda3fd9b6a9f from target iqn.2010-10.org.openstack:bf63d38e-00b8-46cb-acce-fda3fd9b6a9f {{(pid=84696) detach_target_vol /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:606}} Apr 17 15:27:31 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-72a330be-4562-4f8d-a314-3a6d1e24e684 tempest-AttachSCSIVolumeTestJSON-944917744 None] Sending pool request DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:bf63d38e-00b8-46cb-acce-fda3fd9b6a9f/luns/v_bf63d38e-00b8-46cb-acce-fda3fd9b6a9f {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:27:31 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-72a330be-4562-4f8d-a314-3a6d1e24e684 tempest-AttachSCSIVolumeTestJSON-944917744 None] Sending DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:bf63d38e-00b8-46cb-acce-fda3fd9b6a9f/luns/v_bf63d38e-00b8-46cb-acce-fda3fd9b6a9f {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:27:31 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-72a330be-4562-4f8d-a314-3a6d1e24e684 tempest-AttachSCSIVolumeTestJSON-944917744 None] Geting {'code': 404, 'error': {'class': 'opene.exceptions.ItemNotFoundError', 'message': 'Target iqn.2010-10.org.openstack:bf63d38e-00b8-46cb-acce-fda3fd9b6a9f not exists.', 'url': 'http://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:bf63d38e-00b8-46cb-acce-fda3fd9b6a9f/luns/v_bf63d38e-00b8-46cb-acce-fda3fd9b6a9f'}, 'data': None} from DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:bf63d38e-00b8-46cb-acce-fda3fd9b6a9f/luns/v_bf63d38e-00b8-46cb-acce-fda3fd9b6a9f {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:27:31 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-72a330be-4562-4f8d-a314-3a6d1e24e684 tempest-AttachSCSIVolumeTestJSON-944917744 None] failed to remove resource iqn.2010-10.org.openstack:bf63d38e-00b8-46cb-acce-fda3fd9b6a9f because of JDSS resource v_bf63d38e-00b8-46cb-acce-fda3fd9b6a9f DNE. {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:920}} Apr 17 15:27:31 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-72a330be-4562-4f8d-a314-3a6d1e24e684 tempest-AttachSCSIVolumeTestJSON-944917744 None] delete target: iqn.2010-10.org.openstack:bf63d38e-00b8-46cb-acce-fda3fd9b6a9f {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:930}} Apr 17 15:27:31 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-72a330be-4562-4f8d-a314-3a6d1e24e684 tempest-AttachSCSIVolumeTestJSON-944917744 None] delete target iqn.2010-10.org.openstack:bf63d38e-00b8-46cb-acce-fda3fd9b6a9f {{(pid=84696) delete_target /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:442}} Apr 17 15:27:31 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-72a330be-4562-4f8d-a314-3a6d1e24e684 tempest-AttachSCSIVolumeTestJSON-944917744 None] Sending pool request DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:bf63d38e-00b8-46cb-acce-fda3fd9b6a9f {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:27:31 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-72a330be-4562-4f8d-a314-3a6d1e24e684 tempest-AttachSCSIVolumeTestJSON-944917744 None] Sending DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:bf63d38e-00b8-46cb-acce-fda3fd9b6a9f {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:27:31 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-72a330be-4562-4f8d-a314-3a6d1e24e684 tempest-AttachSCSIVolumeTestJSON-944917744 None] Geting {'code': 404, 'error': {'class': 'opene.exceptions.ItemNotFoundError', 'message': 'Target iqn.2010-10.org.openstack:bf63d38e-00b8-46cb-acce-fda3fd9b6a9f not exists.', 'url': 'http://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:bf63d38e-00b8-46cb-acce-fda3fd9b6a9f'}, 'data': None} from DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:bf63d38e-00b8-46cb-acce-fda3fd9b6a9f {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:27:31 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-72a330be-4562-4f8d-a314-3a6d1e24e684 tempest-AttachSCSIVolumeTestJSON-944917744 None] failed to remove resource iqn.2010-10.org.openstack:bf63d38e-00b8-46cb-acce-fda3fd9b6a9f because of JDSS resource iqn.2010-10.org.openstack:bf63d38e-00b8-46cb-acce-fda3fd9b6a9f DNE. {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:935}} Apr 17 15:27:31 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-72a330be-4562-4f8d-a314-3a6d1e24e684 tempest-AttachSCSIVolumeTestJSON-944917744 None] deleating volume v_bf63d38e-00b8-46cb-acce-fda3fd9b6a9f {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 15:27:31 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-72a330be-4562-4f8d-a314-3a6d1e24e684 tempest-AttachSCSIVolumeTestJSON-944917744 None] get snapshots for volume v_bf63d38e-00b8-46cb-acce-fda3fd9b6a9f {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 15:27:31 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-72a330be-4562-4f8d-a314-3a6d1e24e684 tempest-AttachSCSIVolumeTestJSON-944917744 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_bf63d38e-00b8-46cb-acce-fda3fd9b6a9f/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:27:31 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-72a330be-4562-4f8d-a314-3a6d1e24e684 tempest-AttachSCSIVolumeTestJSON-944917744 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_bf63d38e-00b8-46cb-acce-fda3fd9b6a9f/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:27:32 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-72a330be-4562-4f8d-a314-3a6d1e24e684 tempest-AttachSCSIVolumeTestJSON-944917744 None] Geting {'code': 200, 'error': None, 'data': {'results': 0, 'entries': []}} from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_bf63d38e-00b8-46cb-acce-fda3fd9b6a9f/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:27:32 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-72a330be-4562-4f8d-a314-3a6d1e24e684 tempest-AttachSCSIVolumeTestJSON-944917744 None] get volume v_bf63d38e-00b8-46cb-acce-fda3fd9b6a9f info {{(pid=84696) get_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:239}} Apr 17 15:27:32 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-72a330be-4562-4f8d-a314-3a6d1e24e684 tempest-AttachSCSIVolumeTestJSON-944917744 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_bf63d38e-00b8-46cb-acce-fda3fd9b6a9f {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:27:32 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-72a330be-4562-4f8d-a314-3a6d1e24e684 tempest-AttachSCSIVolumeTestJSON-944917744 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_bf63d38e-00b8-46cb-acce-fda3fd9b6a9f {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:27:32 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-72a330be-4562-4f8d-a314-3a6d1e24e684 tempest-AttachSCSIVolumeTestJSON-944917744 None] Geting {'code': 200, 'error': None, 'data': {'origin': None, 'relatime': None, 'acltype': None, 'vscan': None, 'full_name': 'Pool-0/v_bf63d38e-00b8-46cb-acce-fda3fd9b6a9f', 'userrefs': None, 'primarycache': 'all', 'logbias': 'latency', 'creation': '1681746569', 'sync': 'always', 'is_clone': False, 'dedup': 'off', 'sharenfs': None, 'receive_resume_token': None, 'volsize': '1073741824', 'referenced': '57344', 'sharesmb': None, 'createtxg': '188069', 'reservation': '0', 'scontext': None, 'mountpoint': None, 'casesensitivity': None, 'guid': '4720749242843479992', 'usedbyrefreservation': '0', 'dnodesize': None, 'written': '57344', 'logicalused': '28672', 'compressratio': '1.00', 'rootcontext': 'none', 'default_scsi_id': '92d72b6e9a809d92', 'type': 'volume', 'compression': 'lz4', 'snapdir': None, 'overlay': None, 'encryption': 'off', 'xattr': None, 'volmode': 'default', 'copies': '1', 'snapshot_limit': '18446744073709551615', 'aclinherit': None, 'defcontext': 'none', 'readonly': 'off', 'version': None, 'recordsize': None, 'filesystem_limit': None, 'mounted': None, 'mlslabel': 'none', 'secondarycache': 'all', 'refreservation': '0', 'available': '113889280000', 'san:volume_id': '92d72b6e9a809d92fb182ef88b4c4b6dd4b10e21', 'encryptionroot': None, 'exec': None, 'refquota': None, 'refcompressratio': '1.00', 'quota': None, 'utf8only': None, 'keylocation': 'none', 'snapdev': 'hidden', 'snapshot_count': '18446744073709551615', 'fscontext': 'none', 'clones': None, 'canmount': None, 'keystatus': None, 'atime': None, 'usedbysnapshots': '0', 'normalization': None, 'usedbychildren': '0', 'volblocksize': '131072', 'usedbydataset': '57344', 'objsetid': '123085', 'name': 'v_bf63d38e-00b8-46cb-acce-fda3fd9b6a9f', 'defer_destroy': None, 'pbkdf2iters': '0', 'checksum': 'on', 'redundant_metadata': 'all', 'filesystem_count': None, 'devices': None, 'keyformat': 'none', 'setuid': None, 'used': '57344', 'logicalreferenced': '28672', 'context': 'none', 'zoned': None, 'nbmand': None}} from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_bf63d38e-00b8-46cb-acce-fda3fd9b6a9f {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:27:32 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-72a330be-4562-4f8d-a314-3a6d1e24e684 tempest-AttachSCSIVolumeTestJSON-944917744 None] delete volume:v_bf63d38e-00b8-46cb-acce-fda3fd9b6a9f recursively children:{'force_umount': True} {{(pid=84696) delete_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:335}} Apr 17 15:27:32 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-72a330be-4562-4f8d-a314-3a6d1e24e684 tempest-AttachSCSIVolumeTestJSON-944917744 None] Sending pool request DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_bf63d38e-00b8-46cb-acce-fda3fd9b6a9f {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:27:32 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-72a330be-4562-4f8d-a314-3a6d1e24e684 tempest-AttachSCSIVolumeTestJSON-944917744 None] Sending DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_bf63d38e-00b8-46cb-acce-fda3fd9b6a9f {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:27:34 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-72a330be-4562-4f8d-a314-3a6d1e24e684 tempest-AttachSCSIVolumeTestJSON-944917744 None] Geting {'code': 204} from DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_bf63d38e-00b8-46cb-acce-fda3fd9b6a9f {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:27:34 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-72a330be-4562-4f8d-a314-3a6d1e24e684 tempest-AttachSCSIVolumeTestJSON-944917744 None] volume v_bf63d38e-00b8-46cb-acce-fda3fd9b6a9f deleted {{(pid=84696) delete_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:346}} Apr 17 15:27:34 user cinder-volume[84696]: DEBUG cinder.quota [None req-72a330be-4562-4f8d-a314-3a6d1e24e684 tempest-AttachSCSIVolumeTestJSON-944917744 None] Created reservations ['388aa88e-e505-4638-bfb1-7b6948b4ea66', 'f0abb836-1699-4a1a-9921-ed62c225855a', '1f68942d-ac26-4695-8f64-84d335185313', '50239b04-dccb-4104-9163-41d25dcfac26'] {{(pid=84696) reserve /opt/stack/cinder/cinder/quota.py:755}} Apr 17 15:27:34 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-72a330be-4562-4f8d-a314-3a6d1e24e684 tempest-AttachSCSIVolumeTestJSON-944917744 None] Updating volume stats {{(pid=84696) _update_volume_stats /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:724}} Apr 17 15:27:34 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-72a330be-4562-4f8d-a314-3a6d1e24e684 tempest-AttachSCSIVolumeTestJSON-944917744 None] Get pool Pool-0 fsprops {{(pid=84696) get_pool_stats /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:894}} Apr 17 15:27:34 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-72a330be-4562-4f8d-a314-3a6d1e24e684 tempest-AttachSCSIVolumeTestJSON-944917744 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:27:34 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-72a330be-4562-4f8d-a314-3a6d1e24e684 tempest-AttachSCSIVolumeTestJSON-944917744 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:27:35 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-72a330be-4562-4f8d-a314-3a6d1e24e684 tempest-AttachSCSIVolumeTestJSON-944917744 None] Geting {'code': 200, 'error': None, 'data': {'available': '113889361920', 'status': 30, 'import_status': {'import_steps': {'san_setup': True, 'vip_setup': True, 'zfs_import': True, 'mount_system_volume': True, 'nas_setup': True}, 'import_successful': True}, 'scan': None, 'encryption': {'enabled': False}, 'deduplication_rate': '1.00', 'sysvol_upgrade': 'not_needed', 'vdevs': [{'name': 'wwn-0x6000c296b793bf526a107a9d823e947a', 'iostats': {'read': '0', 'write': '0', 'chksum': '0'}, 'disks': [{'origin': 'local', 'slot': '', 'led': 'off', 'name': 'sdn', 'iostats': {'read': '0', 'write': '0', 'chksum': '0'}, 'alias': '', 'health': 'ONLINE', 'sn': '6000c296b793bf526a107a9d823e947a', 'trim_data': {'status': 'trim_unsupported', 'progress': None, 'trim_time': None}, 'path': 'pci-0000:03:00.0-scsi-0:0:14:0', 'model': 'Virtual disk', 'id': 'wwn-0x6000c296b793bf526a107a9d823e947a', 'size': 128849018880}], 'health': 'ONLINE', 'vdev_replacings': [], 'vdev_spares': [], 'type': ''}], 'id': '14181317334440137330', 'health': 'ONLINE', 'iostats': {'read': '0', 'write': '0', 'chksum': '0'}, 'operation': 'none', 'size': '127775277056', 'autotrim': False, 'name': 'Pool-0'}} from GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:27:35 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-72a330be-4562-4f8d-a314-3a6d1e24e684 tempest-AttachSCSIVolumeTestJSON-944917744 None] Total capacity: 119, Free 106. {{(pid=84696) _update_volume_stats /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:757}} Apr 17 15:27:35 user cinder-volume[84696]: DEBUG cinder.manager [None req-72a330be-4562-4f8d-a314-3a6d1e24e684 tempest-AttachSCSIVolumeTestJSON-944917744 None] Notifying Schedulers of capabilities ... {{(pid=84696) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 17 15:27:35 user cinder-volume[84696]: INFO cinder.volume.manager [None req-72a330be-4562-4f8d-a314-3a6d1e24e684 tempest-AttachSCSIVolumeTestJSON-944917744 None] Deleted volume successfully. Apr 17 15:27:35 user cinder-volume[84696]: DEBUG cinder.coordination [None req-72a330be-4562-4f8d-a314-3a6d1e24e684 tempest-AttachSCSIVolumeTestJSON-944917744 None] Lock "cinder-bf63d38e-00b8-46cb-acce-fda3fd9b6a9f-delete_volume" released by "delete_volume" :: held 4.346s {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:207}} Apr 17 15:27:38 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-a82ad222-921f-4371-a335-9977ac4c8cb7 tempest-ServerActionsTestJSON-645547919 None] Volume reschedule parameters: True retry: {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 15:27:38 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-a82ad222-921f-4371-a335-9977ac4c8cb7 tempest-ServerActionsTestJSON-645547919 None] Flow 'volume_create_manager' (129d2651-53b3-4f13-9452-c18bb847d33f) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:27:38 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-a82ad222-921f-4371-a335-9977ac4c8cb7 tempest-ServerActionsTestJSON-645547919 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (3cb59969-cc86-4421-8f70-072bdb3e0ea8) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:27:38 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-a82ad222-921f-4371-a335-9977ac4c8cb7 tempest-ServerActionsTestJSON-645547919 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (3cb59969-cc86-4421-8f70-072bdb3e0ea8) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:27:39Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-ServerActionsTestJSON-volume-1520907984',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=1072d05f-2ebb-4398-902c-902df25464e9,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='cb5f6b5ed2fc43b69c47e1fc18646545',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:27:39Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:27:39Z,use_quota=True,user_id='638a4f535e9740fc9d810639a5eb24d7',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:27:38 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-a82ad222-921f-4371-a335-9977ac4c8cb7 tempest-ServerActionsTestJSON-645547919 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (50e2cc97-fcaf-4f73-936a-bb644bef8400) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:27:38 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-a82ad222-921f-4371-a335-9977ac4c8cb7 tempest-ServerActionsTestJSON-645547919 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (50e2cc97-fcaf-4f73-936a-bb644bef8400) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:27:38 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-a82ad222-921f-4371-a335-9977ac4c8cb7 tempest-ServerActionsTestJSON-645547919 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (fea7cd85-51ed-45c3-9947-1533137f1a41) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:27:38 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-a82ad222-921f-4371-a335-9977ac4c8cb7 tempest-ServerActionsTestJSON-645547919 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (fea7cd85-51ed-45c3-9947-1533137f1a41) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': '1072d05f-2ebb-4398-902c-902df25464e9', 'volume_name': 'volume-1072d05f-2ebb-4398-902c-902df25464e9', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:27:38 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-a82ad222-921f-4371-a335-9977ac4c8cb7 tempest-ServerActionsTestJSON-645547919 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (dddcf0f0-0085-4b50-ac37-8cd26be90ec8) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:27:38 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-a82ad222-921f-4371-a335-9977ac4c8cb7 tempest-ServerActionsTestJSON-645547919 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (dddcf0f0-0085-4b50-ac37-8cd26be90ec8) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:27:38 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-a82ad222-921f-4371-a335-9977ac4c8cb7 tempest-ServerActionsTestJSON-645547919 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (be4a04ed-10ce-4f3c-8f30-3fe3b1efc99c) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:27:38 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-a82ad222-921f-4371-a335-9977ac4c8cb7 tempest-ServerActionsTestJSON-645547919 None] Volume 1072d05f-2ebb-4398-902c-902df25464e9: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-1072d05f-2ebb-4398-902c-902df25464e9', 'volume_size': 1} Apr 17 15:27:38 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-a82ad222-921f-4371-a335-9977ac4c8cb7 tempest-ServerActionsTestJSON-645547919 None] creating volume v_1072d05f-2ebb-4398-902c-902df25464e9. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 15:27:38 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-a82ad222-921f-4371-a335-9977ac4c8cb7 tempest-ServerActionsTestJSON-645547919 None] create volume {'name': 'v_1072d05f-2ebb-4398-902c-902df25464e9', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 15:27:38 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-a82ad222-921f-4371-a335-9977ac4c8cb7 tempest-ServerActionsTestJSON-645547919 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:27:38 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-a82ad222-921f-4371-a335-9977ac4c8cb7 tempest-ServerActionsTestJSON-645547919 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:27:40 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-a82ad222-921f-4371-a335-9977ac4c8cb7 tempest-ServerActionsTestJSON-645547919 None] Geting {'code': 201, 'error': None, 'data': {'full_name': 'Pool-0/v_1072d05f-2ebb-4398-902c-902df25464e9', 'is_clone': False, 'name': 'v_1072d05f-2ebb-4398-902c-902df25464e9'}} from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:27:40 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-a82ad222-921f-4371-a335-9977ac4c8cb7 tempest-ServerActionsTestJSON-645547919 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (be4a04ed-10ce-4f3c-8f30-3fe3b1efc99c) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'volume_name': 'volume-1072d05f-2ebb-4398-902c-902df25464e9', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:27:40 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-a82ad222-921f-4371-a335-9977ac4c8cb7 tempest-ServerActionsTestJSON-645547919 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeOnFinishTask;volume:create, create.end' (26d55b8f-b645-42c3-9bbf-983980d5c3c7) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:27:40 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-a82ad222-921f-4371-a335-9977ac4c8cb7 tempest-ServerActionsTestJSON-645547919 None] Volume volume-1072d05f-2ebb-4398-902c-902df25464e9 (1072d05f-2ebb-4398-902c-902df25464e9): created successfully Apr 17 15:27:40 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-a82ad222-921f-4371-a335-9977ac4c8cb7 tempest-ServerActionsTestJSON-645547919 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeOnFinishTask;volume:create, create.end' (26d55b8f-b645-42c3-9bbf-983980d5c3c7) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:27:40 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-a82ad222-921f-4371-a335-9977ac4c8cb7 tempest-ServerActionsTestJSON-645547919 None] Flow 'volume_create_manager' (129d2651-53b3-4f13-9452-c18bb847d33f) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:27:40 user cinder-volume[84696]: INFO cinder.volume.manager [None req-a82ad222-921f-4371-a335-9977ac4c8cb7 tempest-ServerActionsTestJSON-645547919 None] Created volume successfully. Apr 17 15:27:41 user cinder-volume[84696]: DEBUG cinder.coordination [None req-da8a8d24-05f3-4009-ae8e-bb439b06abdd tempest-TaggedBootDevicesTest_v242-1317742003 None] Acquiring lock "cinder-65a7c5eb-3af6-4afd-918f-ccb69a68641c-delete_volume" by "delete_volume" {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:192}} Apr 17 15:27:41 user cinder-volume[84696]: DEBUG cinder.coordination [None req-da8a8d24-05f3-4009-ae8e-bb439b06abdd tempest-TaggedBootDevicesTest_v242-1317742003 None] Lock "cinder-65a7c5eb-3af6-4afd-918f-ccb69a68641c-delete_volume" acquired by "delete_volume" :: waited 0.011s {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:196}} Apr 17 15:27:41 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-da8a8d24-05f3-4009-ae8e-bb439b06abdd tempest-TaggedBootDevicesTest_v242-1317742003 None] remove_export for volume: 65a7c5eb-3af6-4afd-918f-ccb69a68641c. {{(pid=84696) remove_export /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:718}} Apr 17 15:27:41 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-da8a8d24-05f3-4009-ae8e-bb439b06abdd tempest-TaggedBootDevicesTest_v242-1317742003 None] remove export {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:912}} Apr 17 15:27:41 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-da8a8d24-05f3-4009-ae8e-bb439b06abdd tempest-TaggedBootDevicesTest_v242-1317742003 None] detach volume:Volume(_name_id=None,admin_metadata={},attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:24:36Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TaggedBootDevicesTest_v242-volume-1252584131',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=65a7c5eb-3af6-4afd-918f-ccb69a68641c,launched_at=2023-04-17T15:24:38Z,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='8f0c848b490b47629051e961c682c5a5',provider_auth='CHAP wouATrvO yqXPcs1aB98YHb',provider_geometry=None,provider_id=None,provider_location='172.16.0.220:3260,1 iqn.2010-10.org.openstack:65a7c5eb-3af6-4afd-918f-ccb69a68641c 0',replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:24:36Z,service_uuid='e628352e-c4cf-40eb-8389-cce40663026a',shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='deleting',terminated_at=2023-04-17T15:27:41Z,updated_at=2023-04-17T15:27:41Z,use_quota=True,user_id='3e4b069991954c57993051fbe8665d35',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c) from target:iqn.2010-10.org.openstack:65a7c5eb-3af6-4afd-918f-ccb69a68641c. {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:913}} Apr 17 15:27:41 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-da8a8d24-05f3-4009-ae8e-bb439b06abdd tempest-TaggedBootDevicesTest_v242-1317742003 None] detach volume v_65a7c5eb-3af6-4afd-918f-ccb69a68641c from target iqn.2010-10.org.openstack:65a7c5eb-3af6-4afd-918f-ccb69a68641c {{(pid=84696) detach_target_vol /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:606}} Apr 17 15:27:41 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-da8a8d24-05f3-4009-ae8e-bb439b06abdd tempest-TaggedBootDevicesTest_v242-1317742003 None] Sending pool request DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:65a7c5eb-3af6-4afd-918f-ccb69a68641c/luns/v_65a7c5eb-3af6-4afd-918f-ccb69a68641c {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:27:41 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-da8a8d24-05f3-4009-ae8e-bb439b06abdd tempest-TaggedBootDevicesTest_v242-1317742003 None] Sending DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:65a7c5eb-3af6-4afd-918f-ccb69a68641c/luns/v_65a7c5eb-3af6-4afd-918f-ccb69a68641c {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:27:41 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-da8a8d24-05f3-4009-ae8e-bb439b06abdd tempest-TaggedBootDevicesTest_v242-1317742003 None] Geting {'code': 404, 'error': {'class': 'opene.exceptions.ItemNotFoundError', 'message': 'Target iqn.2010-10.org.openstack:65a7c5eb-3af6-4afd-918f-ccb69a68641c not exists.', 'url': 'http://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:65a7c5eb-3af6-4afd-918f-ccb69a68641c/luns/v_65a7c5eb-3af6-4afd-918f-ccb69a68641c'}, 'data': None} from DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:65a7c5eb-3af6-4afd-918f-ccb69a68641c/luns/v_65a7c5eb-3af6-4afd-918f-ccb69a68641c {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:27:41 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-da8a8d24-05f3-4009-ae8e-bb439b06abdd tempest-TaggedBootDevicesTest_v242-1317742003 None] failed to remove resource iqn.2010-10.org.openstack:65a7c5eb-3af6-4afd-918f-ccb69a68641c because of JDSS resource v_65a7c5eb-3af6-4afd-918f-ccb69a68641c DNE. {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:920}} Apr 17 15:27:41 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-da8a8d24-05f3-4009-ae8e-bb439b06abdd tempest-TaggedBootDevicesTest_v242-1317742003 None] delete target: iqn.2010-10.org.openstack:65a7c5eb-3af6-4afd-918f-ccb69a68641c {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:930}} Apr 17 15:27:41 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-da8a8d24-05f3-4009-ae8e-bb439b06abdd tempest-TaggedBootDevicesTest_v242-1317742003 None] delete target iqn.2010-10.org.openstack:65a7c5eb-3af6-4afd-918f-ccb69a68641c {{(pid=84696) delete_target /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:442}} Apr 17 15:27:41 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-da8a8d24-05f3-4009-ae8e-bb439b06abdd tempest-TaggedBootDevicesTest_v242-1317742003 None] Sending pool request DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:65a7c5eb-3af6-4afd-918f-ccb69a68641c {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:27:41 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-da8a8d24-05f3-4009-ae8e-bb439b06abdd tempest-TaggedBootDevicesTest_v242-1317742003 None] Sending DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:65a7c5eb-3af6-4afd-918f-ccb69a68641c {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:27:41 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-da8a8d24-05f3-4009-ae8e-bb439b06abdd tempest-TaggedBootDevicesTest_v242-1317742003 None] Geting {'code': 404, 'error': {'class': 'opene.exceptions.ItemNotFoundError', 'message': 'Target iqn.2010-10.org.openstack:65a7c5eb-3af6-4afd-918f-ccb69a68641c not exists.', 'url': 'http://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:65a7c5eb-3af6-4afd-918f-ccb69a68641c'}, 'data': None} from DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:65a7c5eb-3af6-4afd-918f-ccb69a68641c {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:27:41 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-da8a8d24-05f3-4009-ae8e-bb439b06abdd tempest-TaggedBootDevicesTest_v242-1317742003 None] failed to remove resource iqn.2010-10.org.openstack:65a7c5eb-3af6-4afd-918f-ccb69a68641c because of JDSS resource iqn.2010-10.org.openstack:65a7c5eb-3af6-4afd-918f-ccb69a68641c DNE. {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:935}} Apr 17 15:27:41 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-da8a8d24-05f3-4009-ae8e-bb439b06abdd tempest-TaggedBootDevicesTest_v242-1317742003 None] deleating volume v_65a7c5eb-3af6-4afd-918f-ccb69a68641c {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 15:27:41 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-da8a8d24-05f3-4009-ae8e-bb439b06abdd tempest-TaggedBootDevicesTest_v242-1317742003 None] get snapshots for volume v_65a7c5eb-3af6-4afd-918f-ccb69a68641c {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 15:27:41 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-da8a8d24-05f3-4009-ae8e-bb439b06abdd tempest-TaggedBootDevicesTest_v242-1317742003 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_65a7c5eb-3af6-4afd-918f-ccb69a68641c/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:27:41 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-da8a8d24-05f3-4009-ae8e-bb439b06abdd tempest-TaggedBootDevicesTest_v242-1317742003 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_65a7c5eb-3af6-4afd-918f-ccb69a68641c/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:27:42 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-da8a8d24-05f3-4009-ae8e-bb439b06abdd tempest-TaggedBootDevicesTest_v242-1317742003 None] Geting {'code': 200, 'error': None, 'data': {'results': 0, 'entries': []}} from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_65a7c5eb-3af6-4afd-918f-ccb69a68641c/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:27:42 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-da8a8d24-05f3-4009-ae8e-bb439b06abdd tempest-TaggedBootDevicesTest_v242-1317742003 None] get volume v_65a7c5eb-3af6-4afd-918f-ccb69a68641c info {{(pid=84696) get_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:239}} Apr 17 15:27:42 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-da8a8d24-05f3-4009-ae8e-bb439b06abdd tempest-TaggedBootDevicesTest_v242-1317742003 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_65a7c5eb-3af6-4afd-918f-ccb69a68641c {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:27:42 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-da8a8d24-05f3-4009-ae8e-bb439b06abdd tempest-TaggedBootDevicesTest_v242-1317742003 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_65a7c5eb-3af6-4afd-918f-ccb69a68641c {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:27:42 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-da8a8d24-05f3-4009-ae8e-bb439b06abdd tempest-TaggedBootDevicesTest_v242-1317742003 None] Geting {'code': 200, 'error': None, 'data': {'origin': None, 'relatime': None, 'acltype': None, 'vscan': None, 'full_name': 'Pool-0/v_65a7c5eb-3af6-4afd-918f-ccb69a68641c', 'userrefs': None, 'primarycache': 'all', 'logbias': 'latency', 'creation': '1681746470', 'sync': 'always', 'is_clone': False, 'dedup': 'off', 'sharenfs': None, 'receive_resume_token': None, 'volsize': '1073741824', 'referenced': '57344', 'sharesmb': None, 'createtxg': '187945', 'reservation': '0', 'scontext': None, 'mountpoint': None, 'casesensitivity': None, 'guid': '11917631500472345329', 'usedbyrefreservation': '0', 'dnodesize': None, 'written': '57344', 'logicalused': '28672', 'compressratio': '1.00', 'rootcontext': 'none', 'default_scsi_id': '0dfd3fcc160aa28c', 'type': 'volume', 'compression': 'lz4', 'snapdir': None, 'overlay': None, 'encryption': 'off', 'xattr': None, 'volmode': 'default', 'copies': '1', 'snapshot_limit': '18446744073709551615', 'aclinherit': None, 'defcontext': 'none', 'readonly': 'off', 'version': None, 'recordsize': None, 'filesystem_limit': None, 'mounted': None, 'mlslabel': 'none', 'secondarycache': 'all', 'refreservation': '0', 'available': '113889292288', 'san:volume_id': '0dfd3fcc160aa28c2e229f9e38c8c190e30268a9', 'encryptionroot': None, 'exec': None, 'refquota': None, 'refcompressratio': '1.00', 'quota': None, 'utf8only': None, 'keylocation': 'none', 'snapdev': 'hidden', 'snapshot_count': '18446744073709551615', 'fscontext': 'none', 'clones': None, 'canmount': None, 'keystatus': None, 'atime': None, 'usedbysnapshots': '0', 'normalization': None, 'usedbychildren': '0', 'volblocksize': '131072', 'usedbydataset': '57344', 'objsetid': '121920', 'name': 'v_65a7c5eb-3af6-4afd-918f-ccb69a68641c', 'defer_destroy': None, 'pbkdf2iters': '0', 'checksum': 'on', 'redundant_metadata': 'all', 'filesystem_count': None, 'devices': None, 'keyformat': 'none', 'setuid': None, 'used': '57344', 'logicalreferenced': '28672', 'context': 'none', 'zoned': None, 'nbmand': None}} from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_65a7c5eb-3af6-4afd-918f-ccb69a68641c {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:27:42 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-da8a8d24-05f3-4009-ae8e-bb439b06abdd tempest-TaggedBootDevicesTest_v242-1317742003 None] delete volume:v_65a7c5eb-3af6-4afd-918f-ccb69a68641c recursively children:{'force_umount': True} {{(pid=84696) delete_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:335}} Apr 17 15:27:42 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-da8a8d24-05f3-4009-ae8e-bb439b06abdd tempest-TaggedBootDevicesTest_v242-1317742003 None] Sending pool request DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_65a7c5eb-3af6-4afd-918f-ccb69a68641c {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:27:42 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-da8a8d24-05f3-4009-ae8e-bb439b06abdd tempest-TaggedBootDevicesTest_v242-1317742003 None] Sending DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_65a7c5eb-3af6-4afd-918f-ccb69a68641c {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:27:42 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-074d632d-94df-495c-b8ee-f8f421be37e3 tempest-ServerStableDeviceRescueTest-869290889 None] Volume reschedule parameters: True retry: {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 15:27:42 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-074d632d-94df-495c-b8ee-f8f421be37e3 tempest-ServerStableDeviceRescueTest-869290889 None] Flow 'volume_create_manager' (ced86d95-af56-4541-9979-29aa76b9b219) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:27:42 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-074d632d-94df-495c-b8ee-f8f421be37e3 tempest-ServerStableDeviceRescueTest-869290889 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (88b0c3f4-fa7f-475b-b388-09d35c168d22) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:27:42 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-074d632d-94df-495c-b8ee-f8f421be37e3 tempest-ServerStableDeviceRescueTest-869290889 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (88b0c3f4-fa7f-475b-b388-09d35c168d22) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:27:42Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-ServerStableDeviceRescueTest-volume-648918308',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=32819ec1-2236-465f-a1c7-ee31aa4a0911,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='6bdfb8642b9341f6ba289e920c33cf47',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:27:43Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:27:43Z,use_quota=True,user_id='a972c04314724233bc8e33b64747fecf',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:27:42 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-074d632d-94df-495c-b8ee-f8f421be37e3 tempest-ServerStableDeviceRescueTest-869290889 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (758cee34-e49c-424f-bf7a-9351466e4a38) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:27:42 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-074d632d-94df-495c-b8ee-f8f421be37e3 tempest-ServerStableDeviceRescueTest-869290889 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (758cee34-e49c-424f-bf7a-9351466e4a38) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:27:42 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-074d632d-94df-495c-b8ee-f8f421be37e3 tempest-ServerStableDeviceRescueTest-869290889 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (2b92e969-a64c-4d40-8560-b3ad28842c35) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:27:42 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-074d632d-94df-495c-b8ee-f8f421be37e3 tempest-ServerStableDeviceRescueTest-869290889 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (2b92e969-a64c-4d40-8560-b3ad28842c35) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': '32819ec1-2236-465f-a1c7-ee31aa4a0911', 'volume_name': 'volume-32819ec1-2236-465f-a1c7-ee31aa4a0911', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:27:42 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-074d632d-94df-495c-b8ee-f8f421be37e3 tempest-ServerStableDeviceRescueTest-869290889 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (c7ad6f76-27d1-408f-aff1-6709a5d161c2) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:27:42 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-074d632d-94df-495c-b8ee-f8f421be37e3 tempest-ServerStableDeviceRescueTest-869290889 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (c7ad6f76-27d1-408f-aff1-6709a5d161c2) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:27:42 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-074d632d-94df-495c-b8ee-f8f421be37e3 tempest-ServerStableDeviceRescueTest-869290889 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (76740cb4-69d5-4d14-b54e-ff677c3c6c17) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:27:42 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-074d632d-94df-495c-b8ee-f8f421be37e3 tempest-ServerStableDeviceRescueTest-869290889 None] Volume 32819ec1-2236-465f-a1c7-ee31aa4a0911: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-32819ec1-2236-465f-a1c7-ee31aa4a0911', 'volume_size': 1} Apr 17 15:27:42 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-074d632d-94df-495c-b8ee-f8f421be37e3 tempest-ServerStableDeviceRescueTest-869290889 None] creating volume v_32819ec1-2236-465f-a1c7-ee31aa4a0911. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 15:27:42 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-074d632d-94df-495c-b8ee-f8f421be37e3 tempest-ServerStableDeviceRescueTest-869290889 None] create volume {'name': 'v_32819ec1-2236-465f-a1c7-ee31aa4a0911', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 15:27:42 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-074d632d-94df-495c-b8ee-f8f421be37e3 tempest-ServerStableDeviceRescueTest-869290889 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:27:42 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-074d632d-94df-495c-b8ee-f8f421be37e3 tempest-ServerStableDeviceRescueTest-869290889 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:27:42 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-ebe6d921-be30-4631-84cb-a171f5836675 req-04233c96-93e7-45df-b8e7-0c48f4607371 tempest-ServerActionsTestJSON-645547919 None] create export for volume: 1072d05f-2ebb-4398-902c-902df25464e9. {{(pid=84696) create_export /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:699}} Apr 17 15:27:42 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-ebe6d921-be30-4631-84cb-a171f5836675 req-04233c96-93e7-45df-b8e7-0c48f4607371 tempest-ServerActionsTestJSON-645547919 None] ensure volume 1072d05f-2ebb-4398-902c-902df25464e9 assigned to a proper target {{(pid=84696) _ensure_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:866}} Apr 17 15:27:42 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [req-ebe6d921-be30-4631-84cb-a171f5836675 req-04233c96-93e7-45df-b8e7-0c48f4607371 tempest-ServerActionsTestJSON-645547919 None] check if targe iqn.2010-10.org.openstack:1072d05f-2ebb-4398-902c-902df25464e9 exists {{(pid=84696) is_target /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:376}} Apr 17 15:27:42 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-ebe6d921-be30-4631-84cb-a171f5836675 req-04233c96-93e7-45df-b8e7-0c48f4607371 tempest-ServerActionsTestJSON-645547919 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:1072d05f-2ebb-4398-902c-902df25464e9 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:27:42 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-ebe6d921-be30-4631-84cb-a171f5836675 req-04233c96-93e7-45df-b8e7-0c48f4607371 tempest-ServerActionsTestJSON-645547919 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:1072d05f-2ebb-4398-902c-902df25464e9 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:27:42 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-ebe6d921-be30-4631-84cb-a171f5836675 req-04233c96-93e7-45df-b8e7-0c48f4607371 tempest-ServerActionsTestJSON-645547919 None] Geting {'code': 404, 'error': {'class': 'opene.exceptions.ItemNotFoundError', 'message': 'Target iqn.2010-10.org.openstack:1072d05f-2ebb-4398-902c-902df25464e9 not exists.', 'url': 'http://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:1072d05f-2ebb-4398-902c-902df25464e9'}, 'data': None} from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:1072d05f-2ebb-4398-902c-902df25464e9 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:27:42 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-ebe6d921-be30-4631-84cb-a171f5836675 req-04233c96-93e7-45df-b8e7-0c48f4607371 tempest-ServerActionsTestJSON-645547919 None] create target and attach volume 1072d05f-2ebb-4398-902c-902df25464e9 to it {{(pid=84696) _create_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:837}} Apr 17 15:27:42 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [req-ebe6d921-be30-4631-84cb-a171f5836675 req-04233c96-93e7-45df-b8e7-0c48f4607371 tempest-ServerActionsTestJSON-645547919 None] create target iqn.2010-10.org.openstack:1072d05f-2ebb-4398-902c-902df25464e9 {{(pid=84696) create_target /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:410}} Apr 17 15:27:42 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-ebe6d921-be30-4631-84cb-a171f5836675 req-04233c96-93e7-45df-b8e7-0c48f4607371 tempest-ServerActionsTestJSON-645547919 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:27:42 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-ebe6d921-be30-4631-84cb-a171f5836675 req-04233c96-93e7-45df-b8e7-0c48f4607371 tempest-ServerActionsTestJSON-645547919 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:27:43 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:UPDATE=27,cinder:INSERT=15,cinder:SELECT=54,cinder:DELETE=4 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:27:44 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-da8a8d24-05f3-4009-ae8e-bb439b06abdd tempest-TaggedBootDevicesTest_v242-1317742003 None] Geting {'code': 204} from DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_65a7c5eb-3af6-4afd-918f-ccb69a68641c {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:27:44 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-da8a8d24-05f3-4009-ae8e-bb439b06abdd tempest-TaggedBootDevicesTest_v242-1317742003 None] volume v_65a7c5eb-3af6-4afd-918f-ccb69a68641c deleted {{(pid=84696) delete_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:346}} Apr 17 15:27:44 user cinder-volume[84696]: DEBUG cinder.quota [None req-da8a8d24-05f3-4009-ae8e-bb439b06abdd tempest-TaggedBootDevicesTest_v242-1317742003 None] Created reservations ['f037dfce-f2b1-4051-8b38-0b4be0086012', '77386f06-f83b-4ab4-9856-10bccb66e251', '9430e2f7-cf68-4193-acb9-1cd71490313e', '2bad9071-eb63-483a-aaf3-7158472a8e63'] {{(pid=84696) reserve /opt/stack/cinder/cinder/quota.py:755}} Apr 17 15:27:44 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-da8a8d24-05f3-4009-ae8e-bb439b06abdd tempest-TaggedBootDevicesTest_v242-1317742003 None] Updating volume stats {{(pid=84696) _update_volume_stats /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:724}} Apr 17 15:27:44 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-da8a8d24-05f3-4009-ae8e-bb439b06abdd tempest-TaggedBootDevicesTest_v242-1317742003 None] Get pool Pool-0 fsprops {{(pid=84696) get_pool_stats /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:894}} Apr 17 15:27:44 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-da8a8d24-05f3-4009-ae8e-bb439b06abdd tempest-TaggedBootDevicesTest_v242-1317742003 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:27:44 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-da8a8d24-05f3-4009-ae8e-bb439b06abdd tempest-TaggedBootDevicesTest_v242-1317742003 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:27:44 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-085d5231-e63e-4c3f-980c-e7ecad1da4e8 tempest-TestStampPattern-173447269 None] Volume reschedule parameters: True retry: {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 15:27:44 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-085d5231-e63e-4c3f-980c-e7ecad1da4e8 tempest-TestStampPattern-173447269 None] Flow 'volume_create_manager' (994a2fd4-d9a7-40ac-b833-8cb6163edecf) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:27:44 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-085d5231-e63e-4c3f-980c-e7ecad1da4e8 tempest-TestStampPattern-173447269 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (176f118e-f286-4227-be5f-ae0f30795add) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:27:44 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-085d5231-e63e-4c3f-980c-e7ecad1da4e8 tempest-TestStampPattern-173447269 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (176f118e-f286-4227-be5f-ae0f30795add) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:27:45Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TestStampPattern-volume-809461214',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=a9c177a1-61f0-4201-bb8d-7f18c9af396c,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='c9ca6aeb2f184cd58cf4b4dde051bdbb',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:27:45Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:27:45Z,use_quota=True,user_id='795213c1e99c46a7b6dd5512b1e9f48c',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:27:44 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-085d5231-e63e-4c3f-980c-e7ecad1da4e8 tempest-TestStampPattern-173447269 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (0aeb0540-b91f-46bf-8f3a-4b72fa36a1bd) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:27:44 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-085d5231-e63e-4c3f-980c-e7ecad1da4e8 tempest-TestStampPattern-173447269 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (0aeb0540-b91f-46bf-8f3a-4b72fa36a1bd) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:27:44 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-085d5231-e63e-4c3f-980c-e7ecad1da4e8 tempest-TestStampPattern-173447269 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (588f425d-d856-437e-ada7-a101858dd41c) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:27:44 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-085d5231-e63e-4c3f-980c-e7ecad1da4e8 tempest-TestStampPattern-173447269 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (588f425d-d856-437e-ada7-a101858dd41c) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': 'a9c177a1-61f0-4201-bb8d-7f18c9af396c', 'volume_name': 'volume-a9c177a1-61f0-4201-bb8d-7f18c9af396c', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:27:44 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-085d5231-e63e-4c3f-980c-e7ecad1da4e8 tempest-TestStampPattern-173447269 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (dfa7295c-eaa6-4e5a-bbce-26f04fa17f64) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:27:44 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-085d5231-e63e-4c3f-980c-e7ecad1da4e8 tempest-TestStampPattern-173447269 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (dfa7295c-eaa6-4e5a-bbce-26f04fa17f64) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:27:44 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-085d5231-e63e-4c3f-980c-e7ecad1da4e8 tempest-TestStampPattern-173447269 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (babafe55-7264-4445-8f6a-f47015e115e3) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:27:44 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-085d5231-e63e-4c3f-980c-e7ecad1da4e8 tempest-TestStampPattern-173447269 None] Volume a9c177a1-61f0-4201-bb8d-7f18c9af396c: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-a9c177a1-61f0-4201-bb8d-7f18c9af396c', 'volume_size': 1} Apr 17 15:27:44 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-085d5231-e63e-4c3f-980c-e7ecad1da4e8 tempest-TestStampPattern-173447269 None] creating volume v_a9c177a1-61f0-4201-bb8d-7f18c9af396c. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 15:27:44 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-085d5231-e63e-4c3f-980c-e7ecad1da4e8 tempest-TestStampPattern-173447269 None] create volume {'name': 'v_a9c177a1-61f0-4201-bb8d-7f18c9af396c', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 15:27:44 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-085d5231-e63e-4c3f-980c-e7ecad1da4e8 tempest-TestStampPattern-173447269 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:27:44 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-085d5231-e63e-4c3f-980c-e7ecad1da4e8 tempest-TestStampPattern-173447269 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:27:45 user cinder-volume[84696]: DEBUG cinder.coordination [None req-43b6f4d5-2620-4b6b-bf68-03834735566f tempest-TaggedBootDevicesTest_v242-1317742003 None] Acquiring lock "cinder-18475864-c202-4937-b18f-cc3e16c6d89f-delete_volume" by "delete_volume" {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:192}} Apr 17 15:27:45 user cinder-volume[84696]: DEBUG cinder.coordination [None req-43b6f4d5-2620-4b6b-bf68-03834735566f tempest-TaggedBootDevicesTest_v242-1317742003 None] Lock "cinder-18475864-c202-4937-b18f-cc3e16c6d89f-delete_volume" acquired by "delete_volume" :: waited 0.009s {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:196}} Apr 17 15:27:45 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-43b6f4d5-2620-4b6b-bf68-03834735566f tempest-TaggedBootDevicesTest_v242-1317742003 None] remove_export for volume: 18475864-c202-4937-b18f-cc3e16c6d89f. {{(pid=84696) remove_export /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:718}} Apr 17 15:27:45 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-43b6f4d5-2620-4b6b-bf68-03834735566f tempest-TaggedBootDevicesTest_v242-1317742003 None] remove export {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:912}} Apr 17 15:27:45 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-43b6f4d5-2620-4b6b-bf68-03834735566f tempest-TaggedBootDevicesTest_v242-1317742003 None] detach volume:Volume(_name_id=None,admin_metadata={},attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:24:33Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TaggedBootDevicesTest_v242-volume-1325806681',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=18475864-c202-4937-b18f-cc3e16c6d89f,launched_at=2023-04-17T15:24:35Z,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='8f0c848b490b47629051e961c682c5a5',provider_auth='CHAP pmlSchIr 83IVml0ZKYD5yo',provider_geometry=None,provider_id=None,provider_location='172.16.0.220:3260,1 iqn.2010-10.org.openstack:18475864-c202-4937-b18f-cc3e16c6d89f 0',replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:24:33Z,service_uuid='e628352e-c4cf-40eb-8389-cce40663026a',shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='deleting',terminated_at=2023-04-17T15:27:46Z,updated_at=2023-04-17T15:27:46Z,use_quota=True,user_id='3e4b069991954c57993051fbe8665d35',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c) from target:iqn.2010-10.org.openstack:18475864-c202-4937-b18f-cc3e16c6d89f. {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:913}} Apr 17 15:27:45 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-43b6f4d5-2620-4b6b-bf68-03834735566f tempest-TaggedBootDevicesTest_v242-1317742003 None] detach volume v_18475864-c202-4937-b18f-cc3e16c6d89f from target iqn.2010-10.org.openstack:18475864-c202-4937-b18f-cc3e16c6d89f {{(pid=84696) detach_target_vol /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:606}} Apr 17 15:27:45 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-43b6f4d5-2620-4b6b-bf68-03834735566f tempest-TaggedBootDevicesTest_v242-1317742003 None] Sending pool request DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:18475864-c202-4937-b18f-cc3e16c6d89f/luns/v_18475864-c202-4937-b18f-cc3e16c6d89f {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:27:45 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-43b6f4d5-2620-4b6b-bf68-03834735566f tempest-TaggedBootDevicesTest_v242-1317742003 None] Sending DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:18475864-c202-4937-b18f-cc3e16c6d89f/luns/v_18475864-c202-4937-b18f-cc3e16c6d89f {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:27:45 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-43b6f4d5-2620-4b6b-bf68-03834735566f tempest-TaggedBootDevicesTest_v242-1317742003 None] Geting {'code': 404, 'error': {'class': 'opene.exceptions.ItemNotFoundError', 'message': 'Target iqn.2010-10.org.openstack:18475864-c202-4937-b18f-cc3e16c6d89f not exists.', 'url': 'http://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:18475864-c202-4937-b18f-cc3e16c6d89f/luns/v_18475864-c202-4937-b18f-cc3e16c6d89f'}, 'data': None} from DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:18475864-c202-4937-b18f-cc3e16c6d89f/luns/v_18475864-c202-4937-b18f-cc3e16c6d89f {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:27:45 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-43b6f4d5-2620-4b6b-bf68-03834735566f tempest-TaggedBootDevicesTest_v242-1317742003 None] failed to remove resource iqn.2010-10.org.openstack:18475864-c202-4937-b18f-cc3e16c6d89f because of JDSS resource v_18475864-c202-4937-b18f-cc3e16c6d89f DNE. {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:920}} Apr 17 15:27:45 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-43b6f4d5-2620-4b6b-bf68-03834735566f tempest-TaggedBootDevicesTest_v242-1317742003 None] delete target: iqn.2010-10.org.openstack:18475864-c202-4937-b18f-cc3e16c6d89f {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:930}} Apr 17 15:27:45 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-43b6f4d5-2620-4b6b-bf68-03834735566f tempest-TaggedBootDevicesTest_v242-1317742003 None] delete target iqn.2010-10.org.openstack:18475864-c202-4937-b18f-cc3e16c6d89f {{(pid=84696) delete_target /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:442}} Apr 17 15:27:45 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-43b6f4d5-2620-4b6b-bf68-03834735566f tempest-TaggedBootDevicesTest_v242-1317742003 None] Sending pool request DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:18475864-c202-4937-b18f-cc3e16c6d89f {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:27:45 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-43b6f4d5-2620-4b6b-bf68-03834735566f tempest-TaggedBootDevicesTest_v242-1317742003 None] Sending DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:18475864-c202-4937-b18f-cc3e16c6d89f {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:27:45 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-43b6f4d5-2620-4b6b-bf68-03834735566f tempest-TaggedBootDevicesTest_v242-1317742003 None] Geting {'code': 404, 'error': {'class': 'opene.exceptions.ItemNotFoundError', 'message': 'Target iqn.2010-10.org.openstack:18475864-c202-4937-b18f-cc3e16c6d89f not exists.', 'url': 'http://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:18475864-c202-4937-b18f-cc3e16c6d89f'}, 'data': None} from DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:18475864-c202-4937-b18f-cc3e16c6d89f {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:27:45 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-43b6f4d5-2620-4b6b-bf68-03834735566f tempest-TaggedBootDevicesTest_v242-1317742003 None] failed to remove resource iqn.2010-10.org.openstack:18475864-c202-4937-b18f-cc3e16c6d89f because of JDSS resource iqn.2010-10.org.openstack:18475864-c202-4937-b18f-cc3e16c6d89f DNE. {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:935}} Apr 17 15:27:45 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-43b6f4d5-2620-4b6b-bf68-03834735566f tempest-TaggedBootDevicesTest_v242-1317742003 None] deleating volume v_18475864-c202-4937-b18f-cc3e16c6d89f {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 15:27:45 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-43b6f4d5-2620-4b6b-bf68-03834735566f tempest-TaggedBootDevicesTest_v242-1317742003 None] get snapshots for volume v_18475864-c202-4937-b18f-cc3e16c6d89f {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 15:27:45 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-43b6f4d5-2620-4b6b-bf68-03834735566f tempest-TaggedBootDevicesTest_v242-1317742003 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_18475864-c202-4937-b18f-cc3e16c6d89f/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:27:45 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-43b6f4d5-2620-4b6b-bf68-03834735566f tempest-TaggedBootDevicesTest_v242-1317742003 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_18475864-c202-4937-b18f-cc3e16c6d89f/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:27:45 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-074d632d-94df-495c-b8ee-f8f421be37e3 tempest-ServerStableDeviceRescueTest-869290889 None] Geting {'code': 201, 'error': None, 'data': {'full_name': 'Pool-0/v_32819ec1-2236-465f-a1c7-ee31aa4a0911', 'is_clone': False, 'name': 'v_32819ec1-2236-465f-a1c7-ee31aa4a0911'}} from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:27:45 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-074d632d-94df-495c-b8ee-f8f421be37e3 tempest-ServerStableDeviceRescueTest-869290889 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (76740cb4-69d5-4d14-b54e-ff677c3c6c17) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'volume_name': 'volume-32819ec1-2236-465f-a1c7-ee31aa4a0911', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:27:45 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-074d632d-94df-495c-b8ee-f8f421be37e3 tempest-ServerStableDeviceRescueTest-869290889 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeOnFinishTask;volume:create, create.end' (8327bfef-b5ff-429e-acfa-e2b0928d8904) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:27:45 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-074d632d-94df-495c-b8ee-f8f421be37e3 tempest-ServerStableDeviceRescueTest-869290889 None] Volume volume-32819ec1-2236-465f-a1c7-ee31aa4a0911 (32819ec1-2236-465f-a1c7-ee31aa4a0911): created successfully Apr 17 15:27:45 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-074d632d-94df-495c-b8ee-f8f421be37e3 tempest-ServerStableDeviceRescueTest-869290889 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeOnFinishTask;volume:create, create.end' (8327bfef-b5ff-429e-acfa-e2b0928d8904) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:27:45 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-074d632d-94df-495c-b8ee-f8f421be37e3 tempest-ServerStableDeviceRescueTest-869290889 None] Flow 'volume_create_manager' (ced86d95-af56-4541-9979-29aa76b9b219) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:27:45 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:UPDATE=32,cinder:SELECT=62,cinder:DELETE=1,cinder:INSERT=5 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:27:45 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-ebe6d921-be30-4631-84cb-a171f5836675 req-04233c96-93e7-45df-b8e7-0c48f4607371 tempest-ServerActionsTestJSON-645547919 None] Geting {'code': 201, 'error': None, 'data': {'active': True, 'allow_ip': [], 'conflicted': False, 'deny_ip': [], 'incoming_users_active': True, 'name': 'iqn.2010-10.org.openstack:1072d05f-2ebb-4398-902c-902df25464e9', 'outgoing_user': None}} from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:27:45 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [req-ebe6d921-be30-4631-84cb-a171f5836675 req-04233c96-93e7-45df-b8e7-0c48f4607371 tempest-ServerActionsTestJSON-645547919 None] atach volume v_1072d05f-2ebb-4398-902c-902df25464e9 to target iqn.2010-10.org.openstack:1072d05f-2ebb-4398-902c-902df25464e9 {{(pid=84696) attach_target_vol /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:576}} Apr 17 15:27:45 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-ebe6d921-be30-4631-84cb-a171f5836675 req-04233c96-93e7-45df-b8e7-0c48f4607371 tempest-ServerActionsTestJSON-645547919 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:1072d05f-2ebb-4398-902c-902df25464e9/luns {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:27:45 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-ebe6d921-be30-4631-84cb-a171f5836675 req-04233c96-93e7-45df-b8e7-0c48f4607371 tempest-ServerActionsTestJSON-645547919 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:1072d05f-2ebb-4398-902c-902df25464e9/luns {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:27:45 user cinder-volume[84696]: INFO cinder.volume.manager [None req-074d632d-94df-495c-b8ee-f8f421be37e3 tempest-ServerStableDeviceRescueTest-869290889 None] Created volume successfully. Apr 17 15:27:46 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-085d5231-e63e-4c3f-980c-e7ecad1da4e8 tempest-TestStampPattern-173447269 None] Geting {'code': 201, 'error': None, 'data': {'full_name': 'Pool-0/v_a9c177a1-61f0-4201-bb8d-7f18c9af396c', 'is_clone': False, 'name': 'v_a9c177a1-61f0-4201-bb8d-7f18c9af396c'}} from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:27:46 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-085d5231-e63e-4c3f-980c-e7ecad1da4e8 tempest-TestStampPattern-173447269 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (babafe55-7264-4445-8f6a-f47015e115e3) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'volume_name': 'volume-a9c177a1-61f0-4201-bb8d-7f18c9af396c', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:27:46 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-085d5231-e63e-4c3f-980c-e7ecad1da4e8 tempest-TestStampPattern-173447269 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeOnFinishTask;volume:create, create.end' (782a70c0-6892-4567-971b-c24754d6f05f) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:27:46 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-085d5231-e63e-4c3f-980c-e7ecad1da4e8 tempest-TestStampPattern-173447269 None] Volume volume-a9c177a1-61f0-4201-bb8d-7f18c9af396c (a9c177a1-61f0-4201-bb8d-7f18c9af396c): created successfully Apr 17 15:27:46 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-085d5231-e63e-4c3f-980c-e7ecad1da4e8 tempest-TestStampPattern-173447269 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeOnFinishTask;volume:create, create.end' (782a70c0-6892-4567-971b-c24754d6f05f) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:27:46 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-085d5231-e63e-4c3f-980c-e7ecad1da4e8 tempest-TestStampPattern-173447269 None] Flow 'volume_create_manager' (994a2fd4-d9a7-40ac-b833-8cb6163edecf) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:27:46 user cinder-volume[84696]: INFO cinder.volume.manager [None req-085d5231-e63e-4c3f-980c-e7ecad1da4e8 tempest-TestStampPattern-173447269 None] Created volume successfully. Apr 17 15:27:46 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-43b6f4d5-2620-4b6b-bf68-03834735566f tempest-TaggedBootDevicesTest_v242-1317742003 None] Geting {'code': 200, 'error': None, 'data': {'results': 0, 'entries': []}} from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_18475864-c202-4937-b18f-cc3e16c6d89f/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:27:46 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-43b6f4d5-2620-4b6b-bf68-03834735566f tempest-TaggedBootDevicesTest_v242-1317742003 None] get volume v_18475864-c202-4937-b18f-cc3e16c6d89f info {{(pid=84696) get_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:239}} Apr 17 15:27:46 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-43b6f4d5-2620-4b6b-bf68-03834735566f tempest-TaggedBootDevicesTest_v242-1317742003 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_18475864-c202-4937-b18f-cc3e16c6d89f {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:27:46 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-43b6f4d5-2620-4b6b-bf68-03834735566f tempest-TaggedBootDevicesTest_v242-1317742003 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_18475864-c202-4937-b18f-cc3e16c6d89f {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:27:46 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-43b6f4d5-2620-4b6b-bf68-03834735566f tempest-TaggedBootDevicesTest_v242-1317742003 None] Geting {'code': 200, 'error': None, 'data': {'origin': None, 'relatime': None, 'acltype': None, 'vscan': None, 'full_name': 'Pool-0/v_18475864-c202-4937-b18f-cc3e16c6d89f', 'userrefs': None, 'primarycache': 'all', 'logbias': 'latency', 'creation': '1681746468', 'sync': 'always', 'is_clone': False, 'dedup': 'off', 'sharenfs': None, 'receive_resume_token': None, 'volsize': '1073741824', 'referenced': '57344', 'sharesmb': None, 'createtxg': '187936', 'reservation': '0', 'scontext': None, 'mountpoint': None, 'casesensitivity': None, 'guid': '865107102125287200', 'usedbyrefreservation': '0', 'dnodesize': None, 'written': '57344', 'logicalused': '28672', 'compressratio': '1.00', 'rootcontext': 'none', 'default_scsi_id': '8a4673a10807941e', 'type': 'volume', 'compression': 'lz4', 'snapdir': None, 'overlay': None, 'encryption': 'off', 'xattr': None, 'volmode': 'default', 'copies': '1', 'snapshot_limit': '18446744073709551615', 'aclinherit': None, 'defcontext': 'none', 'readonly': 'off', 'version': None, 'recordsize': None, 'filesystem_limit': None, 'mounted': None, 'mlslabel': 'none', 'secondarycache': 'all', 'refreservation': '0', 'available': '113889210368', 'san:volume_id': '8a4673a10807941edec1ed8c923830cf38d13846', 'encryptionroot': None, 'exec': None, 'refquota': None, 'refcompressratio': '1.00', 'quota': None, 'utf8only': None, 'keylocation': 'none', 'snapdev': 'hidden', 'snapshot_count': '18446744073709551615', 'fscontext': 'none', 'clones': None, 'canmount': None, 'keystatus': None, 'atime': None, 'usedbysnapshots': '0', 'normalization': None, 'usedbychildren': '0', 'volblocksize': '131072', 'usedbydataset': '57344', 'objsetid': '123046', 'name': 'v_18475864-c202-4937-b18f-cc3e16c6d89f', 'defer_destroy': None, 'pbkdf2iters': '0', 'checksum': 'on', 'redundant_metadata': 'all', 'filesystem_count': None, 'devices': None, 'keyformat': 'none', 'setuid': None, 'used': '57344', 'logicalreferenced': '28672', 'context': 'none', 'zoned': None, 'nbmand': None}} from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_18475864-c202-4937-b18f-cc3e16c6d89f {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:27:46 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-43b6f4d5-2620-4b6b-bf68-03834735566f tempest-TaggedBootDevicesTest_v242-1317742003 None] delete volume:v_18475864-c202-4937-b18f-cc3e16c6d89f recursively children:{'force_umount': True} {{(pid=84696) delete_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:335}} Apr 17 15:27:46 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-43b6f4d5-2620-4b6b-bf68-03834735566f tempest-TaggedBootDevicesTest_v242-1317742003 None] Sending pool request DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_18475864-c202-4937-b18f-cc3e16c6d89f {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:27:46 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-43b6f4d5-2620-4b6b-bf68-03834735566f tempest-TaggedBootDevicesTest_v242-1317742003 None] Sending DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_18475864-c202-4937-b18f-cc3e16c6d89f {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:27:46 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-da8a8d24-05f3-4009-ae8e-bb439b06abdd tempest-TaggedBootDevicesTest_v242-1317742003 None] Geting {'code': 200, 'error': None, 'data': {'available': '113889222656', 'status': 30, 'import_status': {'import_steps': {'san_setup': True, 'vip_setup': True, 'zfs_import': True, 'mount_system_volume': True, 'nas_setup': True}, 'import_successful': True}, 'scan': None, 'encryption': {'enabled': False}, 'deduplication_rate': '1.00', 'sysvol_upgrade': 'not_needed', 'vdevs': [{'name': 'wwn-0x6000c296b793bf526a107a9d823e947a', 'iostats': {'read': '0', 'write': '0', 'chksum': '0'}, 'disks': [{'origin': 'local', 'slot': '', 'led': 'off', 'name': 'sdn', 'iostats': {'read': '0', 'write': '0', 'chksum': '0'}, 'alias': '', 'health': 'ONLINE', 'sn': '6000c296b793bf526a107a9d823e947a', 'trim_data': {'status': 'trim_unsupported', 'progress': None, 'trim_time': None}, 'path': 'pci-0000:03:00.0-scsi-0:0:14:0', 'model': 'Virtual disk', 'id': 'wwn-0x6000c296b793bf526a107a9d823e947a', 'size': 128849018880}], 'health': 'ONLINE', 'vdev_replacings': [], 'vdev_spares': [], 'type': ''}], 'id': '14181317334440137330', 'health': 'ONLINE', 'iostats': {'read': '0', 'write': '0', 'chksum': '0'}, 'operation': 'none', 'size': '127775277056', 'autotrim': False, 'name': 'Pool-0'}} from GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:27:46 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-da8a8d24-05f3-4009-ae8e-bb439b06abdd tempest-TaggedBootDevicesTest_v242-1317742003 None] Total capacity: 119, Free 106. {{(pid=84696) _update_volume_stats /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:757}} Apr 17 15:27:46 user cinder-volume[84696]: DEBUG cinder.manager [None req-da8a8d24-05f3-4009-ae8e-bb439b06abdd tempest-TaggedBootDevicesTest_v242-1317742003 None] Notifying Schedulers of capabilities ... {{(pid=84696) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 17 15:27:46 user cinder-volume[84696]: INFO cinder.volume.manager [None req-da8a8d24-05f3-4009-ae8e-bb439b06abdd tempest-TaggedBootDevicesTest_v242-1317742003 None] Deleted volume successfully. Apr 17 15:27:46 user cinder-volume[84696]: DEBUG cinder.coordination [None req-da8a8d24-05f3-4009-ae8e-bb439b06abdd tempest-TaggedBootDevicesTest_v242-1317742003 None] Lock "cinder-65a7c5eb-3af6-4afd-918f-ccb69a68641c-delete_volume" released by "delete_volume" :: held 5.607s {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:207}} Apr 17 15:27:47 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-ebe6d921-be30-4631-84cb-a171f5836675 req-04233c96-93e7-45df-b8e7-0c48f4607371 tempest-ServerActionsTestJSON-645547919 None] Geting {'code': 201, 'error': None, 'data': {'block_size': 512, 'device_handler': 'vdisk_blockio', 'eui': '3933326166393635', 'lun': 0, 'mode': 'wt', 'name': 'v_1072d05f-2ebb-4398-902c-902df25464e9', 'prod_id': 'Storage', 'scsi_id': '932af96517a71872'}} from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:1072d05f-2ebb-4398-902c-902df25464e9/luns {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:27:47 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [req-ebe6d921-be30-4631-84cb-a171f5836675 req-04233c96-93e7-45df-b8e7-0c48f4607371 tempest-ServerActionsTestJSON-645547919 None] add credentails to target iqn.2010-10.org.openstack:1072d05f-2ebb-4398-902c-902df25464e9 {{(pid=84696) create_target_user /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:474}} Apr 17 15:27:47 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-ebe6d921-be30-4631-84cb-a171f5836675 req-04233c96-93e7-45df-b8e7-0c48f4607371 tempest-ServerActionsTestJSON-645547919 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:1072d05f-2ebb-4398-902c-902df25464e9/incoming-users {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:27:47 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-ebe6d921-be30-4631-84cb-a171f5836675 req-04233c96-93e7-45df-b8e7-0c48f4607371 tempest-ServerActionsTestJSON-645547919 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:1072d05f-2ebb-4398-902c-902df25464e9/incoming-users {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:27:47 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-5346a9d0-f242-43c9-a311-93d0458ac4a7 req-cf528a18-6d05-4f1a-89ad-5074a53f0098 tempest-ServerStableDeviceRescueTest-869290889 None] create export for volume: 32819ec1-2236-465f-a1c7-ee31aa4a0911. {{(pid=84696) create_export /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:699}} Apr 17 15:27:47 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-5346a9d0-f242-43c9-a311-93d0458ac4a7 req-cf528a18-6d05-4f1a-89ad-5074a53f0098 tempest-ServerStableDeviceRescueTest-869290889 None] ensure volume 32819ec1-2236-465f-a1c7-ee31aa4a0911 assigned to a proper target {{(pid=84696) _ensure_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:866}} Apr 17 15:27:47 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [req-5346a9d0-f242-43c9-a311-93d0458ac4a7 req-cf528a18-6d05-4f1a-89ad-5074a53f0098 tempest-ServerStableDeviceRescueTest-869290889 None] check if targe iqn.2010-10.org.openstack:32819ec1-2236-465f-a1c7-ee31aa4a0911 exists {{(pid=84696) is_target /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:376}} Apr 17 15:27:47 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-5346a9d0-f242-43c9-a311-93d0458ac4a7 req-cf528a18-6d05-4f1a-89ad-5074a53f0098 tempest-ServerStableDeviceRescueTest-869290889 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:32819ec1-2236-465f-a1c7-ee31aa4a0911 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:27:47 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-5346a9d0-f242-43c9-a311-93d0458ac4a7 req-cf528a18-6d05-4f1a-89ad-5074a53f0098 tempest-ServerStableDeviceRescueTest-869290889 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:32819ec1-2236-465f-a1c7-ee31aa4a0911 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:27:47 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-5346a9d0-f242-43c9-a311-93d0458ac4a7 req-cf528a18-6d05-4f1a-89ad-5074a53f0098 tempest-ServerStableDeviceRescueTest-869290889 None] Geting {'code': 404, 'error': {'class': 'opene.exceptions.ItemNotFoundError', 'message': 'Target iqn.2010-10.org.openstack:32819ec1-2236-465f-a1c7-ee31aa4a0911 not exists.', 'url': 'http://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:32819ec1-2236-465f-a1c7-ee31aa4a0911'}, 'data': None} from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:32819ec1-2236-465f-a1c7-ee31aa4a0911 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:27:47 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-5346a9d0-f242-43c9-a311-93d0458ac4a7 req-cf528a18-6d05-4f1a-89ad-5074a53f0098 tempest-ServerStableDeviceRescueTest-869290889 None] create target and attach volume 32819ec1-2236-465f-a1c7-ee31aa4a0911 to it {{(pid=84696) _create_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:837}} Apr 17 15:27:47 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [req-5346a9d0-f242-43c9-a311-93d0458ac4a7 req-cf528a18-6d05-4f1a-89ad-5074a53f0098 tempest-ServerStableDeviceRescueTest-869290889 None] create target iqn.2010-10.org.openstack:32819ec1-2236-465f-a1c7-ee31aa4a0911 {{(pid=84696) create_target /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:410}} Apr 17 15:27:47 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-5346a9d0-f242-43c9-a311-93d0458ac4a7 req-cf528a18-6d05-4f1a-89ad-5074a53f0098 tempest-ServerStableDeviceRescueTest-869290889 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:27:47 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-5346a9d0-f242-43c9-a311-93d0458ac4a7 req-cf528a18-6d05-4f1a-89ad-5074a53f0098 tempest-ServerStableDeviceRescueTest-869290889 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:27:47 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-ebe6d921-be30-4631-84cb-a171f5836675 req-04233c96-93e7-45df-b8e7-0c48f4607371 tempest-ServerActionsTestJSON-645547919 None] Geting {'code': 201, 'error': None, 'data': {'name': 'NKISaVyG'}} from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:1072d05f-2ebb-4398-902c-902df25464e9/incoming-users {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:27:47 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-ebe6d921-be30-4631-84cb-a171f5836675 req-04233c96-93e7-45df-b8e7-0c48f4607371 tempest-ServerActionsTestJSON-645547919 None] initialize_connection for 1072d05f-2ebb-4398-902c-902df25464e9 10.0.0.210. {{(pid=84696) initialize_connection /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:1018}} Apr 17 15:27:47 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-ebe6d921-be30-4631-84cb-a171f5836675 req-04233c96-93e7-45df-b8e7-0c48f4607371 tempest-ServerActionsTestJSON-645547919 None] Connection info returned from driver {'target_iqn': 'iqn.2010-10.org.openstack:1072d05f-2ebb-4398-902c-902df25464e9', 'target_portal': '172.16.0.220:3260', 'target_discovered': False, 'auth_method': 'CHAP', 'auth_username': 'NKISaVyG', 'auth_password': '***', 'target_lun': 0, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False, 'driver_volume_type': 'iscsi', 'attachment_id': '2cb8df32-9487-4e02-8868-35c8aefceb87'} {{(pid=84696) _connection_create /opt/stack/cinder/cinder/volume/manager.py:4875}} Apr 17 15:27:48 user cinder-volume[84696]: INFO cinder.volume.manager [req-ebe6d921-be30-4631-84cb-a171f5836675 req-04233c96-93e7-45df-b8e7-0c48f4607371 tempest-ServerActionsTestJSON-645547919 None] attachment_update completed successfully. Apr 17 15:27:49 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-43b6f4d5-2620-4b6b-bf68-03834735566f tempest-TaggedBootDevicesTest_v242-1317742003 None] Geting {'code': 204} from DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_18475864-c202-4937-b18f-cc3e16c6d89f {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:27:49 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-43b6f4d5-2620-4b6b-bf68-03834735566f tempest-TaggedBootDevicesTest_v242-1317742003 None] volume v_18475864-c202-4937-b18f-cc3e16c6d89f deleted {{(pid=84696) delete_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:346}} Apr 17 15:27:49 user cinder-volume[84696]: DEBUG cinder.quota [None req-43b6f4d5-2620-4b6b-bf68-03834735566f tempest-TaggedBootDevicesTest_v242-1317742003 None] Created reservations ['5e7b75c0-a24a-42f9-b2c6-0cd7b334e86d', '9906830b-714f-488e-b8bd-296200f3a36d', 'ea962d61-b197-454f-bc42-428136515949', '20456215-6f9a-4275-8cca-36f5151c168d'] {{(pid=84696) reserve /opt/stack/cinder/cinder/quota.py:755}} Apr 17 15:27:49 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-43b6f4d5-2620-4b6b-bf68-03834735566f tempest-TaggedBootDevicesTest_v242-1317742003 None] Updating volume stats {{(pid=84696) _update_volume_stats /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:724}} Apr 17 15:27:49 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-43b6f4d5-2620-4b6b-bf68-03834735566f tempest-TaggedBootDevicesTest_v242-1317742003 None] Get pool Pool-0 fsprops {{(pid=84696) get_pool_stats /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:894}} Apr 17 15:27:49 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-43b6f4d5-2620-4b6b-bf68-03834735566f tempest-TaggedBootDevicesTest_v242-1317742003 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:27:49 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-43b6f4d5-2620-4b6b-bf68-03834735566f tempest-TaggedBootDevicesTest_v242-1317742003 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:27:50 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-b474c39e-ef83-4b0a-868f-b4857ff3294d req-1c623693-99db-4bf7-964a-a8a3505d21f4 tempest-AttachVolumeShelveTestJSON-1745617049 None] create export for volume: 13bf568b-585c-4556-ab78-e9930ad50214. {{(pid=84696) create_export /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:699}} Apr 17 15:27:50 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-b474c39e-ef83-4b0a-868f-b4857ff3294d req-1c623693-99db-4bf7-964a-a8a3505d21f4 tempest-AttachVolumeShelveTestJSON-1745617049 None] ensure volume 13bf568b-585c-4556-ab78-e9930ad50214 assigned to a proper target {{(pid=84696) _ensure_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:866}} Apr 17 15:27:50 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [req-b474c39e-ef83-4b0a-868f-b4857ff3294d req-1c623693-99db-4bf7-964a-a8a3505d21f4 tempest-AttachVolumeShelveTestJSON-1745617049 None] check if targe iqn.2010-10.org.openstack:13bf568b-585c-4556-ab78-e9930ad50214 exists {{(pid=84696) is_target /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:376}} Apr 17 15:27:50 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-b474c39e-ef83-4b0a-868f-b4857ff3294d req-1c623693-99db-4bf7-964a-a8a3505d21f4 tempest-AttachVolumeShelveTestJSON-1745617049 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:13bf568b-585c-4556-ab78-e9930ad50214 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:27:50 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-b474c39e-ef83-4b0a-868f-b4857ff3294d req-1c623693-99db-4bf7-964a-a8a3505d21f4 tempest-AttachVolumeShelveTestJSON-1745617049 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:13bf568b-585c-4556-ab78-e9930ad50214 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:27:50 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-b474c39e-ef83-4b0a-868f-b4857ff3294d req-1c623693-99db-4bf7-964a-a8a3505d21f4 tempest-AttachVolumeShelveTestJSON-1745617049 None] Geting {'code': 404, 'error': {'class': 'opene.exceptions.ItemNotFoundError', 'message': 'Target iqn.2010-10.org.openstack:13bf568b-585c-4556-ab78-e9930ad50214 not exists.', 'url': 'http://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:13bf568b-585c-4556-ab78-e9930ad50214'}, 'data': None} from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:13bf568b-585c-4556-ab78-e9930ad50214 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:27:50 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-b474c39e-ef83-4b0a-868f-b4857ff3294d req-1c623693-99db-4bf7-964a-a8a3505d21f4 tempest-AttachVolumeShelveTestJSON-1745617049 None] create target and attach volume 13bf568b-585c-4556-ab78-e9930ad50214 to it {{(pid=84696) _create_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:837}} Apr 17 15:27:50 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [req-b474c39e-ef83-4b0a-868f-b4857ff3294d req-1c623693-99db-4bf7-964a-a8a3505d21f4 tempest-AttachVolumeShelveTestJSON-1745617049 None] create target iqn.2010-10.org.openstack:13bf568b-585c-4556-ab78-e9930ad50214 {{(pid=84696) create_target /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:410}} Apr 17 15:27:50 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-b474c39e-ef83-4b0a-868f-b4857ff3294d req-1c623693-99db-4bf7-964a-a8a3505d21f4 tempest-AttachVolumeShelveTestJSON-1745617049 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:27:50 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-b474c39e-ef83-4b0a-868f-b4857ff3294d req-1c623693-99db-4bf7-964a-a8a3505d21f4 tempest-AttachVolumeShelveTestJSON-1745617049 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:27:50 user cinder-volume[84696]: DEBUG cinder.coordination [None req-122fd13e-6b37-4630-a7df-3990d5d9f41d tempest-TaggedBootDevicesTest_v242-1317742003 None] Acquiring lock "cinder-c74b7878-4936-4015-8b63-142923947310-delete_volume" by "delete_volume" {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:192}} Apr 17 15:27:50 user cinder-volume[84696]: DEBUG cinder.coordination [None req-122fd13e-6b37-4630-a7df-3990d5d9f41d tempest-TaggedBootDevicesTest_v242-1317742003 None] Lock "cinder-c74b7878-4936-4015-8b63-142923947310-delete_volume" acquired by "delete_volume" :: waited 0.011s {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:196}} Apr 17 15:27:50 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-122fd13e-6b37-4630-a7df-3990d5d9f41d tempest-TaggedBootDevicesTest_v242-1317742003 None] remove_export for volume: c74b7878-4936-4015-8b63-142923947310. {{(pid=84696) remove_export /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:718}} Apr 17 15:27:50 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-122fd13e-6b37-4630-a7df-3990d5d9f41d tempest-TaggedBootDevicesTest_v242-1317742003 None] remove export {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:912}} Apr 17 15:27:50 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-122fd13e-6b37-4630-a7df-3990d5d9f41d tempest-TaggedBootDevicesTest_v242-1317742003 None] detach volume:Volume(_name_id=None,admin_metadata={},attach_status='detached',availability_zone='nova',bootable=True,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:24:01Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TaggedBootDevicesTest_v242-volume-1367581919',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=c74b7878-4936-4015-8b63-142923947310,launched_at=2023-04-17T15:24:32Z,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='8f0c848b490b47629051e961c682c5a5',provider_auth='CHAP PUvXoCNO 1H8kFoBshXZL64',provider_geometry=None,provider_id=None,provider_location='172.16.0.220:3260,1 iqn.2010-10.org.openstack:c74b7878-4936-4015-8b63-142923947310 0',replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:24:01Z,service_uuid='e628352e-c4cf-40eb-8389-cce40663026a',shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='deleting',terminated_at=2023-04-17T15:27:50Z,updated_at=2023-04-17T15:27:50Z,use_quota=True,user_id='3e4b069991954c57993051fbe8665d35',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c) from target:iqn.2010-10.org.openstack:c74b7878-4936-4015-8b63-142923947310. {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:913}} Apr 17 15:27:50 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-122fd13e-6b37-4630-a7df-3990d5d9f41d tempest-TaggedBootDevicesTest_v242-1317742003 None] detach volume v_c74b7878-4936-4015-8b63-142923947310 from target iqn.2010-10.org.openstack:c74b7878-4936-4015-8b63-142923947310 {{(pid=84696) detach_target_vol /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:606}} Apr 17 15:27:50 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-122fd13e-6b37-4630-a7df-3990d5d9f41d tempest-TaggedBootDevicesTest_v242-1317742003 None] Sending pool request DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:c74b7878-4936-4015-8b63-142923947310/luns/v_c74b7878-4936-4015-8b63-142923947310 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:27:50 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-122fd13e-6b37-4630-a7df-3990d5d9f41d tempest-TaggedBootDevicesTest_v242-1317742003 None] Sending DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:c74b7878-4936-4015-8b63-142923947310/luns/v_c74b7878-4936-4015-8b63-142923947310 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:27:50 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-122fd13e-6b37-4630-a7df-3990d5d9f41d tempest-TaggedBootDevicesTest_v242-1317742003 None] Geting {'code': 404, 'error': {'class': 'opene.exceptions.ItemNotFoundError', 'message': 'Target iqn.2010-10.org.openstack:c74b7878-4936-4015-8b63-142923947310 not exists.', 'url': 'http://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:c74b7878-4936-4015-8b63-142923947310/luns/v_c74b7878-4936-4015-8b63-142923947310'}, 'data': None} from DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:c74b7878-4936-4015-8b63-142923947310/luns/v_c74b7878-4936-4015-8b63-142923947310 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:27:50 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-122fd13e-6b37-4630-a7df-3990d5d9f41d tempest-TaggedBootDevicesTest_v242-1317742003 None] failed to remove resource iqn.2010-10.org.openstack:c74b7878-4936-4015-8b63-142923947310 because of JDSS resource v_c74b7878-4936-4015-8b63-142923947310 DNE. {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:920}} Apr 17 15:27:50 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-122fd13e-6b37-4630-a7df-3990d5d9f41d tempest-TaggedBootDevicesTest_v242-1317742003 None] delete target: iqn.2010-10.org.openstack:c74b7878-4936-4015-8b63-142923947310 {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:930}} Apr 17 15:27:50 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-122fd13e-6b37-4630-a7df-3990d5d9f41d tempest-TaggedBootDevicesTest_v242-1317742003 None] delete target iqn.2010-10.org.openstack:c74b7878-4936-4015-8b63-142923947310 {{(pid=84696) delete_target /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:442}} Apr 17 15:27:50 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-122fd13e-6b37-4630-a7df-3990d5d9f41d tempest-TaggedBootDevicesTest_v242-1317742003 None] Sending pool request DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:c74b7878-4936-4015-8b63-142923947310 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:27:50 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-122fd13e-6b37-4630-a7df-3990d5d9f41d tempest-TaggedBootDevicesTest_v242-1317742003 None] Sending DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:c74b7878-4936-4015-8b63-142923947310 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:27:50 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-122fd13e-6b37-4630-a7df-3990d5d9f41d tempest-TaggedBootDevicesTest_v242-1317742003 None] Geting {'code': 404, 'error': {'class': 'opene.exceptions.ItemNotFoundError', 'message': 'Target iqn.2010-10.org.openstack:c74b7878-4936-4015-8b63-142923947310 not exists.', 'url': 'http://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:c74b7878-4936-4015-8b63-142923947310'}, 'data': None} from DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:c74b7878-4936-4015-8b63-142923947310 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:27:50 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-122fd13e-6b37-4630-a7df-3990d5d9f41d tempest-TaggedBootDevicesTest_v242-1317742003 None] failed to remove resource iqn.2010-10.org.openstack:c74b7878-4936-4015-8b63-142923947310 because of JDSS resource iqn.2010-10.org.openstack:c74b7878-4936-4015-8b63-142923947310 DNE. {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:935}} Apr 17 15:27:50 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-122fd13e-6b37-4630-a7df-3990d5d9f41d tempest-TaggedBootDevicesTest_v242-1317742003 None] deleating volume v_c74b7878-4936-4015-8b63-142923947310 {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 15:27:50 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-122fd13e-6b37-4630-a7df-3990d5d9f41d tempest-TaggedBootDevicesTest_v242-1317742003 None] get snapshots for volume v_c74b7878-4936-4015-8b63-142923947310 {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 15:27:50 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-122fd13e-6b37-4630-a7df-3990d5d9f41d tempest-TaggedBootDevicesTest_v242-1317742003 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_c74b7878-4936-4015-8b63-142923947310/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:27:50 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-122fd13e-6b37-4630-a7df-3990d5d9f41d tempest-TaggedBootDevicesTest_v242-1317742003 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_c74b7878-4936-4015-8b63-142923947310/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:27:50 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-43b6f4d5-2620-4b6b-bf68-03834735566f tempest-TaggedBootDevicesTest_v242-1317742003 None] Geting {'code': 200, 'error': None, 'data': {'available': '113889292288', 'status': 30, 'import_status': {'import_steps': {'san_setup': True, 'vip_setup': True, 'zfs_import': True, 'mount_system_volume': True, 'nas_setup': True}, 'import_successful': True}, 'scan': None, 'encryption': {'enabled': False}, 'deduplication_rate': '1.00', 'sysvol_upgrade': 'not_needed', 'vdevs': [{'name': 'wwn-0x6000c296b793bf526a107a9d823e947a', 'iostats': {'read': '0', 'write': '0', 'chksum': '0'}, 'disks': [{'origin': 'local', 'slot': '', 'led': 'off', 'name': 'sdn', 'iostats': {'read': '0', 'write': '0', 'chksum': '0'}, 'alias': '', 'health': 'ONLINE', 'sn': '6000c296b793bf526a107a9d823e947a', 'trim_data': {'status': 'trim_unsupported', 'progress': None, 'trim_time': None}, 'path': 'pci-0000:03:00.0-scsi-0:0:14:0', 'model': 'Virtual disk', 'id': 'wwn-0x6000c296b793bf526a107a9d823e947a', 'size': 128849018880}], 'health': 'ONLINE', 'vdev_replacings': [], 'vdev_spares': [], 'type': ''}], 'id': '14181317334440137330', 'health': 'ONLINE', 'iostats': {'read': '0', 'write': '0', 'chksum': '0'}, 'operation': 'none', 'size': '127775277056', 'autotrim': False, 'name': 'Pool-0'}} from GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:27:50 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-43b6f4d5-2620-4b6b-bf68-03834735566f tempest-TaggedBootDevicesTest_v242-1317742003 None] Total capacity: 119, Free 106. {{(pid=84696) _update_volume_stats /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:757}} Apr 17 15:27:50 user cinder-volume[84696]: DEBUG cinder.manager [None req-43b6f4d5-2620-4b6b-bf68-03834735566f tempest-TaggedBootDevicesTest_v242-1317742003 None] Notifying Schedulers of capabilities ... {{(pid=84696) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 17 15:27:50 user cinder-volume[84696]: INFO cinder.volume.manager [None req-43b6f4d5-2620-4b6b-bf68-03834735566f tempest-TaggedBootDevicesTest_v242-1317742003 None] Deleted volume successfully. Apr 17 15:27:50 user cinder-volume[84696]: DEBUG cinder.coordination [None req-43b6f4d5-2620-4b6b-bf68-03834735566f tempest-TaggedBootDevicesTest_v242-1317742003 None] Lock "cinder-18475864-c202-4937-b18f-cc3e16c6d89f-delete_volume" released by "delete_volume" :: held 5.129s {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:207}} Apr 17 15:27:51 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-122fd13e-6b37-4630-a7df-3990d5d9f41d tempest-TaggedBootDevicesTest_v242-1317742003 None] Geting {'code': 200, 'error': None, 'data': {'results': 1, 'entries': [{'referenced': '16285696', 'userrefs': '0', 'primarycache': 'all', 'creation': '2023-04-17 17:47:42', 'volsize': '1073741824', 'createtxg': '187924', 'guid': '6717454584025623929', 'compressratio': '1.17', 'rootcontext': 'none', 'encryption': 'off', 'defcontext': 'none', 'written': '16285696', 'type': 'snapshot', 'secondarycache': 'all', 'used': '307200', 'refcompressratio': '1.17', 'fscontext': 'none', 'clones': 'Pool-0/v_006f20ae-5306-44fe-afdf-e433f79284e8', 'objsetid': '121585', 'name': 'v_006f20ae-5306-44fe-afdf-e433f79284e8', 'defer_destroy': 'off', 'san:volume_id': 'a29eef3759db89b1a074dbd4c6626ee4dc29a3ab', 'mlslabel': 'none', 'logicalreferenced': '19042304', 'context': 'none'}]}} from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_c74b7878-4936-4015-8b63-142923947310/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:27:51 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-122fd13e-6b37-4630-a7df-3990d5d9f41d tempest-TaggedBootDevicesTest_v242-1317742003 None] Sending pool request PUT to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_c74b7878-4936-4015-8b63-142923947310 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:27:51 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-122fd13e-6b37-4630-a7df-3990d5d9f41d tempest-TaggedBootDevicesTest_v242-1317742003 None] Sending PUT to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_c74b7878-4936-4015-8b63-142923947310 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:27:51 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-5346a9d0-f242-43c9-a311-93d0458ac4a7 req-cf528a18-6d05-4f1a-89ad-5074a53f0098 tempest-ServerStableDeviceRescueTest-869290889 None] Geting {'code': 201, 'error': None, 'data': {'active': True, 'allow_ip': [], 'conflicted': False, 'deny_ip': [], 'incoming_users_active': True, 'name': 'iqn.2010-10.org.openstack:32819ec1-2236-465f-a1c7-ee31aa4a0911', 'outgoing_user': None}} from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:27:51 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [req-5346a9d0-f242-43c9-a311-93d0458ac4a7 req-cf528a18-6d05-4f1a-89ad-5074a53f0098 tempest-ServerStableDeviceRescueTest-869290889 None] atach volume v_32819ec1-2236-465f-a1c7-ee31aa4a0911 to target iqn.2010-10.org.openstack:32819ec1-2236-465f-a1c7-ee31aa4a0911 {{(pid=84696) attach_target_vol /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:576}} Apr 17 15:27:51 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-5346a9d0-f242-43c9-a311-93d0458ac4a7 req-cf528a18-6d05-4f1a-89ad-5074a53f0098 tempest-ServerStableDeviceRescueTest-869290889 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:32819ec1-2236-465f-a1c7-ee31aa4a0911/luns {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:27:51 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-5346a9d0-f242-43c9-a311-93d0458ac4a7 req-cf528a18-6d05-4f1a-89ad-5074a53f0098 tempest-ServerStableDeviceRescueTest-869290889 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:32819ec1-2236-465f-a1c7-ee31aa4a0911/luns {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:27:53 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-5346a9d0-f242-43c9-a311-93d0458ac4a7 req-cf528a18-6d05-4f1a-89ad-5074a53f0098 tempest-ServerStableDeviceRescueTest-869290889 None] Geting {'code': 201, 'error': None, 'data': {'block_size': 512, 'device_handler': 'vdisk_blockio', 'eui': '6638333433313139', 'lun': 0, 'mode': 'wt', 'name': 'v_32819ec1-2236-465f-a1c7-ee31aa4a0911', 'prod_id': 'Storage', 'scsi_id': 'f834311982a3f3d2'}} from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:32819ec1-2236-465f-a1c7-ee31aa4a0911/luns {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:27:53 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [req-5346a9d0-f242-43c9-a311-93d0458ac4a7 req-cf528a18-6d05-4f1a-89ad-5074a53f0098 tempest-ServerStableDeviceRescueTest-869290889 None] add credentails to target iqn.2010-10.org.openstack:32819ec1-2236-465f-a1c7-ee31aa4a0911 {{(pid=84696) create_target_user /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:474}} Apr 17 15:27:53 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-5346a9d0-f242-43c9-a311-93d0458ac4a7 req-cf528a18-6d05-4f1a-89ad-5074a53f0098 tempest-ServerStableDeviceRescueTest-869290889 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:32819ec1-2236-465f-a1c7-ee31aa4a0911/incoming-users {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:27:53 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-5346a9d0-f242-43c9-a311-93d0458ac4a7 req-cf528a18-6d05-4f1a-89ad-5074a53f0098 tempest-ServerStableDeviceRescueTest-869290889 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:32819ec1-2236-465f-a1c7-ee31aa4a0911/incoming-users {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:27:53 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-5346a9d0-f242-43c9-a311-93d0458ac4a7 req-cf528a18-6d05-4f1a-89ad-5074a53f0098 tempest-ServerStableDeviceRescueTest-869290889 None] Geting {'code': 201, 'error': None, 'data': {'name': 'CmhLYVJy'}} from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:32819ec1-2236-465f-a1c7-ee31aa4a0911/incoming-users {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:27:53 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-5346a9d0-f242-43c9-a311-93d0458ac4a7 req-cf528a18-6d05-4f1a-89ad-5074a53f0098 tempest-ServerStableDeviceRescueTest-869290889 None] initialize_connection for 32819ec1-2236-465f-a1c7-ee31aa4a0911 10.0.0.210. {{(pid=84696) initialize_connection /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:1018}} Apr 17 15:27:53 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-5346a9d0-f242-43c9-a311-93d0458ac4a7 req-cf528a18-6d05-4f1a-89ad-5074a53f0098 tempest-ServerStableDeviceRescueTest-869290889 None] Connection info returned from driver {'target_iqn': 'iqn.2010-10.org.openstack:32819ec1-2236-465f-a1c7-ee31aa4a0911', 'target_portal': '172.16.0.220:3260', 'target_discovered': False, 'auth_method': 'CHAP', 'auth_username': 'CmhLYVJy', 'auth_password': '***', 'target_lun': 0, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False, 'driver_volume_type': 'iscsi', 'attachment_id': 'fb44dce3-1af2-43cf-95fa-68ff027e0699'} {{(pid=84696) _connection_create /opt/stack/cinder/cinder/volume/manager.py:4875}} Apr 17 15:27:53 user cinder-volume[84696]: INFO cinder.volume.manager [req-5346a9d0-f242-43c9-a311-93d0458ac4a7 req-cf528a18-6d05-4f1a-89ad-5074a53f0098 tempest-ServerStableDeviceRescueTest-869290889 None] attachment_update completed successfully. Apr 17 15:27:53 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-b474c39e-ef83-4b0a-868f-b4857ff3294d req-1c623693-99db-4bf7-964a-a8a3505d21f4 tempest-AttachVolumeShelveTestJSON-1745617049 None] Geting {'code': 201, 'error': None, 'data': {'active': True, 'allow_ip': [], 'conflicted': False, 'deny_ip': [], 'incoming_users_active': True, 'name': 'iqn.2010-10.org.openstack:13bf568b-585c-4556-ab78-e9930ad50214', 'outgoing_user': None}} from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:27:53 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [req-b474c39e-ef83-4b0a-868f-b4857ff3294d req-1c623693-99db-4bf7-964a-a8a3505d21f4 tempest-AttachVolumeShelveTestJSON-1745617049 None] atach volume v_13bf568b-585c-4556-ab78-e9930ad50214 to target iqn.2010-10.org.openstack:13bf568b-585c-4556-ab78-e9930ad50214 {{(pid=84696) attach_target_vol /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:576}} Apr 17 15:27:53 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-b474c39e-ef83-4b0a-868f-b4857ff3294d req-1c623693-99db-4bf7-964a-a8a3505d21f4 tempest-AttachVolumeShelveTestJSON-1745617049 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:13bf568b-585c-4556-ab78-e9930ad50214/luns {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:27:53 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-b474c39e-ef83-4b0a-868f-b4857ff3294d req-1c623693-99db-4bf7-964a-a8a3505d21f4 tempest-AttachVolumeShelveTestJSON-1745617049 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:13bf568b-585c-4556-ab78-e9930ad50214/luns {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:27:54 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-122fd13e-6b37-4630-a7df-3990d5d9f41d tempest-TaggedBootDevicesTest_v242-1317742003 None] Geting {'code': 201, 'error': None, 'data': None} from PUT to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_c74b7878-4936-4015-8b63-142923947310 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:27:54 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-122fd13e-6b37-4630-a7df-3990d5d9f41d tempest-TaggedBootDevicesTest_v242-1317742003 None] volume v_c74b7878-4936-4015-8b63-142923947310 updated {{(pid=84696) modify_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:267}} Apr 17 15:27:54 user cinder-volume[84696]: DEBUG cinder.quota [None req-122fd13e-6b37-4630-a7df-3990d5d9f41d tempest-TaggedBootDevicesTest_v242-1317742003 None] Created reservations ['b59d461a-f52a-4524-9ce1-40d3a8a7f01c', '8ccff58d-5f7b-4957-a891-10bc70045636', '266cbb1a-3205-473d-8bb6-8040339c68f3', 'e317f36f-a9be-4e5a-908a-f5ed16dd22c8'] {{(pid=84696) reserve /opt/stack/cinder/cinder/quota.py:755}} Apr 17 15:27:54 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-6c844c62-e6e1-4124-88a9-31cd15b5f835 req-3b1a6d51-2d5f-41b1-8f4d-c8709ad2c060 tempest-ServerActionsTestJSON-645547919 None] terminate connection for 1072d05f-2ebb-4398-902c-902df25464e9 {{(pid=84696) terminate_connection /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:1032}} Apr 17 15:27:54 user cinder-volume[84696]: INFO cinder.volume.manager [req-6c844c62-e6e1-4124-88a9-31cd15b5f835 req-3b1a6d51-2d5f-41b1-8f4d-c8709ad2c060 tempest-ServerActionsTestJSON-645547919 None] Terminate volume connection completed successfully. Apr 17 15:27:54 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-6c844c62-e6e1-4124-88a9-31cd15b5f835 req-3b1a6d51-2d5f-41b1-8f4d-c8709ad2c060 tempest-ServerActionsTestJSON-645547919 None] Deleting attachment 2cb8df32-9487-4e02-8868-35c8aefceb87. {{(pid=84696) attachment_delete /opt/stack/cinder/cinder/volume/manager.py:5006}} Apr 17 15:27:54 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-6c844c62-e6e1-4124-88a9-31cd15b5f835 req-3b1a6d51-2d5f-41b1-8f4d-c8709ad2c060 tempest-ServerActionsTestJSON-645547919 None] remove_export for volume: 1072d05f-2ebb-4398-902c-902df25464e9. {{(pid=84696) remove_export /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:718}} Apr 17 15:27:54 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-6c844c62-e6e1-4124-88a9-31cd15b5f835 req-3b1a6d51-2d5f-41b1-8f4d-c8709ad2c060 tempest-ServerActionsTestJSON-645547919 None] remove export {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:912}} Apr 17 15:27:54 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-6c844c62-e6e1-4124-88a9-31cd15b5f835 req-3b1a6d51-2d5f-41b1-8f4d-c8709ad2c060 tempest-ServerActionsTestJSON-645547919 None] detach volume:Volume(_name_id=None,admin_metadata=,attach_status='attached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:27:39Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-ServerActionsTestJSON-volume-1520907984',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=1072d05f-2ebb-4398-902c-902df25464e9,launched_at=2023-04-17T15:27:40Z,metadata=,migration_status=None,multiattach=False,previous_status=None,project_id='cb5f6b5ed2fc43b69c47e1fc18646545',provider_auth='CHAP NKISaVyG nHFlRwNr8b5KS2',provider_geometry=None,provider_id=None,provider_location='172.16.0.220:3260,1 iqn.2010-10.org.openstack:1072d05f-2ebb-4398-902c-902df25464e9 0',replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:27:39Z,service_uuid='e628352e-c4cf-40eb-8389-cce40663026a',shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='in-use',terminated_at=None,updated_at=2023-04-17T15:27:51Z,use_quota=True,user_id='638a4f535e9740fc9d810639a5eb24d7',volume_attachment=,volume_type=,volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c) from target:iqn.2010-10.org.openstack:1072d05f-2ebb-4398-902c-902df25464e9. {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:913}} Apr 17 15:27:54 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [req-6c844c62-e6e1-4124-88a9-31cd15b5f835 req-3b1a6d51-2d5f-41b1-8f4d-c8709ad2c060 tempest-ServerActionsTestJSON-645547919 None] detach volume v_1072d05f-2ebb-4398-902c-902df25464e9 from target iqn.2010-10.org.openstack:1072d05f-2ebb-4398-902c-902df25464e9 {{(pid=84696) detach_target_vol /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:606}} Apr 17 15:27:54 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-6c844c62-e6e1-4124-88a9-31cd15b5f835 req-3b1a6d51-2d5f-41b1-8f4d-c8709ad2c060 tempest-ServerActionsTestJSON-645547919 None] Sending pool request DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:1072d05f-2ebb-4398-902c-902df25464e9/luns/v_1072d05f-2ebb-4398-902c-902df25464e9 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:27:54 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-6c844c62-e6e1-4124-88a9-31cd15b5f835 req-3b1a6d51-2d5f-41b1-8f4d-c8709ad2c060 tempest-ServerActionsTestJSON-645547919 None] Sending DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:1072d05f-2ebb-4398-902c-902df25464e9/luns/v_1072d05f-2ebb-4398-902c-902df25464e9 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:27:54 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-122fd13e-6b37-4630-a7df-3990d5d9f41d tempest-TaggedBootDevicesTest_v242-1317742003 None] Updating volume stats {{(pid=84696) _update_volume_stats /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:724}} Apr 17 15:27:54 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-122fd13e-6b37-4630-a7df-3990d5d9f41d tempest-TaggedBootDevicesTest_v242-1317742003 None] Get pool Pool-0 fsprops {{(pid=84696) get_pool_stats /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:894}} Apr 17 15:27:54 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-122fd13e-6b37-4630-a7df-3990d5d9f41d tempest-TaggedBootDevicesTest_v242-1317742003 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:27:54 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-122fd13e-6b37-4630-a7df-3990d5d9f41d tempest-TaggedBootDevicesTest_v242-1317742003 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:27:54 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-6c844c62-e6e1-4124-88a9-31cd15b5f835 req-3b1a6d51-2d5f-41b1-8f4d-c8709ad2c060 tempest-ServerActionsTestJSON-645547919 None] Geting {'code': 204} from DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:1072d05f-2ebb-4398-902c-902df25464e9/luns/v_1072d05f-2ebb-4398-902c-902df25464e9 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:27:54 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-6c844c62-e6e1-4124-88a9-31cd15b5f835 req-3b1a6d51-2d5f-41b1-8f4d-c8709ad2c060 tempest-ServerActionsTestJSON-645547919 None] delete target: iqn.2010-10.org.openstack:1072d05f-2ebb-4398-902c-902df25464e9 {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:930}} Apr 17 15:27:54 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [req-6c844c62-e6e1-4124-88a9-31cd15b5f835 req-3b1a6d51-2d5f-41b1-8f4d-c8709ad2c060 tempest-ServerActionsTestJSON-645547919 None] delete target iqn.2010-10.org.openstack:1072d05f-2ebb-4398-902c-902df25464e9 {{(pid=84696) delete_target /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:442}} Apr 17 15:27:54 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-6c844c62-e6e1-4124-88a9-31cd15b5f835 req-3b1a6d51-2d5f-41b1-8f4d-c8709ad2c060 tempest-ServerActionsTestJSON-645547919 None] Sending pool request DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:1072d05f-2ebb-4398-902c-902df25464e9 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:27:54 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-6c844c62-e6e1-4124-88a9-31cd15b5f835 req-3b1a6d51-2d5f-41b1-8f4d-c8709ad2c060 tempest-ServerActionsTestJSON-645547919 None] Sending DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:1072d05f-2ebb-4398-902c-902df25464e9 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:27:55 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-6c844c62-e6e1-4124-88a9-31cd15b5f835 req-3b1a6d51-2d5f-41b1-8f4d-c8709ad2c060 tempest-ServerActionsTestJSON-645547919 None] Geting {'code': 204} from DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:1072d05f-2ebb-4398-902c-902df25464e9 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:27:55 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [req-6c844c62-e6e1-4124-88a9-31cd15b5f835 req-3b1a6d51-2d5f-41b1-8f4d-c8709ad2c060 tempest-ServerActionsTestJSON-645547919 None] target iqn.2010-10.org.openstack:1072d05f-2ebb-4398-902c-902df25464e9 deleted {{(pid=84696) delete_target /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:447}} Apr 17 15:27:55 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-122fd13e-6b37-4630-a7df-3990d5d9f41d tempest-TaggedBootDevicesTest_v242-1317742003 None] Geting {'code': 200, 'error': None, 'data': {'available': '113889292288', 'status': 30, 'import_status': {'import_steps': {'san_setup': True, 'vip_setup': True, 'zfs_import': True, 'mount_system_volume': True, 'nas_setup': True}, 'import_successful': True}, 'scan': None, 'encryption': {'enabled': False}, 'deduplication_rate': '1.00', 'sysvol_upgrade': 'not_needed', 'vdevs': [{'name': 'wwn-0x6000c296b793bf526a107a9d823e947a', 'iostats': {'read': '0', 'write': '0', 'chksum': '0'}, 'disks': [{'origin': 'local', 'slot': '', 'led': 'off', 'name': 'sdn', 'iostats': {'read': '0', 'write': '0', 'chksum': '0'}, 'alias': '', 'health': 'ONLINE', 'sn': '6000c296b793bf526a107a9d823e947a', 'trim_data': {'status': 'trim_unsupported', 'progress': None, 'trim_time': None}, 'path': 'pci-0000:03:00.0-scsi-0:0:14:0', 'model': 'Virtual disk', 'id': 'wwn-0x6000c296b793bf526a107a9d823e947a', 'size': 128849018880}], 'health': 'ONLINE', 'vdev_replacings': [], 'vdev_spares': [], 'type': ''}], 'id': '14181317334440137330', 'health': 'ONLINE', 'iostats': {'read': '0', 'write': '0', 'chksum': '0'}, 'operation': 'none', 'size': '127775277056', 'autotrim': False, 'name': 'Pool-0'}} from GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:27:55 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-122fd13e-6b37-4630-a7df-3990d5d9f41d tempest-TaggedBootDevicesTest_v242-1317742003 None] Total capacity: 119, Free 106. {{(pid=84696) _update_volume_stats /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:757}} Apr 17 15:27:55 user cinder-volume[84696]: DEBUG cinder.manager [None req-122fd13e-6b37-4630-a7df-3990d5d9f41d tempest-TaggedBootDevicesTest_v242-1317742003 None] Notifying Schedulers of capabilities ... {{(pid=84696) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 17 15:27:55 user cinder-volume[84696]: INFO cinder.volume.manager [None req-122fd13e-6b37-4630-a7df-3990d5d9f41d tempest-TaggedBootDevicesTest_v242-1317742003 None] Deleted volume successfully. Apr 17 15:27:55 user cinder-volume[84696]: DEBUG cinder.coordination [None req-122fd13e-6b37-4630-a7df-3990d5d9f41d tempest-TaggedBootDevicesTest_v242-1317742003 None] Lock "cinder-c74b7878-4936-4015-8b63-142923947310-delete_volume" released by "delete_volume" :: held 4.787s {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:207}} Apr 17 15:27:55 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-b474c39e-ef83-4b0a-868f-b4857ff3294d req-1c623693-99db-4bf7-964a-a8a3505d21f4 tempest-AttachVolumeShelveTestJSON-1745617049 None] Geting {'code': 201, 'error': None, 'data': {'block_size': 512, 'device_handler': 'vdisk_blockio', 'eui': '6431616337386434', 'lun': 0, 'mode': 'wt', 'name': 'v_13bf568b-585c-4556-ab78-e9930ad50214', 'prod_id': 'Storage', 'scsi_id': 'd1ac78d408011c57'}} from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:13bf568b-585c-4556-ab78-e9930ad50214/luns {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:27:55 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [req-b474c39e-ef83-4b0a-868f-b4857ff3294d req-1c623693-99db-4bf7-964a-a8a3505d21f4 tempest-AttachVolumeShelveTestJSON-1745617049 None] add credentails to target iqn.2010-10.org.openstack:13bf568b-585c-4556-ab78-e9930ad50214 {{(pid=84696) create_target_user /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:474}} Apr 17 15:27:55 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-b474c39e-ef83-4b0a-868f-b4857ff3294d req-1c623693-99db-4bf7-964a-a8a3505d21f4 tempest-AttachVolumeShelveTestJSON-1745617049 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:13bf568b-585c-4556-ab78-e9930ad50214/incoming-users {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:27:55 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-b474c39e-ef83-4b0a-868f-b4857ff3294d req-1c623693-99db-4bf7-964a-a8a3505d21f4 tempest-AttachVolumeShelveTestJSON-1745617049 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:13bf568b-585c-4556-ab78-e9930ad50214/incoming-users {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:27:56 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-b474c39e-ef83-4b0a-868f-b4857ff3294d req-1c623693-99db-4bf7-964a-a8a3505d21f4 tempest-AttachVolumeShelveTestJSON-1745617049 None] Geting {'code': 201, 'error': None, 'data': {'name': 'cajgFQvq'}} from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:13bf568b-585c-4556-ab78-e9930ad50214/incoming-users {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:27:56 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=65,cinder:INSERT=5,cinder:UPDATE=27,cinder:DELETE=3 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:27:56 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-b474c39e-ef83-4b0a-868f-b4857ff3294d req-1c623693-99db-4bf7-964a-a8a3505d21f4 tempest-AttachVolumeShelveTestJSON-1745617049 None] initialize_connection for 13bf568b-585c-4556-ab78-e9930ad50214 10.0.0.210. {{(pid=84696) initialize_connection /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:1018}} Apr 17 15:27:56 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-b474c39e-ef83-4b0a-868f-b4857ff3294d req-1c623693-99db-4bf7-964a-a8a3505d21f4 tempest-AttachVolumeShelveTestJSON-1745617049 None] Connection info returned from driver {'target_iqn': 'iqn.2010-10.org.openstack:13bf568b-585c-4556-ab78-e9930ad50214', 'target_portal': '172.16.0.220:3260', 'target_discovered': False, 'auth_method': 'CHAP', 'auth_username': 'cajgFQvq', 'auth_password': '***', 'target_lun': 0, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False, 'driver_volume_type': 'iscsi', 'attachment_id': '1634a7d0-3313-448e-9888-8de015f268af'} {{(pid=84696) _connection_create /opt/stack/cinder/cinder/volume/manager.py:4875}} Apr 17 15:27:56 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-6c844c62-e6e1-4124-88a9-31cd15b5f835 req-1c1e3e83-0474-452f-a676-79a28b3edcdd tempest-ServerActionsTestJSON-645547919 None] create export for volume: 1072d05f-2ebb-4398-902c-902df25464e9. {{(pid=84696) create_export /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:699}} Apr 17 15:27:56 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-6c844c62-e6e1-4124-88a9-31cd15b5f835 req-1c1e3e83-0474-452f-a676-79a28b3edcdd tempest-ServerActionsTestJSON-645547919 None] ensure volume 1072d05f-2ebb-4398-902c-902df25464e9 assigned to a proper target {{(pid=84696) _ensure_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:866}} Apr 17 15:27:56 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [req-6c844c62-e6e1-4124-88a9-31cd15b5f835 req-1c1e3e83-0474-452f-a676-79a28b3edcdd tempest-ServerActionsTestJSON-645547919 None] check if targe iqn.2010-10.org.openstack:1072d05f-2ebb-4398-902c-902df25464e9 exists {{(pid=84696) is_target /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:376}} Apr 17 15:27:56 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-6c844c62-e6e1-4124-88a9-31cd15b5f835 req-1c1e3e83-0474-452f-a676-79a28b3edcdd tempest-ServerActionsTestJSON-645547919 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:1072d05f-2ebb-4398-902c-902df25464e9 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:27:56 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-6c844c62-e6e1-4124-88a9-31cd15b5f835 req-1c1e3e83-0474-452f-a676-79a28b3edcdd tempest-ServerActionsTestJSON-645547919 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:1072d05f-2ebb-4398-902c-902df25464e9 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:27:56 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-6c844c62-e6e1-4124-88a9-31cd15b5f835 req-1c1e3e83-0474-452f-a676-79a28b3edcdd tempest-ServerActionsTestJSON-645547919 None] Geting {'code': 404, 'error': {'class': 'opene.exceptions.ItemNotFoundError', 'message': 'Target iqn.2010-10.org.openstack:1072d05f-2ebb-4398-902c-902df25464e9 not exists.', 'url': 'http://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:1072d05f-2ebb-4398-902c-902df25464e9'}, 'data': None} from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:1072d05f-2ebb-4398-902c-902df25464e9 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:27:56 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-6c844c62-e6e1-4124-88a9-31cd15b5f835 req-1c1e3e83-0474-452f-a676-79a28b3edcdd tempest-ServerActionsTestJSON-645547919 None] create target and attach volume 1072d05f-2ebb-4398-902c-902df25464e9 to it {{(pid=84696) _create_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:837}} Apr 17 15:27:56 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [req-6c844c62-e6e1-4124-88a9-31cd15b5f835 req-1c1e3e83-0474-452f-a676-79a28b3edcdd tempest-ServerActionsTestJSON-645547919 None] create target iqn.2010-10.org.openstack:1072d05f-2ebb-4398-902c-902df25464e9 {{(pid=84696) create_target /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:410}} Apr 17 15:27:56 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-6c844c62-e6e1-4124-88a9-31cd15b5f835 req-1c1e3e83-0474-452f-a676-79a28b3edcdd tempest-ServerActionsTestJSON-645547919 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:27:56 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-6c844c62-e6e1-4124-88a9-31cd15b5f835 req-1c1e3e83-0474-452f-a676-79a28b3edcdd tempest-ServerActionsTestJSON-645547919 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:27:56 user cinder-volume[84696]: INFO cinder.volume.manager [req-b474c39e-ef83-4b0a-868f-b4857ff3294d req-1c623693-99db-4bf7-964a-a8a3505d21f4 tempest-AttachVolumeShelveTestJSON-1745617049 None] attachment_update completed successfully. Apr 17 15:28:00 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-6c844c62-e6e1-4124-88a9-31cd15b5f835 req-1c1e3e83-0474-452f-a676-79a28b3edcdd tempest-ServerActionsTestJSON-645547919 None] Geting {'code': 201, 'error': None, 'data': {'active': True, 'allow_ip': [], 'conflicted': False, 'deny_ip': [], 'incoming_users_active': True, 'name': 'iqn.2010-10.org.openstack:1072d05f-2ebb-4398-902c-902df25464e9', 'outgoing_user': None}} from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:28:00 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [req-6c844c62-e6e1-4124-88a9-31cd15b5f835 req-1c1e3e83-0474-452f-a676-79a28b3edcdd tempest-ServerActionsTestJSON-645547919 None] atach volume v_1072d05f-2ebb-4398-902c-902df25464e9 to target iqn.2010-10.org.openstack:1072d05f-2ebb-4398-902c-902df25464e9 {{(pid=84696) attach_target_vol /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:576}} Apr 17 15:28:00 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-6c844c62-e6e1-4124-88a9-31cd15b5f835 req-1c1e3e83-0474-452f-a676-79a28b3edcdd tempest-ServerActionsTestJSON-645547919 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:1072d05f-2ebb-4398-902c-902df25464e9/luns {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:28:00 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-6c844c62-e6e1-4124-88a9-31cd15b5f835 req-1c1e3e83-0474-452f-a676-79a28b3edcdd tempest-ServerActionsTestJSON-645547919 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:1072d05f-2ebb-4398-902c-902df25464e9/luns {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:28:01 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-6c844c62-e6e1-4124-88a9-31cd15b5f835 req-1c1e3e83-0474-452f-a676-79a28b3edcdd tempest-ServerActionsTestJSON-645547919 None] Geting {'code': 201, 'error': None, 'data': {'block_size': 512, 'device_handler': 'vdisk_blockio', 'eui': '3933326166393635', 'lun': 0, 'mode': 'wt', 'name': 'v_1072d05f-2ebb-4398-902c-902df25464e9', 'prod_id': 'Storage', 'scsi_id': '932af96517a71872'}} from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:1072d05f-2ebb-4398-902c-902df25464e9/luns {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:28:01 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [req-6c844c62-e6e1-4124-88a9-31cd15b5f835 req-1c1e3e83-0474-452f-a676-79a28b3edcdd tempest-ServerActionsTestJSON-645547919 None] add credentails to target iqn.2010-10.org.openstack:1072d05f-2ebb-4398-902c-902df25464e9 {{(pid=84696) create_target_user /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:474}} Apr 17 15:28:01 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-6c844c62-e6e1-4124-88a9-31cd15b5f835 req-1c1e3e83-0474-452f-a676-79a28b3edcdd tempest-ServerActionsTestJSON-645547919 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:1072d05f-2ebb-4398-902c-902df25464e9/incoming-users {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:28:01 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-6c844c62-e6e1-4124-88a9-31cd15b5f835 req-1c1e3e83-0474-452f-a676-79a28b3edcdd tempest-ServerActionsTestJSON-645547919 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:1072d05f-2ebb-4398-902c-902df25464e9/incoming-users {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:28:02 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-6c844c62-e6e1-4124-88a9-31cd15b5f835 req-1c1e3e83-0474-452f-a676-79a28b3edcdd tempest-ServerActionsTestJSON-645547919 None] Geting {'code': 201, 'error': None, 'data': {'name': 'NKISaVyG'}} from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:1072d05f-2ebb-4398-902c-902df25464e9/incoming-users {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:28:02 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-6c844c62-e6e1-4124-88a9-31cd15b5f835 req-1c1e3e83-0474-452f-a676-79a28b3edcdd tempest-ServerActionsTestJSON-645547919 None] initialize_connection for 1072d05f-2ebb-4398-902c-902df25464e9 10.0.0.210. {{(pid=84696) initialize_connection /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:1018}} Apr 17 15:28:02 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=64,cinder:UPDATE=29,cinder:DELETE=2,cinder:INSERT=5 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:28:02 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-6c844c62-e6e1-4124-88a9-31cd15b5f835 req-1c1e3e83-0474-452f-a676-79a28b3edcdd tempest-ServerActionsTestJSON-645547919 None] Connection info returned from driver {'target_iqn': 'iqn.2010-10.org.openstack:1072d05f-2ebb-4398-902c-902df25464e9', 'target_portal': '172.16.0.220:3260', 'target_discovered': False, 'auth_method': 'CHAP', 'auth_username': 'NKISaVyG', 'auth_password': '***', 'target_lun': 0, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False, 'driver_volume_type': 'iscsi', 'attachment_id': 'a68f5fc4-dea3-48a4-bdb0-1d89db356c29'} {{(pid=84696) _connection_create /opt/stack/cinder/cinder/volume/manager.py:4875}} Apr 17 15:28:02 user cinder-volume[84696]: INFO cinder.volume.manager [req-6c844c62-e6e1-4124-88a9-31cd15b5f835 req-1c1e3e83-0474-452f-a676-79a28b3edcdd tempest-ServerActionsTestJSON-645547919 None] attachment_update completed successfully. Apr 17 15:28:12 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Volume reschedule parameters: True retry: {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 15:28:12 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Flow 'volume_create_manager' (5996bb83-4d49-4e7f-98f8-dbad127cce17) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:28:12 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (91b35ce4-f3c0-43ac-a18f-1a39667c87f7) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:28:12 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (91b35ce4-f3c0-43ac-a18f-1a39667c87f7) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:28:12Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesActionsTest-Volume-620204537',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=62709a81-268b-4227-ba4a-070034f36414,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='a8b97ff6ed6047f0a33e74684a1c162b',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:28:13Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:28:13Z,use_quota=True,user_id='c993ceeb4a5c49eea14057e2125518ad',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:28:12 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (35ddcbc2-13dd-4fbe-8faa-513f4b35e511) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:28:12 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (35ddcbc2-13dd-4fbe-8faa-513f4b35e511) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:28:12 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (6bf018c4-d42e-448d-8618-f7ed9f876d9e) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:28:12 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (6bf018c4-d42e-448d-8618-f7ed9f876d9e) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': '62709a81-268b-4227-ba4a-070034f36414', 'volume_name': 'volume-62709a81-268b-4227-ba4a-070034f36414', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:28:12 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (b5a33e7e-cb4d-4acb-811f-cd44ece1b863) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:28:12 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (b5a33e7e-cb4d-4acb-811f-cd44ece1b863) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:28:12 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (0407bfe2-385a-4571-9294-c3366a4fa775) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:28:12 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Volume 62709a81-268b-4227-ba4a-070034f36414: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-62709a81-268b-4227-ba4a-070034f36414', 'volume_size': 1} Apr 17 15:28:12 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] creating volume v_62709a81-268b-4227-ba4a-070034f36414. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 15:28:12 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] create volume {'name': 'v_62709a81-268b-4227-ba4a-070034f36414', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 15:28:12 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:28:12 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:28:12 user cinder-volume[84696]: DEBUG cinder.utils [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 0.003(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:28:12 user cinder-volume[84696]: DEBUG cinder.utils [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:28:14 user cinder-volume[84696]: DEBUG cinder.utils [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.008(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:28:14 user cinder-volume[84696]: DEBUG cinder.utils [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:28:15 user cinder-volume[84696]: DEBUG oslo_service.periodic_task [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Running periodic task VolumeManager.publish_service_capabilities {{(pid=84696) run_periodic_tasks /usr/local/lib/python3.10/dist-packages/oslo_service/periodic_task.py:210}} Apr 17 15:28:15 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Updating volume stats {{(pid=84696) _update_volume_stats /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:724}} Apr 17 15:28:15 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Get pool Pool-0 fsprops {{(pid=84696) get_pool_stats /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:894}} Apr 17 15:28:15 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:28:15 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:28:15 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 0.005(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:28:15 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:28:17 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.009(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:28:17 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:28:18 user cinder-volume[84696]: DEBUG cinder.utils [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.014(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:28:18 user cinder-volume[84696]: DEBUG cinder.utils [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:28:20 user cinder-volume[84696]: DEBUG cinder.coordination [None req-d0984215-eaf6-4942-b2a1-997c1c6518b5 tempest-TaggedAttachmentsTest-1309189231 None] Acquiring lock "cinder-3c05f4f0-17a8-4f4c-9987-1fcb502ffd3a-delete_volume" by "delete_volume" {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:192}} Apr 17 15:28:20 user cinder-volume[84696]: DEBUG cinder.coordination [None req-d0984215-eaf6-4942-b2a1-997c1c6518b5 tempest-TaggedAttachmentsTest-1309189231 None] Lock "cinder-3c05f4f0-17a8-4f4c-9987-1fcb502ffd3a-delete_volume" acquired by "delete_volume" :: waited 0.011s {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:196}} Apr 17 15:28:21 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-d0984215-eaf6-4942-b2a1-997c1c6518b5 tempest-TaggedAttachmentsTest-1309189231 None] remove_export for volume: 3c05f4f0-17a8-4f4c-9987-1fcb502ffd3a. {{(pid=84696) remove_export /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:718}} Apr 17 15:28:21 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-d0984215-eaf6-4942-b2a1-997c1c6518b5 tempest-TaggedAttachmentsTest-1309189231 None] remove export {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:912}} Apr 17 15:28:21 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-d0984215-eaf6-4942-b2a1-997c1c6518b5 tempest-TaggedAttachmentsTest-1309189231 None] detach volume:Volume(_name_id=None,admin_metadata={},attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:24:14Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TaggedAttachmentsTest-volume-2098521363',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=3c05f4f0-17a8-4f4c-9987-1fcb502ffd3a,launched_at=2023-04-17T15:24:17Z,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='9dfa73323c3c4c689f4012505d8b4833',provider_auth='CHAP vjBFoMiS JvaW8BYGPNwxzR',provider_geometry=None,provider_id=None,provider_location='172.16.0.220:3260,1 iqn.2010-10.org.openstack:3c05f4f0-17a8-4f4c-9987-1fcb502ffd3a 0',replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:24:15Z,service_uuid='e628352e-c4cf-40eb-8389-cce40663026a',shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='deleting',terminated_at=2023-04-17T15:28:21Z,updated_at=2023-04-17T15:28:21Z,use_quota=True,user_id='f081ecb75ec14053b68d74cacf61c802',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c) from target:iqn.2010-10.org.openstack:3c05f4f0-17a8-4f4c-9987-1fcb502ffd3a. {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:913}} Apr 17 15:28:21 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-d0984215-eaf6-4942-b2a1-997c1c6518b5 tempest-TaggedAttachmentsTest-1309189231 None] detach volume v_3c05f4f0-17a8-4f4c-9987-1fcb502ffd3a from target iqn.2010-10.org.openstack:3c05f4f0-17a8-4f4c-9987-1fcb502ffd3a {{(pid=84696) detach_target_vol /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:606}} Apr 17 15:28:21 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-d0984215-eaf6-4942-b2a1-997c1c6518b5 tempest-TaggedAttachmentsTest-1309189231 None] Sending pool request DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:3c05f4f0-17a8-4f4c-9987-1fcb502ffd3a/luns/v_3c05f4f0-17a8-4f4c-9987-1fcb502ffd3a {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:28:21 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-d0984215-eaf6-4942-b2a1-997c1c6518b5 tempest-TaggedAttachmentsTest-1309189231 None] Sending DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:3c05f4f0-17a8-4f4c-9987-1fcb502ffd3a/luns/v_3c05f4f0-17a8-4f4c-9987-1fcb502ffd3a {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:28:21 user cinder-volume[84696]: DEBUG cinder.utils [None req-d0984215-eaf6-4942-b2a1-997c1c6518b5 tempest-TaggedAttachmentsTest-1309189231 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 0.002(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:28:21 user cinder-volume[84696]: DEBUG cinder.utils [None req-d0984215-eaf6-4942-b2a1-997c1c6518b5 tempest-TaggedAttachmentsTest-1309189231 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:3c05f4f0-17a8-4f4c-9987-1fcb502ffd3a/luns/v_3c05f4f0-17a8-4f4c-9987-1fcb502ffd3a (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:28:21 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.013(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:28:21 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:28:22 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Volume reschedule parameters: True retry: {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 15:28:22 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Flow 'volume_create_manager' (fa9685c1-a073-4d82-8bb0-ee053948ec78) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:28:22 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (bea28282-7b49-4c64-877a-2f9a3ec04a85) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:28:22 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (bea28282-7b49-4c64-877a-2f9a3ec04a85) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:28:22Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-SnapshotMetadataTestJSON-Volume-902610845',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=17f8d4f7-2dcd-4041-8d0b-5c8afebb75bf,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='a4653bf4d2914c8d956e86ad3405051d',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:28:22Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:28:22Z,use_quota=True,user_id='5598ef2700af47d9bcf6fadea4460cb2',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:28:22 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (ba0a21a5-fc9a-4815-9416-e53538e13902) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:28:22 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (ba0a21a5-fc9a-4815-9416-e53538e13902) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:28:22 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (400785c0-7281-49a4-a487-ca23510184ff) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:28:22 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (400785c0-7281-49a4-a487-ca23510184ff) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': '17f8d4f7-2dcd-4041-8d0b-5c8afebb75bf', 'volume_name': 'volume-17f8d4f7-2dcd-4041-8d0b-5c8afebb75bf', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:28:22 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (58cbf18b-a802-4594-9e2c-dbd6f30e4edd) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:28:22 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (58cbf18b-a802-4594-9e2c-dbd6f30e4edd) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:28:22 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (9d098b7c-98f7-4428-8290-d91cfbe936a7) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:28:22 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Volume 17f8d4f7-2dcd-4041-8d0b-5c8afebb75bf: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-17f8d4f7-2dcd-4041-8d0b-5c8afebb75bf', 'volume_size': 1} Apr 17 15:28:22 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] creating volume v_17f8d4f7-2dcd-4041-8d0b-5c8afebb75bf. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 15:28:22 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] create volume {'name': 'v_17f8d4f7-2dcd-4041-8d0b-5c8afebb75bf', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 15:28:22 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:28:22 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:28:22 user cinder-volume[84696]: DEBUG cinder.utils [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 0.003(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:28:22 user cinder-volume[84696]: DEBUG cinder.utils [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:28:23 user cinder-volume[84696]: DEBUG cinder.utils [None req-d0984215-eaf6-4942-b2a1-997c1c6518b5 tempest-TaggedAttachmentsTest-1309189231 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.006(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:28:23 user cinder-volume[84696]: DEBUG cinder.utils [None req-d0984215-eaf6-4942-b2a1-997c1c6518b5 tempest-TaggedAttachmentsTest-1309189231 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:3c05f4f0-17a8-4f4c-9987-1fcb502ffd3a/luns/v_3c05f4f0-17a8-4f4c-9987-1fcb502ffd3a (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:28:24 user cinder-volume[84696]: DEBUG cinder.utils [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.007(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:28:24 user cinder-volume[84696]: DEBUG cinder.utils [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:28:26 user cinder-volume[84696]: DEBUG cinder.utils [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 14.018(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:28:26 user cinder-volume[84696]: DEBUG cinder.utils [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:28:27 user cinder-volume[84696]: DEBUG cinder.utils [None req-d0984215-eaf6-4942-b2a1-997c1c6518b5 tempest-TaggedAttachmentsTest-1309189231 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.013(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:28:27 user cinder-volume[84696]: DEBUG cinder.utils [None req-d0984215-eaf6-4942-b2a1-997c1c6518b5 tempest-TaggedAttachmentsTest-1309189231 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:3c05f4f0-17a8-4f4c-9987-1fcb502ffd3a/luns/v_3c05f4f0-17a8-4f4c-9987-1fcb502ffd3a (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:28:28 user cinder-volume[84696]: DEBUG cinder.utils [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.011(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:28:28 user cinder-volume[84696]: DEBUG cinder.utils [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:28:29 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 14.016(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:28:29 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:28:33 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=14,cinder:UPDATE=2 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:28:33 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=38,cinder:UPDATE=9 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:28:35 user cinder-volume[84696]: DEBUG cinder.utils [None req-d0984215-eaf6-4942-b2a1-997c1c6518b5 tempest-TaggedAttachmentsTest-1309189231 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 14.018(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:28:35 user cinder-volume[84696]: DEBUG cinder.utils [None req-d0984215-eaf6-4942-b2a1-997c1c6518b5 tempest-TaggedAttachmentsTest-1309189231 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:3c05f4f0-17a8-4f4c-9987-1fcb502ffd3a/luns/v_3c05f4f0-17a8-4f4c-9987-1fcb502ffd3a (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:28:36 user cinder-volume[84696]: DEBUG cinder.utils [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 14.015(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:28:36 user cinder-volume[84696]: DEBUG cinder.utils [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:28:45 user cinder-volume[84696]: DEBUG cinder.coordination [None req-643b65b6-f16f-4041-b9cc-be64c983a38a tempest-TestVolumeSwap-1714109745 None] Acquiring lock "cinder-485ea70c-618d-4373-a78f-a460914fc095-delete_volume" by "delete_volume" {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:192}} Apr 17 15:28:45 user cinder-volume[84696]: DEBUG cinder.coordination [None req-643b65b6-f16f-4041-b9cc-be64c983a38a tempest-TestVolumeSwap-1714109745 None] Lock "cinder-485ea70c-618d-4373-a78f-a460914fc095-delete_volume" acquired by "delete_volume" :: waited 0.012s {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:196}} Apr 17 15:28:45 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-643b65b6-f16f-4041-b9cc-be64c983a38a tempest-TestVolumeSwap-1714109745 None] remove_export for volume: 485ea70c-618d-4373-a78f-a460914fc095. {{(pid=84696) remove_export /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:718}} Apr 17 15:28:45 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-643b65b6-f16f-4041-b9cc-be64c983a38a tempest-TestVolumeSwap-1714109745 None] remove export {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:912}} Apr 17 15:28:45 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-643b65b6-f16f-4041-b9cc-be64c983a38a tempest-TestVolumeSwap-1714109745 None] detach volume:Volume(_name_id=None,admin_metadata={},attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:24:11Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TestVolumeSwap-volume-2088850205',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=485ea70c-618d-4373-a78f-a460914fc095,launched_at=2023-04-17T15:24:14Z,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='00b574796d1942938ed2f348803fcf87',provider_auth='CHAP WBubQvJS XBNtnWHrPfMu3U',provider_geometry=None,provider_id=None,provider_location='172.16.0.220:3260,1 iqn.2010-10.org.openstack:485ea70c-618d-4373-a78f-a460914fc095 0',replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:24:12Z,service_uuid='e628352e-c4cf-40eb-8389-cce40663026a',shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='deleting',terminated_at=2023-04-17T15:28:45Z,updated_at=2023-04-17T15:28:45Z,use_quota=True,user_id='4c2bfcb4119e483ab1a41e3c598ee1b0',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c) from target:iqn.2010-10.org.openstack:485ea70c-618d-4373-a78f-a460914fc095. {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:913}} Apr 17 15:28:45 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-643b65b6-f16f-4041-b9cc-be64c983a38a tempest-TestVolumeSwap-1714109745 None] detach volume v_485ea70c-618d-4373-a78f-a460914fc095 from target iqn.2010-10.org.openstack:485ea70c-618d-4373-a78f-a460914fc095 {{(pid=84696) detach_target_vol /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:606}} Apr 17 15:28:45 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-643b65b6-f16f-4041-b9cc-be64c983a38a tempest-TestVolumeSwap-1714109745 None] Sending pool request DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:485ea70c-618d-4373-a78f-a460914fc095/luns/v_485ea70c-618d-4373-a78f-a460914fc095 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:28:45 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-643b65b6-f16f-4041-b9cc-be64c983a38a tempest-TestVolumeSwap-1714109745 None] Sending DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:485ea70c-618d-4373-a78f-a460914fc095/luns/v_485ea70c-618d-4373-a78f-a460914fc095 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:29:03 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=9,cinder:UPDATE=1 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:29:03 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=7,cinder:UPDATE=3 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:29:10 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Volume reschedule parameters: True retry: {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 15:29:10 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Flow 'volume_create_manager' (2c25ab92-a180-4932-b5b0-a57aae59fc6f) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:29:10 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (ddc5e7b4-61a3-411a-89c1-aaa92116a739) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:29:10 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (ddc5e7b4-61a3-411a-89c1-aaa92116a739) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:29:10Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesAdminNegativeTest-volume-1635380936',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=a4d96ab8-2072-4eff-b803-53b4f27d056f,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='b334c5bc33f143c484518d9bd0e750dd',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:29:10Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:29:10Z,use_quota=True,user_id='3dfadfd30c394b46abe75ec930007850',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:29:10 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (d9af24a3-9896-49dc-ade6-3b00092bae74) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:29:10 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (d9af24a3-9896-49dc-ade6-3b00092bae74) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:29:10 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (a41cf2d4-36d2-439f-b298-2e81f71ca70a) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:29:10 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (a41cf2d4-36d2-439f-b298-2e81f71ca70a) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': 'a4d96ab8-2072-4eff-b803-53b4f27d056f', 'volume_name': 'volume-a4d96ab8-2072-4eff-b803-53b4f27d056f', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:29:10 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (056e72f1-43ea-47ca-80c8-6b85909c1e4e) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:29:10 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (056e72f1-43ea-47ca-80c8-6b85909c1e4e) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:29:10 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (59a3d70b-a74f-4873-b767-cd5a57a63fa7) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:29:10 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Volume a4d96ab8-2072-4eff-b803-53b4f27d056f: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-a4d96ab8-2072-4eff-b803-53b4f27d056f', 'volume_size': 1} Apr 17 15:29:10 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] creating volume v_a4d96ab8-2072-4eff-b803-53b4f27d056f. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 15:29:10 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] create volume {'name': 'v_a4d96ab8-2072-4eff-b803-53b4f27d056f', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 15:29:10 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:29:10 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:29:12 user cinder-volume[84696]: DEBUG cinder.utils [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 1.884(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:29:12 user cinder-volume[84696]: DEBUG cinder.utils [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:29:15 user cinder-volume[84696]: DEBUG cinder.utils [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 62.635(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:29:15 user cinder-volume[84696]: DEBUG cinder.utils [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:29:15 user cinder-volume[84696]: DEBUG cinder.utils [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 4.953(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:29:15 user cinder-volume[84696]: DEBUG cinder.utils [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:29:16 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-cddb5331-13a7-4304-8a88-324e7cbf6185 req-43c9eb22-6970-4755-97a1-9763d442bfdd tempest-TestShelveInstance-1683612675 None] create export for volume: d895d11d-6880-4418-a9ad-feca813ed5d6. {{(pid=84696) create_export /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:699}} Apr 17 15:29:16 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-cddb5331-13a7-4304-8a88-324e7cbf6185 req-43c9eb22-6970-4755-97a1-9763d442bfdd tempest-TestShelveInstance-1683612675 None] ensure volume d895d11d-6880-4418-a9ad-feca813ed5d6 assigned to a proper target {{(pid=84696) _ensure_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:866}} Apr 17 15:29:16 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [req-cddb5331-13a7-4304-8a88-324e7cbf6185 req-43c9eb22-6970-4755-97a1-9763d442bfdd tempest-TestShelveInstance-1683612675 None] check if targe iqn.2010-10.org.openstack:d895d11d-6880-4418-a9ad-feca813ed5d6 exists {{(pid=84696) is_target /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:376}} Apr 17 15:29:16 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-cddb5331-13a7-4304-8a88-324e7cbf6185 req-43c9eb22-6970-4755-97a1-9763d442bfdd tempest-TestShelveInstance-1683612675 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:d895d11d-6880-4418-a9ad-feca813ed5d6 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:29:16 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-cddb5331-13a7-4304-8a88-324e7cbf6185 req-43c9eb22-6970-4755-97a1-9763d442bfdd tempest-TestShelveInstance-1683612675 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:d895d11d-6880-4418-a9ad-feca813ed5d6 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:29:18 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 62.548(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:29:18 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:29:18 user cinder-volume[84696]: DEBUG cinder.utils [None req-643b65b6-f16f-4041-b9cc-be64c983a38a tempest-TestVolumeSwap-1714109745 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 32.843(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:29:18 user cinder-volume[84696]: DEBUG cinder.utils [None req-643b65b6-f16f-4041-b9cc-be64c983a38a tempest-TestVolumeSwap-1714109745 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:485ea70c-618d-4373-a78f-a460914fc095/luns/v_485ea70c-618d-4373-a78f-a460914fc095 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:29:18 user cinder-volume[84696]: DEBUG cinder.utils [req-cddb5331-13a7-4304-8a88-324e7cbf6185 req-43c9eb22-6970-4755-97a1-9763d442bfdd tempest-TestShelveInstance-1683612675 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 1.713(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:29:18 user cinder-volume[84696]: DEBUG cinder.utils [req-cddb5331-13a7-4304-8a88-324e7cbf6185 req-43c9eb22-6970-4755-97a1-9763d442bfdd tempest-TestShelveInstance-1683612675 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:d895d11d-6880-4418-a9ad-feca813ed5d6 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:29:21 user cinder-volume[84696]: DEBUG cinder.utils [None req-643b65b6-f16f-4041-b9cc-be64c983a38a tempest-TestVolumeSwap-1714109745 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 35.915(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:29:21 user cinder-volume[84696]: DEBUG cinder.utils [None req-643b65b6-f16f-4041-b9cc-be64c983a38a tempest-TestVolumeSwap-1714109745 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:485ea70c-618d-4373-a78f-a460914fc095/luns/v_485ea70c-618d-4373-a78f-a460914fc095 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:29:21 user cinder-volume[84696]: DEBUG cinder.utils [req-cddb5331-13a7-4304-8a88-324e7cbf6185 req-43c9eb22-6970-4755-97a1-9763d442bfdd tempest-TestShelveInstance-1683612675 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 4.787(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:29:21 user cinder-volume[84696]: DEBUG cinder.utils [req-cddb5331-13a7-4304-8a88-324e7cbf6185 req-43c9eb22-6970-4755-97a1-9763d442bfdd tempest-TestShelveInstance-1683612675 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:d895d11d-6880-4418-a9ad-feca813ed5d6 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:29:21 user cinder-volume[84696]: DEBUG cinder.utils [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 11.104(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:29:21 user cinder-volume[84696]: DEBUG cinder.utils [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:29:24 user cinder-volume[84696]: DEBUG cinder.utils [None req-d0984215-eaf6-4942-b2a1-997c1c6518b5 tempest-TaggedAttachmentsTest-1309189231 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 63.509(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:29:24 user cinder-volume[84696]: DEBUG cinder.utils [None req-d0984215-eaf6-4942-b2a1-997c1c6518b5 tempest-TaggedAttachmentsTest-1309189231 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:3c05f4f0-17a8-4f4c-9987-1fcb502ffd3a/luns/v_3c05f4f0-17a8-4f4c-9987-1fcb502ffd3a (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:29:24 user cinder-volume[84696]: DEBUG cinder.utils [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 61.948(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:29:24 user cinder-volume[84696]: DEBUG cinder.utils [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:29:27 user cinder-volume[84696]: DEBUG cinder.utils [None req-643b65b6-f16f-4041-b9cc-be64c983a38a tempest-TestVolumeSwap-1714109745 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 42.055(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:29:27 user cinder-volume[84696]: DEBUG cinder.utils [None req-643b65b6-f16f-4041-b9cc-be64c983a38a tempest-TestVolumeSwap-1714109745 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:485ea70c-618d-4373-a78f-a460914fc095/luns/v_485ea70c-618d-4373-a78f-a460914fc095 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:29:27 user cinder-volume[84696]: DEBUG cinder.utils [req-cddb5331-13a7-4304-8a88-324e7cbf6185 req-43c9eb22-6970-4755-97a1-9763d442bfdd tempest-TestShelveInstance-1683612675 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 10.926(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:29:27 user cinder-volume[84696]: DEBUG cinder.utils [req-cddb5331-13a7-4304-8a88-324e7cbf6185 req-43c9eb22-6970-4755-97a1-9763d442bfdd tempest-TestShelveInstance-1683612675 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:d895d11d-6880-4418-a9ad-feca813ed5d6 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:29:30 user cinder-volume[84696]: DEBUG cinder.utils [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 20.312(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:29:30 user cinder-volume[84696]: DEBUG cinder.utils [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:29:33 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=8 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:29:33 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=11,cinder:UPDATE=4 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:29:36 user cinder-volume[84696]: DEBUG cinder.utils [None req-643b65b6-f16f-4041-b9cc-be64c983a38a tempest-TestVolumeSwap-1714109745 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 51.271(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:29:36 user cinder-volume[84696]: DEBUG cinder.utils [None req-643b65b6-f16f-4041-b9cc-be64c983a38a tempest-TestVolumeSwap-1714109745 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:485ea70c-618d-4373-a78f-a460914fc095/luns/v_485ea70c-618d-4373-a78f-a460914fc095 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:29:36 user cinder-volume[84696]: DEBUG cinder.utils [req-cddb5331-13a7-4304-8a88-324e7cbf6185 req-43c9eb22-6970-4755-97a1-9763d442bfdd tempest-TestShelveInstance-1683612675 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 20.143(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:29:36 user cinder-volume[84696]: DEBUG cinder.utils [req-cddb5331-13a7-4304-8a88-324e7cbf6185 req-43c9eb22-6970-4755-97a1-9763d442bfdd tempest-TestShelveInstance-1683612675 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:d895d11d-6880-4418-a9ad-feca813ed5d6 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:29:42 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-38bdff97-cca4-4bb8-8df0-3261c844629a req-073af777-dfca-4d61-85d6-9ed616a692d0 tempest-TestStampPattern-173447269 None] create export for volume: a9c177a1-61f0-4201-bb8d-7f18c9af396c. {{(pid=84696) create_export /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:699}} Apr 17 15:29:42 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-38bdff97-cca4-4bb8-8df0-3261c844629a req-073af777-dfca-4d61-85d6-9ed616a692d0 tempest-TestStampPattern-173447269 None] ensure volume a9c177a1-61f0-4201-bb8d-7f18c9af396c assigned to a proper target {{(pid=84696) _ensure_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:866}} Apr 17 15:29:42 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [req-38bdff97-cca4-4bb8-8df0-3261c844629a req-073af777-dfca-4d61-85d6-9ed616a692d0 tempest-TestStampPattern-173447269 None] check if targe iqn.2010-10.org.openstack:a9c177a1-61f0-4201-bb8d-7f18c9af396c exists {{(pid=84696) is_target /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:376}} Apr 17 15:29:42 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-38bdff97-cca4-4bb8-8df0-3261c844629a req-073af777-dfca-4d61-85d6-9ed616a692d0 tempest-TestStampPattern-173447269 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:a9c177a1-61f0-4201-bb8d-7f18c9af396c {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:29:42 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-38bdff97-cca4-4bb8-8df0-3261c844629a req-073af777-dfca-4d61-85d6-9ed616a692d0 tempest-TestStampPattern-173447269 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:a9c177a1-61f0-4201-bb8d-7f18c9af396c {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:29:42 user cinder-volume[84696]: DEBUG cinder.utils [req-38bdff97-cca4-4bb8-8df0-3261c844629a req-073af777-dfca-4d61-85d6-9ed616a692d0 tempest-TestStampPattern-173447269 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 0.774(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:29:42 user cinder-volume[84696]: DEBUG cinder.utils [req-38bdff97-cca4-4bb8-8df0-3261c844629a req-073af777-dfca-4d61-85d6-9ed616a692d0 tempest-TestStampPattern-173447269 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:a9c177a1-61f0-4201-bb8d-7f18c9af396c (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:29:46 user cinder-volume[84696]: DEBUG cinder.utils [req-38bdff97-cca4-4bb8-8df0-3261c844629a req-073af777-dfca-4d61-85d6-9ed616a692d0 tempest-TestStampPattern-173447269 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 3.846(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:29:46 user cinder-volume[84696]: DEBUG cinder.utils [req-38bdff97-cca4-4bb8-8df0-3261c844629a req-073af777-dfca-4d61-85d6-9ed616a692d0 tempest-TestStampPattern-173447269 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:a9c177a1-61f0-4201-bb8d-7f18c9af396c (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:29:49 user cinder-volume[84696]: DEBUG cinder.utils [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 96.427(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:29:49 user cinder-volume[84696]: DEBUG cinder.utils [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:29:49 user cinder-volume[84696]: DEBUG cinder.utils [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 38.745(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:29:49 user cinder-volume[84696]: DEBUG cinder.utils [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:29:52 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 96.339(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:29:52 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:29:52 user cinder-volume[84696]: DEBUG cinder.utils [req-38bdff97-cca4-4bb8-8df0-3261c844629a req-073af777-dfca-4d61-85d6-9ed616a692d0 tempest-TestStampPattern-173447269 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 9.991(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:29:52 user cinder-volume[84696]: DEBUG cinder.utils [req-38bdff97-cca4-4bb8-8df0-3261c844629a req-073af777-dfca-4d61-85d6-9ed616a692d0 tempest-TestStampPattern-173447269 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:a9c177a1-61f0-4201-bb8d-7f18c9af396c (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:29:53 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=5 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:29:55 user cinder-volume[84696]: DEBUG cinder.utils [req-cddb5331-13a7-4304-8a88-324e7cbf6185 req-43c9eb22-6970-4755-97a1-9763d442bfdd tempest-TestShelveInstance-1683612675 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 38.572(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:29:55 user cinder-volume[84696]: DEBUG cinder.utils [req-cddb5331-13a7-4304-8a88-324e7cbf6185 req-43c9eb22-6970-4755-97a1-9763d442bfdd tempest-TestShelveInstance-1683612675 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:d895d11d-6880-4418-a9ad-feca813ed5d6 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:29:55 user cinder-volume[84696]: DEBUG cinder.utils [None req-643b65b6-f16f-4041-b9cc-be64c983a38a tempest-TestVolumeSwap-1714109745 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 69.705(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:29:55 user cinder-volume[84696]: DEBUG cinder.utils [None req-643b65b6-f16f-4041-b9cc-be64c983a38a tempest-TestVolumeSwap-1714109745 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:485ea70c-618d-4373-a78f-a460914fc095/luns/v_485ea70c-618d-4373-a78f-a460914fc095 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:29:58 user cinder-volume[84696]: DEBUG cinder.utils [None req-d0984215-eaf6-4942-b2a1-997c1c6518b5 tempest-TaggedAttachmentsTest-1309189231 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 97.301(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:29:58 user cinder-volume[84696]: DEBUG cinder.utils [None req-d0984215-eaf6-4942-b2a1-997c1c6518b5 tempest-TaggedAttachmentsTest-1309189231 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:3c05f4f0-17a8-4f4c-9987-1fcb502ffd3a/luns/v_3c05f4f0-17a8-4f4c-9987-1fcb502ffd3a (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:29:58 user cinder-volume[84696]: DEBUG cinder.utils [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 95.739(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:29:58 user cinder-volume[84696]: DEBUG cinder.utils [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:30:01 user cinder-volume[84696]: DEBUG cinder.utils [req-38bdff97-cca4-4bb8-8df0-3261c844629a req-073af777-dfca-4d61-85d6-9ed616a692d0 tempest-TestStampPattern-173447269 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 19.206(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:30:01 user cinder-volume[84696]: DEBUG cinder.utils [req-38bdff97-cca4-4bb8-8df0-3261c844629a req-073af777-dfca-4d61-85d6-9ed616a692d0 tempest-TestStampPattern-173447269 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:a9c177a1-61f0-4201-bb8d-7f18c9af396c (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:30:03 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=7,cinder:UPDATE=3 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:30:19 user cinder-volume[84696]: DEBUG cinder.utils [req-38bdff97-cca4-4bb8-8df0-3261c844629a req-073af777-dfca-4d61-85d6-9ed616a692d0 tempest-TestStampPattern-173447269 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 37.638(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:30:19 user cinder-volume[84696]: DEBUG cinder.utils [req-38bdff97-cca4-4bb8-8df0-3261c844629a req-073af777-dfca-4d61-85d6-9ed616a692d0 tempest-TestStampPattern-173447269 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:a9c177a1-61f0-4201-bb8d-7f18c9af396c (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:30:22 user cinder-volume[84696]: DEBUG cinder.utils [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 72.536(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:30:22 user cinder-volume[84696]: DEBUG cinder.utils [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:30:23 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=3,cinder:UPDATE=1 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:30:29 user cinder-volume[84696]: DEBUG cinder.utils [req-cddb5331-13a7-4304-8a88-324e7cbf6185 req-43c9eb22-6970-4755-97a1-9763d442bfdd tempest-TestShelveInstance-1683612675 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 72.369(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:30:29 user cinder-volume[84696]: DEBUG cinder.utils [req-cddb5331-13a7-4304-8a88-324e7cbf6185 req-43c9eb22-6970-4755-97a1-9763d442bfdd tempest-TestShelveInstance-1683612675 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:d895d11d-6880-4418-a9ad-feca813ed5d6 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:30:29 user cinder-volume[84696]: DEBUG cinder.utils [None req-643b65b6-f16f-4041-b9cc-be64c983a38a tempest-TestVolumeSwap-1714109745 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 103.500(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:30:29 user cinder-volume[84696]: DEBUG cinder.utils [None req-643b65b6-f16f-4041-b9cc-be64c983a38a tempest-TestVolumeSwap-1714109745 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:485ea70c-618d-4373-a78f-a460914fc095/luns/v_485ea70c-618d-4373-a78f-a460914fc095 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:30:33 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=6,cinder:UPDATE=2 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:30:53 user cinder-volume[84696]: DEBUG cinder.utils [req-38bdff97-cca4-4bb8-8df0-3261c844629a req-073af777-dfca-4d61-85d6-9ed616a692d0 tempest-TestStampPattern-173447269 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 71.430(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:30:53 user cinder-volume[84696]: DEBUG cinder.utils [req-38bdff97-cca4-4bb8-8df0-3261c844629a req-073af777-dfca-4d61-85d6-9ed616a692d0 tempest-TestStampPattern-173447269 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:a9c177a1-61f0-4201-bb8d-7f18c9af396c (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:30:53 user cinder-volume[84696]: DEBUG cinder.utils [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 160.942(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:30:53 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:30:53 user cinder-volume[84696]: INFO cinder.message.api [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Creating message record for request_id = req-e7b5301d-81ea-4ab5-9d69-766347595e11 Apr 17 15:30:53 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 15:30:53 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (0407bfe2-385a-4571-9294-c3366a4fa775) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 15:30:53 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 15:30:53 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:28:12Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesActionsTest-Volume-620204537',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=62709a81-268b-4227-ba4a-070034f36414,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='a8b97ff6ed6047f0a33e74684a1c162b',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:28:13Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:28:13Z,use_quota=True,user_id='c993ceeb4a5c49eea14057e2125518ad',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 15:30:53 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:28:12Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesActionsTest-Volume-620204537',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=62709a81-268b-4227-ba4a-070034f36414,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='a8b97ff6ed6047f0a33e74684a1c162b',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:28:13Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:28:13Z,use_quota=True,user_id='c993ceeb4a5c49eea14057e2125518ad',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(62709a81-268b-4227-ba4a-070034f36414),volume_id=62709a81-268b-4227-ba4a-070034f36414,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': '62709a81-268b-4227-ba4a-070034f36414', 'volume_name': 'volume-62709a81-268b-4227-ba4a-070034f36414', 'volume_size': 1}} Apr 17 15:30:53 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'retry': {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']}, 'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': '62709a81-268b-4227-ba4a-070034f36414', 'volume': {'id': '62709a81-268b-4227-ba4a-070034f36414', '_name_id': None, 'ec2_id': None, 'user_id': 'c993ceeb4a5c49eea14057e2125518ad', 'project_id': 'a8b97ff6ed6047f0a33e74684a1c162b', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-VolumesActionsTest-Volume-620204537', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T15:28:12.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-62709a81-268b-4227-ba4a-070034f36414', 'name_id': '62709a81-268b-4227-ba4a-070034f36414', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumesActionsTest-Volume-620204537', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': 'a8b97ff6ed6047f0a33e74684a1c162b', 'qos_specs': None, 'reservations': ['1b325ba3-edbb-4c50-91e6-5be6258c7eee', '4eb6a10f-f088-4780-a8d5-294b078de0ce', 'ede206e9-4b4d-4f26-8a58-97a099b605d5', '2beaf0e1-5633-41fc-ad41-398657c8ddea'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'c993ceeb4a5c49eea14057e2125518ad', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumesActionsTest-Volume-620204537', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': 'a8b97ff6ed6047f0a33e74684a1c162b', 'qos_specs': None, 'reservations': ['1b325ba3-edbb-4c50-91e6-5be6258c7eee', '4eb6a10f-f088-4780-a8d5-294b078de0ce', 'ede206e9-4b4d-4f26-8a58-97a099b605d5', '2beaf0e1-5633-41fc-ad41-398657c8ddea'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'c993ceeb4a5c49eea14057e2125518ad', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': 'c993ceeb4a5c49eea14057e2125518ad', 'metadata': {}, 'qos_specs': None}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(62709a81-268b-4227-ba4a-070034f36414),volume_id=62709a81-268b-4227-ba4a-070034f36414,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:28:12Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesActionsTest-Volume-620204537',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=62709a81-268b-4227-ba4a-070034f36414,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='a8b97ff6ed6047f0a33e74684a1c162b',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:28:13Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:28:13Z,use_quota=True,user_id='c993ceeb4a5c49eea14057e2125518ad',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 15:30:53 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:28:12Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesActionsTest-Volume-620204537',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=62709a81-268b-4227-ba4a-070034f36414,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='a8b97ff6ed6047f0a33e74684a1c162b',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:28:13Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:28:13Z,use_quota=True,user_id='c993ceeb4a5c49eea14057e2125518ad',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:28:12Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesActionsTest-Volume-620204537',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=62709a81-268b-4227-ba4a-070034f36414,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='a8b97ff6ed6047f0a33e74684a1c162b',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:28:13Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:28:13Z,use_quota=True,user_id='c993ceeb4a5c49eea14057e2125518ad',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 15:30:53 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 15:30:53 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 15:30:53 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:30:53 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 15:30:53 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 15:30:53 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 15:30:53 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 15:30:53 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 15:30:53 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 15:30:53 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 15:30:53 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 15:30:53 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 15:30:53 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 15:30:53 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 15:30:53 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 15:30:53 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 15:30:53 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 15:30:53 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 15:30:53 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 15:30:53 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 15:30:53 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (0407bfe2-385a-4571-9294-c3366a4fa775) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:30:53 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (0407bfe2-385a-4571-9294-c3366a4fa775) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:30:53 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (b5a33e7e-cb4d-4acb-811f-cd44ece1b863) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:30:53 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (b5a33e7e-cb4d-4acb-811f-cd44ece1b863) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:30:53 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (6bf018c4-d42e-448d-8618-f7ed9f876d9e) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:30:53 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (6bf018c4-d42e-448d-8618-f7ed9f876d9e) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:30:53 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (35ddcbc2-13dd-4fbe-8faa-513f4b35e511) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:30:53 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Updating volume 62709a81-268b-4227-ba4a-070034f36414 with {'scheduled_at': datetime.datetime(2023, 4, 17, 15, 30, 53, 706640), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 15:30:53 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Volume 62709a81-268b-4227-ba4a-070034f36414: re-scheduling SchedulerAPI.create_volume attempt 1 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 15:30:53 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Volume 62709a81-268b-4227-ba4a-070034f36414: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 15:30:53 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] deleating volume v_62709a81-268b-4227-ba4a-070034f36414 {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 15:30:53 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] get snapshots for volume v_62709a81-268b-4227-ba4a-070034f36414 {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 15:30:53 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_62709a81-268b-4227-ba4a-070034f36414/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:30:53 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_62709a81-268b-4227-ba4a-070034f36414/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:30:53 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Volume reschedule parameters: True retry: {'num_attempts': 2, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 15:30:53 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Flow 'volume_create_manager' (45c1607c-735d-41db-8ea0-22e6846280ce) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:30:53 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (2da3cb5d-0cee-45d1-9121-1f4d3236e757) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:30:53 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (2da3cb5d-0cee-45d1-9121-1f4d3236e757) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:28:12Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesActionsTest-Volume-620204537',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=62709a81-268b-4227-ba4a-070034f36414,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='a8b97ff6ed6047f0a33e74684a1c162b',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:30:54Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:30:54Z,use_quota=True,user_id='c993ceeb4a5c49eea14057e2125518ad',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:30:53 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (f282ab50-6be5-4b8b-a8ff-74df520036f3) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:30:53 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (f282ab50-6be5-4b8b-a8ff-74df520036f3) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:30:53 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (d2d8b72d-5b36-4789-b831-089b51523205) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:30:53 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (d2d8b72d-5b36-4789-b831-089b51523205) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': '62709a81-268b-4227-ba4a-070034f36414', 'volume_name': 'volume-62709a81-268b-4227-ba4a-070034f36414', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:30:53 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (25f6a879-3c9a-40e5-bf1b-dd26fbb692a4) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:30:53 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (25f6a879-3c9a-40e5-bf1b-dd26fbb692a4) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:30:53 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (90879f34-89e0-49ff-9cee-80b45a763611) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:30:53 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Volume 62709a81-268b-4227-ba4a-070034f36414: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-62709a81-268b-4227-ba4a-070034f36414', 'volume_size': 1} Apr 17 15:30:53 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] creating volume v_62709a81-268b-4227-ba4a-070034f36414. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 15:30:53 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] create volume {'name': 'v_62709a81-268b-4227-ba4a-070034f36414', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 15:30:53 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:30:53 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:30:56 user cinder-volume[84696]: DEBUG cinder.utils [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.964(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:30:56 user cinder-volume[84696]: DEBUG cinder.utils [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_62709a81-268b-4227-ba4a-070034f36414/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:30:56 user cinder-volume[84696]: DEBUG cinder.utils [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.782(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:30:56 user cinder-volume[84696]: DEBUG cinder.utils [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:30:56 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 160.856(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:30:56 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:30:56 user cinder-volume[84696]: ERROR oslo_service.periodic_task [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Error during VolumeManager.publish_service_capabilities: TypeError: 'NoneType' object is not subscriptable Apr 17 15:30:56 user cinder-volume[84696]: ERROR oslo_service.periodic_task Traceback (most recent call last): Apr 17 15:30:56 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/usr/local/lib/python3.10/dist-packages/oslo_service/periodic_task.py", line 216, in run_periodic_tasks Apr 17 15:30:56 user cinder-volume[84696]: ERROR oslo_service.periodic_task task(self, context) Apr 17 15:30:56 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/manager.py", line 2876, in publish_service_capabilities Apr 17 15:30:56 user cinder-volume[84696]: ERROR oslo_service.periodic_task self._report_driver_status(context) Apr 17 15:30:56 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/manager.py", line 2755, in _report_driver_status Apr 17 15:30:56 user cinder-volume[84696]: ERROR oslo_service.periodic_task volume_stats = get_stats() Apr 17 15:30:56 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/usr/local/lib/python3.10/dist-packages/oslo_utils/timeutils.py", line 319, in wrapper Apr 17 15:30:56 user cinder-volume[84696]: ERROR oslo_service.periodic_task result = func(*args, **kwargs) Apr 17 15:30:56 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/manager.py", line 2753, in get_stats Apr 17 15:30:56 user cinder-volume[84696]: ERROR oslo_service.periodic_task return self.driver.get_volume_stats(refresh=True) Apr 17 15:30:56 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/driver.py", line 666, in get_volume_stats Apr 17 15:30:56 user cinder-volume[84696]: ERROR oslo_service.periodic_task self._update_volume_stats() Apr 17 15:30:56 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 726, in _update_volume_stats Apr 17 15:30:56 user cinder-volume[84696]: ERROR oslo_service.periodic_task pool_stats = self.ra.get_pool_stats() Apr 17 15:30:56 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 897, in get_pool_stats Apr 17 15:30:56 user cinder-volume[84696]: ERROR oslo_service.periodic_task if not resp["error"] and resp["code"] == 200: Apr 17 15:30:56 user cinder-volume[84696]: ERROR oslo_service.periodic_task TypeError: 'NoneType' object is not subscriptable Apr 17 15:30:56 user cinder-volume[84696]: ERROR oslo_service.periodic_task Apr 17 15:30:56 user cinder-volume[84696]: WARNING oslo.service.loopingcall [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Function 'cinder.service.Service.periodic_tasks' run outlasted interval by 100.86 sec Apr 17 15:30:56 user cinder-volume[84696]: DEBUG oslo_service.periodic_task [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Running periodic task VolumeManager.publish_service_capabilities {{(pid=84696) run_periodic_tasks /usr/local/lib/python3.10/dist-packages/oslo_service/periodic_task.py:210}} Apr 17 15:30:56 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Updating volume stats {{(pid=84696) _update_volume_stats /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:724}} Apr 17 15:30:56 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Get pool Pool-0 fsprops {{(pid=84696) get_pool_stats /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:894}} Apr 17 15:30:56 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:30:56 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:30:59 user cinder-volume[84696]: DEBUG cinder.utils [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.850(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:30:59 user cinder-volume[84696]: DEBUG cinder.utils [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:30:59 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 3.061(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:30:59 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:30:59 user cinder-volume[84696]: DEBUG cinder.utils [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.038(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:30:59 user cinder-volume[84696]: DEBUG cinder.utils [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_62709a81-268b-4227-ba4a-070034f36414/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:31:02 user cinder-volume[84696]: DEBUG cinder.utils [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 160.250(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:31:02 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:31:02 user cinder-volume[84696]: INFO cinder.message.api [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Creating message record for request_id = req-051b1fdc-54da-4587-a61f-deb7faa24c2c Apr 17 15:31:02 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.138(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:31:02 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:31:02 user cinder-volume[84696]: DEBUG cinder.utils [None req-d0984215-eaf6-4942-b2a1-997c1c6518b5 tempest-TaggedAttachmentsTest-1309189231 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 161.820(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:31:02 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-d0984215-eaf6-4942-b2a1-997c1c6518b5 tempest-TaggedAttachmentsTest-1309189231 None] Geting None from DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:3c05f4f0-17a8-4f4c-9987-1fcb502ffd3a/luns/v_3c05f4f0-17a8-4f4c-9987-1fcb502ffd3a {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:31:02 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 15:31:02 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=11,cinder:UPDATE=3,cinder:INSERT=2 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:31:02 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (9d098b7c-98f7-4428-8290-d91cfbe936a7) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 15:31:02 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 15:31:02 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:28:22Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-SnapshotMetadataTestJSON-Volume-902610845',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=17f8d4f7-2dcd-4041-8d0b-5c8afebb75bf,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='a4653bf4d2914c8d956e86ad3405051d',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:28:22Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:28:22Z,use_quota=True,user_id='5598ef2700af47d9bcf6fadea4460cb2',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 15:31:02 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:28:22Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-SnapshotMetadataTestJSON-Volume-902610845',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=17f8d4f7-2dcd-4041-8d0b-5c8afebb75bf,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='a4653bf4d2914c8d956e86ad3405051d',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:28:22Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:28:22Z,use_quota=True,user_id='5598ef2700af47d9bcf6fadea4460cb2',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(17f8d4f7-2dcd-4041-8d0b-5c8afebb75bf),volume_id=17f8d4f7-2dcd-4041-8d0b-5c8afebb75bf,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': '17f8d4f7-2dcd-4041-8d0b-5c8afebb75bf', 'volume_name': 'volume-17f8d4f7-2dcd-4041-8d0b-5c8afebb75bf', 'volume_size': 1}} Apr 17 15:31:02 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'retry': {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']}, 'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': '17f8d4f7-2dcd-4041-8d0b-5c8afebb75bf', 'volume': {'id': '17f8d4f7-2dcd-4041-8d0b-5c8afebb75bf', '_name_id': None, 'ec2_id': None, 'user_id': '5598ef2700af47d9bcf6fadea4460cb2', 'project_id': 'a4653bf4d2914c8d956e86ad3405051d', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-SnapshotMetadataTestJSON-Volume-902610845', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T15:28:22.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-17f8d4f7-2dcd-4041-8d0b-5c8afebb75bf', 'name_id': '17f8d4f7-2dcd-4041-8d0b-5c8afebb75bf', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-SnapshotMetadataTestJSON-Volume-902610845', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': 'a4653bf4d2914c8d956e86ad3405051d', 'qos_specs': None, 'reservations': ['ab8ffe08-13a2-4be5-8c18-55343d579bf3', 'd62e062b-7f8a-4b18-8a96-5ae0097970db', '0311c219-244b-45f9-903f-bdbceee93caf', 'f31b8283-7526-44ae-bd04-3726c40be823'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': '5598ef2700af47d9bcf6fadea4460cb2', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-SnapshotMetadataTestJSON-Volume-902610845', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': 'a4653bf4d2914c8d956e86ad3405051d', 'qos_specs': None, 'reservations': ['ab8ffe08-13a2-4be5-8c18-55343d579bf3', 'd62e062b-7f8a-4b18-8a96-5ae0097970db', '0311c219-244b-45f9-903f-bdbceee93caf', 'f31b8283-7526-44ae-bd04-3726c40be823'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': '5598ef2700af47d9bcf6fadea4460cb2', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': '5598ef2700af47d9bcf6fadea4460cb2', 'metadata': {}, 'qos_specs': None}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(17f8d4f7-2dcd-4041-8d0b-5c8afebb75bf),volume_id=17f8d4f7-2dcd-4041-8d0b-5c8afebb75bf,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:28:22Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-SnapshotMetadataTestJSON-Volume-902610845',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=17f8d4f7-2dcd-4041-8d0b-5c8afebb75bf,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='a4653bf4d2914c8d956e86ad3405051d',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:28:22Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:28:22Z,use_quota=True,user_id='5598ef2700af47d9bcf6fadea4460cb2',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 15:31:02 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:28:22Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-SnapshotMetadataTestJSON-Volume-902610845',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=17f8d4f7-2dcd-4041-8d0b-5c8afebb75bf,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='a4653bf4d2914c8d956e86ad3405051d',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:28:22Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:28:22Z,use_quota=True,user_id='5598ef2700af47d9bcf6fadea4460cb2',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:28:22Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-SnapshotMetadataTestJSON-Volume-902610845',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=17f8d4f7-2dcd-4041-8d0b-5c8afebb75bf,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='a4653bf4d2914c8d956e86ad3405051d',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:28:22Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:28:22Z,use_quota=True,user_id='5598ef2700af47d9bcf6fadea4460cb2',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 15:31:02 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 15:31:02 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 15:31:02 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:31:02 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 15:31:02 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 15:31:02 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 15:31:02 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 15:31:02 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 15:31:02 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 15:31:02 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 15:31:02 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 15:31:02 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 15:31:02 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 15:31:02 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 15:31:02 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 15:31:02 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 15:31:02 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 15:31:02 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 15:31:02 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 15:31:02 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 15:31:02 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (9d098b7c-98f7-4428-8290-d91cfbe936a7) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:31:02 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (9d098b7c-98f7-4428-8290-d91cfbe936a7) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:31:02 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (58cbf18b-a802-4594-9e2c-dbd6f30e4edd) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:31:02 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (58cbf18b-a802-4594-9e2c-dbd6f30e4edd) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:31:02 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (400785c0-7281-49a4-a487-ca23510184ff) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:31:02 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (400785c0-7281-49a4-a487-ca23510184ff) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:31:02 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (ba0a21a5-fc9a-4815-9416-e53538e13902) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:31:02 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Updating volume 17f8d4f7-2dcd-4041-8d0b-5c8afebb75bf with {'scheduled_at': datetime.datetime(2023, 4, 17, 15, 31, 2, 929435), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 15:31:02 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Volume 17f8d4f7-2dcd-4041-8d0b-5c8afebb75bf: re-scheduling SchedulerAPI.create_volume attempt 1 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 15:31:02 user cinder-volume[84696]: DEBUG cinder.coordination [None req-d0984215-eaf6-4942-b2a1-997c1c6518b5 tempest-TaggedAttachmentsTest-1309189231 None] Lock "cinder-3c05f4f0-17a8-4f4c-9987-1fcb502ffd3a-delete_volume" released by "delete_volume" :: held 161.969s {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:207}} Apr 17 15:31:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-d0984215-eaf6-4942-b2a1-997c1c6518b5 tempest-TaggedAttachmentsTest-1309189231 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 15:31:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 15:31:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 15:31:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 15:31:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 15:31:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 15:31:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 15:31:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 15:31:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 203, in wrapper Apr 17 15:31:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server skip_clean = func(self, context, volume, *args, **kwargs) Apr 17 15:31:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 15:31:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 15:31:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/coordination.py", line 200, in _synchronized Apr 17 15:31:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return f(*a, **k) Apr 17 15:31:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 15:31:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 15:31:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 15:31:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 15:31:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 1035, in delete_volume Apr 17 15:31:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 15:31:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 15:31:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 15:31:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 15:31:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 15:31:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 1004, in delete_volume Apr 17 15:31:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.driver.remove_export(context, volume) Apr 17 15:31:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 720, in remove_export Apr 17 15:31:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self._remove_target_volume(volume) Apr 17 15:31:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 918, in _remove_target_volume Apr 17 15:31:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.detach_target_vol(target_name, jcom.vname(volume.id)) Apr 17 15:31:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 612, in detach_target_vol Apr 17 15:31:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if resp["code"] in (200, 201, 204): Apr 17 15:31:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 15:31:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 15:31:02 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Volume 17f8d4f7-2dcd-4041-8d0b-5c8afebb75bf: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 15:31:02 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] deleating volume v_17f8d4f7-2dcd-4041-8d0b-5c8afebb75bf {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 15:31:02 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] get snapshots for volume v_17f8d4f7-2dcd-4041-8d0b-5c8afebb75bf {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 15:31:02 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_17f8d4f7-2dcd-4041-8d0b-5c8afebb75bf/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:31:02 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_17f8d4f7-2dcd-4041-8d0b-5c8afebb75bf/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:31:03 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Volume reschedule parameters: True retry: {'num_attempts': 2, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 15:31:03 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Flow 'volume_create_manager' (49907c15-6e1c-41b2-acbc-bfe4f1435174) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:31:03 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (a8345d2b-66ce-4522-b887-89959f362043) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:31:03 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (a8345d2b-66ce-4522-b887-89959f362043) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:28:22Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-SnapshotMetadataTestJSON-Volume-902610845',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=17f8d4f7-2dcd-4041-8d0b-5c8afebb75bf,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='a4653bf4d2914c8d956e86ad3405051d',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:31:03Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:31:03Z,use_quota=True,user_id='5598ef2700af47d9bcf6fadea4460cb2',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:31:03 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (0cff7168-848c-4fc6-98f8-492aeec7e936) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:31:03 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (0cff7168-848c-4fc6-98f8-492aeec7e936) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:31:03 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (d6f3f11b-ca43-4d6d-930a-d0dab99eef4f) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:31:03 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (d6f3f11b-ca43-4d6d-930a-d0dab99eef4f) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': '17f8d4f7-2dcd-4041-8d0b-5c8afebb75bf', 'volume_name': 'volume-17f8d4f7-2dcd-4041-8d0b-5c8afebb75bf', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:31:03 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (0fd09b44-7dcc-45d0-80e7-e05317bdc25c) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:31:03 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (0fd09b44-7dcc-45d0-80e7-e05317bdc25c) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:31:03 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (eeba2ba2-6604-4eb9-8edd-18cf42c1cbbe) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:31:03 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Volume 17f8d4f7-2dcd-4041-8d0b-5c8afebb75bf: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-17f8d4f7-2dcd-4041-8d0b-5c8afebb75bf', 'volume_size': 1} Apr 17 15:31:03 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] creating volume v_17f8d4f7-2dcd-4041-8d0b-5c8afebb75bf. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 15:31:03 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] create volume {'name': 'v_17f8d4f7-2dcd-4041-8d0b-5c8afebb75bf', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 15:31:03 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:31:03 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:31:05 user cinder-volume[84696]: DEBUG cinder.utils [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.955(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:31:05 user cinder-volume[84696]: DEBUG cinder.utils [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_17f8d4f7-2dcd-4041-8d0b-5c8afebb75bf/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:31:05 user cinder-volume[84696]: DEBUG cinder.utils [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.000(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:31:05 user cinder-volume[84696]: DEBUG cinder.utils [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:31:05 user cinder-volume[84696]: DEBUG cinder.utils [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.781(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:31:05 user cinder-volume[84696]: DEBUG cinder.utils [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:31:05 user cinder-volume[84696]: DEBUG cinder.utils [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.187(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:31:05 user cinder-volume[84696]: DEBUG cinder.utils [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_62709a81-268b-4227-ba4a-070034f36414/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:31:08 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.276(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:31:08 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:31:08 user cinder-volume[84696]: DEBUG cinder.utils [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.848(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:31:08 user cinder-volume[84696]: DEBUG cinder.utils [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:31:08 user cinder-volume[84696]: DEBUG cinder.utils [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.026(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:31:08 user cinder-volume[84696]: DEBUG cinder.utils [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_17f8d4f7-2dcd-4041-8d0b-5c8afebb75bf/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:31:13 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=17,cinder:UPDATE=6,cinder:INSERT=2,cinder:DELETE=1 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:31:15 user cinder-volume[84696]: DEBUG cinder.utils [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.342(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:31:15 user cinder-volume[84696]: DEBUG cinder.utils [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:31:15 user cinder-volume[84696]: DEBUG cinder.utils [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.521(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:31:15 user cinder-volume[84696]: DEBUG cinder.utils [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_17f8d4f7-2dcd-4041-8d0b-5c8afebb75bf/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:31:15 user cinder-volume[84696]: DEBUG cinder.utils [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.566(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:31:15 user cinder-volume[84696]: DEBUG cinder.utils [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:31:15 user cinder-volume[84696]: DEBUG cinder.utils [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.752(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:31:15 user cinder-volume[84696]: DEBUG cinder.utils [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_62709a81-268b-4227-ba4a-070034f36414/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:31:15 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Volume reschedule parameters: True retry: {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 15:31:15 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Flow 'volume_create_manager' (3cc5893e-f96c-4dae-87bb-39268ed40abb) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:31:15 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (5c41ff22-6093-4dcb-8723-b5c4d8323022) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:31:15 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (5c41ff22-6093-4dcb-8723-b5c4d8323022) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:31:16Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesServicesTestJSON-Volume-968297362',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=5d9c86cd-3e22-4926-bfc9-e5a37f6ae76c,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='6cf93c7193f842d58d323e62e508a5eb',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:31:16Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:31:16Z,use_quota=True,user_id='225768ea457e484f8790d273f178b879',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:31:15 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (29b5c233-605f-4eac-8f7f-55e5c3c0fb1a) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:31:15 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (29b5c233-605f-4eac-8f7f-55e5c3c0fb1a) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:31:15 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (f94ca729-d3ce-478f-ab08-2a1718eb461c) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:31:15 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (f94ca729-d3ce-478f-ab08-2a1718eb461c) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': '5d9c86cd-3e22-4926-bfc9-e5a37f6ae76c', 'volume_name': 'volume-5d9c86cd-3e22-4926-bfc9-e5a37f6ae76c', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:31:15 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (15976f23-7d64-473d-9d4c-f1bba487221b) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:31:15 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (15976f23-7d64-473d-9d4c-f1bba487221b) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:31:15 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (4f3cacdc-98a3-4a79-8d25-30f63ddc05a9) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:31:15 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Volume 5d9c86cd-3e22-4926-bfc9-e5a37f6ae76c: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-5d9c86cd-3e22-4926-bfc9-e5a37f6ae76c', 'volume_size': 1} Apr 17 15:31:15 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] creating volume v_5d9c86cd-3e22-4926-bfc9-e5a37f6ae76c. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 15:31:15 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] create volume {'name': 'v_5d9c86cd-3e22-4926-bfc9-e5a37f6ae76c', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 15:31:15 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:31:15 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:31:18 user cinder-volume[84696]: DEBUG cinder.utils [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.569(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:31:18 user cinder-volume[84696]: DEBUG cinder.utils [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:31:18 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.846(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:31:18 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:31:21 user cinder-volume[84696]: DEBUG cinder.utils [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.642(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:31:21 user cinder-volume[84696]: DEBUG cinder.utils [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:31:24 user cinder-volume[84696]: DEBUG cinder.utils [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.735(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:31:24 user cinder-volume[84696]: DEBUG cinder.utils [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_17f8d4f7-2dcd-4041-8d0b-5c8afebb75bf/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:31:24 user cinder-volume[84696]: DEBUG cinder.utils [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.559(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:31:24 user cinder-volume[84696]: DEBUG cinder.utils [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:31:27 user cinder-volume[84696]: DEBUG cinder.utils [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 137.404(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:31:27 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:31:27 user cinder-volume[84696]: INFO cinder.message.api [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Creating message record for request_id = req-6e1f5cff-98a0-49d3-b32d-29a667939689 Apr 17 15:31:27 user cinder-volume[84696]: DEBUG cinder.utils [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 11.793(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:31:27 user cinder-volume[84696]: DEBUG cinder.utils [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:31:27 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 15:31:27 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (59a3d70b-a74f-4873-b767-cd5a57a63fa7) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 15:31:27 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 15:31:27 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:29:10Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesAdminNegativeTest-volume-1635380936',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=a4d96ab8-2072-4eff-b803-53b4f27d056f,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='b334c5bc33f143c484518d9bd0e750dd',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:29:10Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:29:10Z,use_quota=True,user_id='3dfadfd30c394b46abe75ec930007850',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 15:31:27 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:29:10Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesAdminNegativeTest-volume-1635380936',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=a4d96ab8-2072-4eff-b803-53b4f27d056f,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='b334c5bc33f143c484518d9bd0e750dd',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:29:10Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:29:10Z,use_quota=True,user_id='3dfadfd30c394b46abe75ec930007850',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(a4d96ab8-2072-4eff-b803-53b4f27d056f),volume_id=a4d96ab8-2072-4eff-b803-53b4f27d056f,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': 'a4d96ab8-2072-4eff-b803-53b4f27d056f', 'volume_name': 'volume-a4d96ab8-2072-4eff-b803-53b4f27d056f', 'volume_size': 1}} Apr 17 15:31:27 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'retry': {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']}, 'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': 'a4d96ab8-2072-4eff-b803-53b4f27d056f', 'volume': {'id': 'a4d96ab8-2072-4eff-b803-53b4f27d056f', '_name_id': None, 'ec2_id': None, 'user_id': '3dfadfd30c394b46abe75ec930007850', 'project_id': 'b334c5bc33f143c484518d9bd0e750dd', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-VolumesAdminNegativeTest-volume-1635380936', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T15:29:10.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-a4d96ab8-2072-4eff-b803-53b4f27d056f', 'name_id': 'a4d96ab8-2072-4eff-b803-53b4f27d056f', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumesAdminNegativeTest-volume-1635380936', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': 'b334c5bc33f143c484518d9bd0e750dd', 'qos_specs': None, 'reservations': ['09f94177-ab67-4a35-91b5-ec04b542f529', '7b3828ae-e615-4ad0-8847-f1bf1e17c182', '8e9d3d0e-b4d4-4bfc-b097-6eaec633b485', 'b33f40af-c5d7-4f4f-b730-d04956abccb8'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': '3dfadfd30c394b46abe75ec930007850', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumesAdminNegativeTest-volume-1635380936', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': 'b334c5bc33f143c484518d9bd0e750dd', 'qos_specs': None, 'reservations': ['09f94177-ab67-4a35-91b5-ec04b542f529', '7b3828ae-e615-4ad0-8847-f1bf1e17c182', '8e9d3d0e-b4d4-4bfc-b097-6eaec633b485', 'b33f40af-c5d7-4f4f-b730-d04956abccb8'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': '3dfadfd30c394b46abe75ec930007850', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': '3dfadfd30c394b46abe75ec930007850', 'metadata': {}, 'qos_specs': None}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(a4d96ab8-2072-4eff-b803-53b4f27d056f),volume_id=a4d96ab8-2072-4eff-b803-53b4f27d056f,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:29:10Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesAdminNegativeTest-volume-1635380936',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=a4d96ab8-2072-4eff-b803-53b4f27d056f,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='b334c5bc33f143c484518d9bd0e750dd',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:29:10Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:29:10Z,use_quota=True,user_id='3dfadfd30c394b46abe75ec930007850',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 15:31:27 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:29:10Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesAdminNegativeTest-volume-1635380936',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=a4d96ab8-2072-4eff-b803-53b4f27d056f,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='b334c5bc33f143c484518d9bd0e750dd',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:29:10Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:29:10Z,use_quota=True,user_id='3dfadfd30c394b46abe75ec930007850',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:29:10Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesAdminNegativeTest-volume-1635380936',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=a4d96ab8-2072-4eff-b803-53b4f27d056f,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='b334c5bc33f143c484518d9bd0e750dd',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:29:10Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:29:10Z,use_quota=True,user_id='3dfadfd30c394b46abe75ec930007850',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 15:31:27 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 15:31:27 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 15:31:27 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:31:27 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 15:31:27 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 15:31:27 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 15:31:27 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 15:31:27 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 15:31:27 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 15:31:27 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 15:31:27 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 15:31:27 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 15:31:27 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 15:31:27 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 15:31:27 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 15:31:27 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 15:31:27 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 15:31:27 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 15:31:27 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 15:31:27 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 15:31:27 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (59a3d70b-a74f-4873-b767-cd5a57a63fa7) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:31:27 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (59a3d70b-a74f-4873-b767-cd5a57a63fa7) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:31:27 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (056e72f1-43ea-47ca-80c8-6b85909c1e4e) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:31:27 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (056e72f1-43ea-47ca-80c8-6b85909c1e4e) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:31:27 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (a41cf2d4-36d2-439f-b298-2e81f71ca70a) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:31:27 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (a41cf2d4-36d2-439f-b298-2e81f71ca70a) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:31:27 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (d9af24a3-9896-49dc-ade6-3b00092bae74) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:31:27 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Updating volume a4d96ab8-2072-4eff-b803-53b4f27d056f with {'scheduled_at': datetime.datetime(2023, 4, 17, 15, 31, 27, 846284), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 15:31:27 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Volume a4d96ab8-2072-4eff-b803-53b4f27d056f: re-scheduling SchedulerAPI.create_volume attempt 1 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 15:31:27 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Volume a4d96ab8-2072-4eff-b803-53b4f27d056f: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 15:31:27 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] deleating volume v_a4d96ab8-2072-4eff-b803-53b4f27d056f {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 15:31:27 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] get snapshots for volume v_a4d96ab8-2072-4eff-b803-53b4f27d056f {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 15:31:27 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_a4d96ab8-2072-4eff-b803-53b4f27d056f/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:31:27 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_a4d96ab8-2072-4eff-b803-53b4f27d056f/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:31:27 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Volume reschedule parameters: True retry: {'num_attempts': 2, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 15:31:27 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Flow 'volume_create_manager' (317bb325-927f-4b35-a3ee-ea16297f0cc5) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:31:28 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (1c905cf7-91a9-416f-9ba6-b7270ad88166) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:31:28 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (1c905cf7-91a9-416f-9ba6-b7270ad88166) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:29:10Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesAdminNegativeTest-volume-1635380936',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=a4d96ab8-2072-4eff-b803-53b4f27d056f,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='b334c5bc33f143c484518d9bd0e750dd',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:31:28Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:31:28Z,use_quota=True,user_id='3dfadfd30c394b46abe75ec930007850',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:31:28 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (c9447cd5-5af6-4746-9bd5-3d7def3fb398) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:31:28 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (c9447cd5-5af6-4746-9bd5-3d7def3fb398) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:31:28 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (5317e8e4-e74e-4485-ba6d-c2a70b449951) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:31:28 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (5317e8e4-e74e-4485-ba6d-c2a70b449951) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': 'a4d96ab8-2072-4eff-b803-53b4f27d056f', 'volume_name': 'volume-a4d96ab8-2072-4eff-b803-53b4f27d056f', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:31:28 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (1d412cb4-8814-423b-b191-1190fc67df04) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:31:28 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (1d412cb4-8814-423b-b191-1190fc67df04) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:31:28 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (15948cbe-9b14-4690-a310-1c0808bdb5c1) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:31:28 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Volume a4d96ab8-2072-4eff-b803-53b4f27d056f: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-a4d96ab8-2072-4eff-b803-53b4f27d056f', 'volume_size': 1} Apr 17 15:31:28 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] creating volume v_a4d96ab8-2072-4eff-b803-53b4f27d056f. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 15:31:28 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] create volume {'name': 'v_a4d96ab8-2072-4eff-b803-53b4f27d056f', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 15:31:28 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:31:28 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:31:28 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Volume reschedule parameters: True retry: {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 15:31:28 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Flow 'volume_create_manager' (e20bcd95-ec7f-4e4f-b424-798fc06d1894) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:31:28 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (8b0cf505-b2d1-4665-83d7-5f45705099b6) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:31:28 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (8b0cf505-b2d1-4665-83d7-5f45705099b6) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:31:28Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesActionsTest-Volume-2081546358',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=44f909cf-7774-41f6-9f86-805456609ff8,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='a8b97ff6ed6047f0a33e74684a1c162b',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:31:28Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:31:28Z,use_quota=True,user_id='c993ceeb4a5c49eea14057e2125518ad',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:31:28 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (5ca814ee-ed16-440a-807b-02886c96097c) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:31:28 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (5ca814ee-ed16-440a-807b-02886c96097c) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:31:28 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (0c6ccd05-560d-49f4-9081-54365613f3a8) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:31:28 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (0c6ccd05-560d-49f4-9081-54365613f3a8) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': '44f909cf-7774-41f6-9f86-805456609ff8', 'volume_name': 'volume-44f909cf-7774-41f6-9f86-805456609ff8', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:31:28 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (384faa1c-3604-4481-a6db-06d70874262f) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:31:28 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (384faa1c-3604-4481-a6db-06d70874262f) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:31:28 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (5387c66d-2398-4e22-85d4-97447502a627) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:31:28 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Volume 44f909cf-7774-41f6-9f86-805456609ff8: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-44f909cf-7774-41f6-9f86-805456609ff8', 'volume_size': 1} Apr 17 15:31:28 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] creating volume v_44f909cf-7774-41f6-9f86-805456609ff8. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 15:31:28 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] create volume {'name': 'v_44f909cf-7774-41f6-9f86-805456609ff8', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 15:31:28 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:31:28 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:31:30 user cinder-volume[84696]: DEBUG cinder.utils [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.968(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:31:30 user cinder-volume[84696]: DEBUG cinder.utils [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_a4d96ab8-2072-4eff-b803-53b4f27d056f/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:31:30 user cinder-volume[84696]: DEBUG cinder.utils [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.269(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:31:30 user cinder-volume[84696]: DEBUG cinder.utils [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:31:30 user cinder-volume[84696]: DEBUG cinder.utils [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.798(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:31:30 user cinder-volume[84696]: DEBUG cinder.utils [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:31:33 user cinder-volume[84696]: DEBUG cinder.utils [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.994(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:31:33 user cinder-volume[84696]: DEBUG cinder.utils [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:31:33 user cinder-volume[84696]: DEBUG cinder.utils [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 40.181(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:31:33 user cinder-volume[84696]: DEBUG cinder.utils [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_62709a81-268b-4227-ba4a-070034f36414/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:31:33 user cinder-volume[84696]: DEBUG cinder.utils [req-cddb5331-13a7-4304-8a88-324e7cbf6185 req-43c9eb22-6970-4755-97a1-9763d442bfdd tempest-TestShelveInstance-1683612675 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 137.231(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:31:33 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-cddb5331-13a7-4304-8a88-324e7cbf6185 req-43c9eb22-6970-4755-97a1-9763d442bfdd tempest-TestShelveInstance-1683612675 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:d895d11d-6880-4418-a9ad-feca813ed5d6 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:31:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [req-cddb5331-13a7-4304-8a88-324e7cbf6185 req-43c9eb22-6970-4755-97a1-9763d442bfdd tempest-TestShelveInstance-1683612675 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 15:31:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 15:31:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 15:31:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 15:31:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 15:31:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 15:31:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 15:31:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 15:31:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 4911, in attachment_update Apr 17 15:31:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server connection_info = self._connection_create(context, Apr 17 15:31:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 4838, in _connection_create Apr 17 15:31:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self.driver.create_export(ctxt.elevated(), Apr 17 15:31:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 701, in create_export Apr 17 15:31:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self._ensure_target_volume(volume) Apr 17 15:31:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 880, in _ensure_target_volume Apr 17 15:31:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not self.ra.is_target(target_name): Apr 17 15:31:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 379, in is_target Apr 17 15:31:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if resp["error"] or resp["code"] not in (200, 201): Apr 17 15:31:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 15:31:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 15:31:33 user cinder-volume[84696]: DEBUG cinder.utils [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.045(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:31:33 user cinder-volume[84696]: DEBUG cinder.utils [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_a4d96ab8-2072-4eff-b803-53b4f27d056f/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:31:33 user cinder-volume[84696]: DEBUG cinder.utils [None req-643b65b6-f16f-4041-b9cc-be64c983a38a tempest-TestVolumeSwap-1714109745 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 168.365(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:31:33 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-643b65b6-f16f-4041-b9cc-be64c983a38a tempest-TestVolumeSwap-1714109745 None] Geting None from DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:485ea70c-618d-4373-a78f-a460914fc095/luns/v_485ea70c-618d-4373-a78f-a460914fc095 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:31:33 user cinder-volume[84696]: DEBUG cinder.utils [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.877(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:31:33 user cinder-volume[84696]: DEBUG cinder.utils [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:31:33 user cinder-volume[84696]: DEBUG cinder.utils [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.352(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:31:33 user cinder-volume[84696]: DEBUG cinder.utils [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:31:33 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=16,cinder:UPDATE=6,cinder:INSERT=2 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:31:33 user cinder-volume[84696]: DEBUG cinder.coordination [None req-643b65b6-f16f-4041-b9cc-be64c983a38a tempest-TestVolumeSwap-1714109745 None] Lock "cinder-485ea70c-618d-4373-a78f-a460914fc095-delete_volume" released by "delete_volume" :: held 168.423s {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:207}} Apr 17 15:31:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-643b65b6-f16f-4041-b9cc-be64c983a38a tempest-TestVolumeSwap-1714109745 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 15:31:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 15:31:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 15:31:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 15:31:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 15:31:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 15:31:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 15:31:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 15:31:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 203, in wrapper Apr 17 15:31:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server skip_clean = func(self, context, volume, *args, **kwargs) Apr 17 15:31:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 15:31:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 15:31:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/coordination.py", line 200, in _synchronized Apr 17 15:31:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return f(*a, **k) Apr 17 15:31:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 15:31:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 15:31:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 15:31:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 15:31:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 1035, in delete_volume Apr 17 15:31:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 15:31:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 15:31:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 15:31:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 15:31:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 15:31:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 1004, in delete_volume Apr 17 15:31:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.driver.remove_export(context, volume) Apr 17 15:31:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 720, in remove_export Apr 17 15:31:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self._remove_target_volume(volume) Apr 17 15:31:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 918, in _remove_target_volume Apr 17 15:31:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.detach_target_vol(target_name, jcom.vname(volume.id)) Apr 17 15:31:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 612, in detach_target_vol Apr 17 15:31:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if resp["code"] in (200, 201, 204): Apr 17 15:31:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 15:31:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 15:31:34 user cinder-volume[84696]: DEBUG cinder.coordination [None req-d7e6b19d-3872-4fec-88e0-811907804f16 tempest-TestVolumeSwap-1714109745 None] Acquiring lock "cinder-66000c93-eb12-4c23-8a69-71e1aa6b521f-delete_volume" by "delete_volume" {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:192}} Apr 17 15:31:34 user cinder-volume[84696]: DEBUG cinder.coordination [None req-d7e6b19d-3872-4fec-88e0-811907804f16 tempest-TestVolumeSwap-1714109745 None] Lock "cinder-66000c93-eb12-4c23-8a69-71e1aa6b521f-delete_volume" acquired by "delete_volume" :: waited 0.010s {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:196}} Apr 17 15:31:34 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-d7e6b19d-3872-4fec-88e0-811907804f16 tempest-TestVolumeSwap-1714109745 None] remove_export for volume: 66000c93-eb12-4c23-8a69-71e1aa6b521f. {{(pid=84696) remove_export /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:718}} Apr 17 15:31:34 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-d7e6b19d-3872-4fec-88e0-811907804f16 tempest-TestVolumeSwap-1714109745 None] remove export {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:912}} Apr 17 15:31:34 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-d7e6b19d-3872-4fec-88e0-811907804f16 tempest-TestVolumeSwap-1714109745 None] detach volume:Volume(_name_id=None,admin_metadata={},attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:24:08Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TestVolumeSwap-volume-1757059761',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=66000c93-eb12-4c23-8a69-71e1aa6b521f,launched_at=2023-04-17T15:24:10Z,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='00b574796d1942938ed2f348803fcf87',provider_auth='CHAP ygxXCJfw kRj6WPpS5zq2r1',provider_geometry=None,provider_id=None,provider_location='172.16.0.220:3260,1 iqn.2010-10.org.openstack:66000c93-eb12-4c23-8a69-71e1aa6b521f 0',replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:24:08Z,service_uuid='e628352e-c4cf-40eb-8389-cce40663026a',shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='deleting',terminated_at=2023-04-17T15:31:34Z,updated_at=2023-04-17T15:31:34Z,use_quota=True,user_id='4c2bfcb4119e483ab1a41e3c598ee1b0',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c) from target:iqn.2010-10.org.openstack:66000c93-eb12-4c23-8a69-71e1aa6b521f. {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:913}} Apr 17 15:31:34 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-d7e6b19d-3872-4fec-88e0-811907804f16 tempest-TestVolumeSwap-1714109745 None] detach volume v_66000c93-eb12-4c23-8a69-71e1aa6b521f from target iqn.2010-10.org.openstack:66000c93-eb12-4c23-8a69-71e1aa6b521f {{(pid=84696) detach_target_vol /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:606}} Apr 17 15:31:34 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-d7e6b19d-3872-4fec-88e0-811907804f16 tempest-TestVolumeSwap-1714109745 None] Sending pool request DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:66000c93-eb12-4c23-8a69-71e1aa6b521f/luns/v_66000c93-eb12-4c23-8a69-71e1aa6b521f {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:31:34 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-d7e6b19d-3872-4fec-88e0-811907804f16 tempest-TestVolumeSwap-1714109745 None] Sending DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:66000c93-eb12-4c23-8a69-71e1aa6b521f/luns/v_66000c93-eb12-4c23-8a69-71e1aa6b521f {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:31:36 user cinder-volume[84696]: DEBUG cinder.utils [None req-d7e6b19d-3872-4fec-88e0-811907804f16 tempest-TestVolumeSwap-1714109745 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.506(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:31:36 user cinder-volume[84696]: DEBUG cinder.utils [None req-d7e6b19d-3872-4fec-88e0-811907804f16 tempest-TestVolumeSwap-1714109745 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:66000c93-eb12-4c23-8a69-71e1aa6b521f/luns/v_66000c93-eb12-4c23-8a69-71e1aa6b521f (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:31:36 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 40.281(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:31:36 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:31:36 user cinder-volume[84696]: DEBUG cinder.utils [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.008(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:31:36 user cinder-volume[84696]: DEBUG cinder.utils [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:31:40 user cinder-volume[84696]: DEBUG cinder.utils [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 11.483(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:31:40 user cinder-volume[84696]: DEBUG cinder.utils [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:31:40 user cinder-volume[84696]: DEBUG cinder.utils [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.187(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:31:40 user cinder-volume[84696]: DEBUG cinder.utils [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_a4d96ab8-2072-4eff-b803-53b4f27d056f/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:31:40 user cinder-volume[84696]: DEBUG cinder.utils [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.015(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:31:40 user cinder-volume[84696]: DEBUG cinder.utils [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:31:40 user cinder-volume[84696]: DEBUG cinder.utils [None req-d7e6b19d-3872-4fec-88e0-811907804f16 tempest-TestVolumeSwap-1714109745 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.579(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:31:40 user cinder-volume[84696]: DEBUG cinder.utils [None req-d7e6b19d-3872-4fec-88e0-811907804f16 tempest-TestVolumeSwap-1714109745 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:66000c93-eb12-4c23-8a69-71e1aa6b521f/luns/v_66000c93-eb12-4c23-8a69-71e1aa6b521f (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:31:41 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Volume reschedule parameters: True retry: {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 15:31:41 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Flow 'volume_create_manager' (9a8b737c-677b-49d2-acfc-3ad07a3df649) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:31:41 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (e4c7a40d-0db8-4b52-973c-6b0be4467495) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:31:41 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (e4c7a40d-0db8-4b52-973c-6b0be4467495) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:31:42Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesCloneTest-Volume-297264340',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=0fed6b21-0520-4554-bee4-9540a03f9af0,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='82fce1724afa40dd914e0791c3ee69a3',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:31:42Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:31:42Z,use_quota=True,user_id='773ce6c2a0f844edaf666e190504e258',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:31:41 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (975708e3-2525-423b-9a96-cdb2cd196821) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:31:41 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (975708e3-2525-423b-9a96-cdb2cd196821) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:31:41 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (5fbc725d-6cae-4136-9318-996fe6a0b29d) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:31:41 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (5fbc725d-6cae-4136-9318-996fe6a0b29d) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'image', 'volume_id': '0fed6b21-0520-4554-bee4-9540a03f9af0', 'volume_name': 'volume-0fed6b21-0520-4554-bee4-9540a03f9af0', 'volume_size': 1, 'image_id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'image_location': (None, None), 'image_meta': {'name': 'cirros-0.5.2-x86_64-disk', 'disk_format': 'qcow2', 'container_format': 'bare', 'visibility': 'public', 'size': 16300544, 'virtual_size': 117440512, 'status': 'active', 'checksum': 'b874c39491a2377b8490f5f1e89761a4', 'protected': False, 'min_ram': 0, 'min_disk': 0, 'owner': 'a43c211b4ed542538799349489b1d884', 'os_hidden': False, 'os_hash_algo': 'sha512', 'os_hash_value': '6b813aa46bb90b4da216a4d19376593fa3f4fc7e617f03a92b7fe11e9a3981cbe8f0959dbebe36225e5f53dc4492341a4863cac4ed1ee0909f3fc78ef9c3e869', 'id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'created_at': datetime.datetime(2023, 4, 17, 15, 19, 53, tzinfo=datetime.timezone.utc), 'updated_at': datetime.datetime(2023, 4, 17, 15, 19, 55, tzinfo=datetime.timezone.utc), 'tags': [], 'file': '/v2/images/b3f0dec1-7758-44a8-82e0-76ada8837428/file', 'properties': {'hw_rng_model': 'virtio', 'owner_specified.openstack.md5': '', 'owner_specified.openstack.object': 'images/cirros-0.5.2-x86_64-disk', 'owner_specified.openstack.sha256': ''}}, 'image_service': }' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:31:41 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (d60fd341-b3cb-4d3a-9e50-9730bbd40470) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:31:41 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (d60fd341-b3cb-4d3a-9e50-9730bbd40470) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:31:41 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (57e44147-da06-48f9-9d75-0dc7f5ae6de7) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:31:41 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Volume 0fed6b21-0520-4554-bee4-9540a03f9af0: being created as image with specification: {'status': 'creating', 'volume_name': 'volume-0fed6b21-0520-4554-bee4-9540a03f9af0', 'volume_size': 1, 'image_id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'image_location': (None, None), 'image_meta': {'name': 'cirros-0.5.2-x86_64-disk', 'disk_format': 'qcow2', 'container_format': 'bare', 'visibility': 'public', 'size': 16300544, 'virtual_size': 117440512, 'status': 'active', 'checksum': 'b874c39491a2377b8490f5f1e89761a4', 'protected': False, 'min_ram': 0, 'min_disk': 0, 'owner': 'a43c211b4ed542538799349489b1d884', 'os_hidden': False, 'os_hash_algo': 'sha512', 'os_hash_value': '6b813aa46bb90b4da216a4d19376593fa3f4fc7e617f03a92b7fe11e9a3981cbe8f0959dbebe36225e5f53dc4492341a4863cac4ed1ee0909f3fc78ef9c3e869', 'id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'created_at': datetime.datetime(2023, 4, 17, 15, 19, 53, tzinfo=datetime.timezone.utc), 'updated_at': datetime.datetime(2023, 4, 17, 15, 19, 55, tzinfo=datetime.timezone.utc), 'tags': [], 'file': '/v2/images/b3f0dec1-7758-44a8-82e0-76ada8837428/file', 'properties': {'hw_rng_model': 'virtio', 'owner_specified.openstack.md5': '', 'owner_specified.openstack.object': 'images/cirros-0.5.2-x86_64-disk', 'owner_specified.openstack.sha256': ''}}, 'image_service': } Apr 17 15:31:41 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Cloning 0fed6b21-0520-4554-bee4-9540a03f9af0 from image b3f0dec1-7758-44a8-82e0-76ada8837428 at location (None, None). {{(pid=84696) _create_from_image /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1071}} Apr 17 15:31:41 user cinder-volume[84696]: DEBUG cinder.coordination [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Acquiring lock "cinder-b3f0dec1-7758-44a8-82e0-76ada8837428" by "_prepare_image_cache_entry" {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:192}} Apr 17 15:31:41 user cinder-volume[84696]: DEBUG cinder.coordination [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Lock "cinder-b3f0dec1-7758-44a8-82e0-76ada8837428" acquired by "_prepare_image_cache_entry" :: waited 0.007s {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:196}} Apr 17 15:31:41 user cinder-volume[84696]: DEBUG cinder.image.cache [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Found image-volume cache entry: {'id': 1, 'image_id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'volume_id': '006f20ae-5306-44fe-afdf-e433f79284e8', 'host': 'user@joviandss-2#joviandss-2', 'size': 1, 'image_updated_at': datetime.datetime(2023, 4, 17, 15, 19, 55), 'last_used': datetime.datetime(2023, 4, 17, 15, 31, 41, 870621)}. {{(pid=84696) get_entry /opt/stack/cinder/cinder/image/cache.py:76}} Apr 17 15:31:41 user cinder-volume[84696]: DEBUG cinder.image.cache [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Image-volume cache entry image_update_at = 2023-04-17 15:19:55+00:00, requested image updated_at = 2023-04-17 15:19:55+00:00. {{(pid=84696) _should_update_entry /opt/stack/cinder/cinder/image/cache.py:259}} Apr 17 15:31:41 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Found cache entry for image = b3f0dec1-7758-44a8-82e0-76ada8837428 on host user@joviandss-2#joviandss-2. {{(pid=84696) _prepare_image_cache_entry /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:886}} Apr 17 15:31:41 user cinder-volume[84696]: DEBUG cinder.coordination [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Lock "cinder-b3f0dec1-7758-44a8-82e0-76ada8837428" released by "_prepare_image_cache_entry" :: held 0.012s {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:207}} Apr 17 15:31:41 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Attempting to retrieve cache entry for image = b3f0dec1-7758-44a8-82e0-76ada8837428 on host user@joviandss-2#joviandss-2. {{(pid=84696) _create_from_image_cache /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:827}} Apr 17 15:31:41 user cinder-volume[84696]: DEBUG cinder.image.cache [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Found image-volume cache entry: {'id': 1, 'image_id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'volume_id': '006f20ae-5306-44fe-afdf-e433f79284e8', 'host': 'user@joviandss-2#joviandss-2', 'size': 1, 'image_updated_at': datetime.datetime(2023, 4, 17, 15, 19, 55), 'last_used': datetime.datetime(2023, 4, 17, 15, 31, 41, 902280)}. {{(pid=84696) get_entry /opt/stack/cinder/cinder/image/cache.py:76}} Apr 17 15:31:41 user cinder-volume[84696]: DEBUG cinder.image.cache [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Image-volume cache entry image_update_at = 2023-04-17 15:19:55+00:00, requested image updated_at = 2023-04-17 15:19:55+00:00. {{(pid=84696) _should_update_entry /opt/stack/cinder/cinder/image/cache.py:259}} Apr 17 15:31:41 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Creating from source image-volume 006f20ae-5306-44fe-afdf-e433f79284e8 {{(pid=84696) _create_from_image_cache /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:841}} Apr 17 15:31:41 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] cloned volume 006f20ae-5306-44fe-afdf-e433f79284e8 to 0fed6b21-0520-4554-bee4-9540a03f9af0 {{(pid=84696) create_cloned_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:521}} Apr 17 15:31:41 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] cloning v_006f20ae-5306-44fe-afdf-e433f79284e8 to v_0fed6b21-0520-4554-bee4-9540a03f9af0 {{(pid=84696) _clone_object /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:456}} Apr 17 15:31:41 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] create snapshot v_0fed6b21-0520-4554-bee4-9540a03f9af0 {{(pid=84696) create_snapshot /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:635}} Apr 17 15:31:41 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:31:41 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:31:43 user cinder-volume[84696]: DEBUG cinder.utils [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.991(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:31:43 user cinder-volume[84696]: DEBUG cinder.utils [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:31:43 user cinder-volume[84696]: DEBUG cinder.utils [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 40.170(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:31:43 user cinder-volume[84696]: DEBUG cinder.utils [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_17f8d4f7-2dcd-4041-8d0b-5c8afebb75bf/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:31:43 user cinder-volume[84696]: DEBUG cinder.utils [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 1.208(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:31:43 user cinder-volume[84696]: DEBUG cinder.utils [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:31:44 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=26,cinder:UPDATE=7,cinder:DELETE=1 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:31:46 user cinder-volume[84696]: DEBUG cinder.utils [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 4.276(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:31:46 user cinder-volume[84696]: DEBUG cinder.utils [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:31:46 user cinder-volume[84696]: DEBUG cinder.utils [None req-d7e6b19d-3872-4fec-88e0-811907804f16 tempest-TestVolumeSwap-1714109745 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 11.720(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:31:46 user cinder-volume[84696]: DEBUG cinder.utils [None req-d7e6b19d-3872-4fec-88e0-811907804f16 tempest-TestVolumeSwap-1714109745 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:66000c93-eb12-4c23-8a69-71e1aa6b521f/luns/v_66000c93-eb12-4c23-8a69-71e1aa6b521f (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:31:49 user cinder-volume[84696]: DEBUG cinder.utils [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.227(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:31:49 user cinder-volume[84696]: DEBUG cinder.utils [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:31:49 user cinder-volume[84696]: DEBUG cinder.utils [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.401(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:31:49 user cinder-volume[84696]: DEBUG cinder.utils [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_a4d96ab8-2072-4eff-b803-53b4f27d056f/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:31:49 user cinder-volume[84696]: DEBUG cinder.utils [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 20.701(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:31:49 user cinder-volume[84696]: DEBUG cinder.utils [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:31:52 user cinder-volume[84696]: DEBUG cinder.utils [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 10.421(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:31:52 user cinder-volume[84696]: DEBUG cinder.utils [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:31:55 user cinder-volume[84696]: DEBUG cinder.utils [None req-d7e6b19d-3872-4fec-88e0-811907804f16 tempest-TestVolumeSwap-1714109745 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 20.934(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:31:55 user cinder-volume[84696]: DEBUG cinder.utils [None req-d7e6b19d-3872-4fec-88e0-811907804f16 tempest-TestVolumeSwap-1714109745 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:66000c93-eb12-4c23-8a69-71e1aa6b521f/luns/v_66000c93-eb12-4c23-8a69-71e1aa6b521f (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:31:55 user cinder-volume[84696]: DEBUG cinder.utils [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.435(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:31:55 user cinder-volume[84696]: DEBUG cinder.utils [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:31:58 user cinder-volume[84696]: DEBUG cinder.utils [req-38bdff97-cca4-4bb8-8df0-3261c844629a req-073af777-dfca-4d61-85d6-9ed616a692d0 tempest-TestStampPattern-173447269 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 136.294(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:31:58 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-38bdff97-cca4-4bb8-8df0-3261c844629a req-073af777-dfca-4d61-85d6-9ed616a692d0 tempest-TestStampPattern-173447269 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:a9c177a1-61f0-4201-bb8d-7f18c9af396c {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:31:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [req-38bdff97-cca4-4bb8-8df0-3261c844629a req-073af777-dfca-4d61-85d6-9ed616a692d0 tempest-TestStampPattern-173447269 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 15:31:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 15:31:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 15:31:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 15:31:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 15:31:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 15:31:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 15:31:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 15:31:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 4911, in attachment_update Apr 17 15:31:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server connection_info = self._connection_create(context, Apr 17 15:31:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 4838, in _connection_create Apr 17 15:31:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self.driver.create_export(ctxt.elevated(), Apr 17 15:31:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 701, in create_export Apr 17 15:31:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self._ensure_target_volume(volume) Apr 17 15:31:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 880, in _ensure_target_volume Apr 17 15:31:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not self.ra.is_target(target_name): Apr 17 15:31:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 379, in is_target Apr 17 15:31:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if resp["error"] or resp["code"] not in (200, 201): Apr 17 15:31:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 15:31:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 15:32:01 user cinder-volume[84696]: DEBUG cinder.utils [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 19.636(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:32:01 user cinder-volume[84696]: DEBUG cinder.utils [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:32:04 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=16,cinder:UPDATE=4 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:32:07 user cinder-volume[84696]: DEBUG cinder.utils [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.131(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:32:07 user cinder-volume[84696]: DEBUG cinder.utils [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:32:07 user cinder-volume[84696]: DEBUG cinder.utils [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.833(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:32:07 user cinder-volume[84696]: DEBUG cinder.utils [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_a4d96ab8-2072-4eff-b803-53b4f27d056f/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:32:07 user cinder-volume[84696]: DEBUG cinder.utils [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.662(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:32:07 user cinder-volume[84696]: DEBUG cinder.utils [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:32:07 user cinder-volume[84696]: DEBUG cinder.utils [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 73.975(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:32:07 user cinder-volume[84696]: DEBUG cinder.utils [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_62709a81-268b-4227-ba4a-070034f36414/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:32:07 user cinder-volume[84696]: DEBUG cinder.utils [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 73.792(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:32:07 user cinder-volume[84696]: DEBUG cinder.utils [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:32:10 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 74.068(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:32:10 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:32:13 user cinder-volume[84696]: DEBUG cinder.utils [None req-d7e6b19d-3872-4fec-88e0-811907804f16 tempest-TestVolumeSwap-1714109745 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.370(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:32:13 user cinder-volume[84696]: DEBUG cinder.utils [None req-d7e6b19d-3872-4fec-88e0-811907804f16 tempest-TestVolumeSwap-1714109745 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:66000c93-eb12-4c23-8a69-71e1aa6b521f/luns/v_66000c93-eb12-4c23-8a69-71e1aa6b521f (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:32:14 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=4,cinder:UPDATE=2 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:32:16 user cinder-volume[84696]: DEBUG cinder.utils [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 73.782(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:32:16 user cinder-volume[84696]: DEBUG cinder.utils [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:32:16 user cinder-volume[84696]: DEBUG cinder.utils [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 73.960(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:32:16 user cinder-volume[84696]: DEBUG cinder.utils [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_17f8d4f7-2dcd-4041-8d0b-5c8afebb75bf/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:32:19 user cinder-volume[84696]: DEBUG cinder.utils [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 38.068(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:32:19 user cinder-volume[84696]: DEBUG cinder.utils [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:32:29 user cinder-volume[84696]: DEBUG cinder.utils [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 73.550(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:32:29 user cinder-volume[84696]: DEBUG cinder.utils [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:32:42 user cinder-volume[84696]: DEBUG cinder.utils [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 74.904(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:32:42 user cinder-volume[84696]: DEBUG cinder.utils [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_a4d96ab8-2072-4eff-b803-53b4f27d056f/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:32:42 user cinder-volume[84696]: DEBUG cinder.utils [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 74.205(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:32:42 user cinder-volume[84696]: DEBUG cinder.utils [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:32:42 user cinder-volume[84696]: DEBUG cinder.utils [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 74.734(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:32:42 user cinder-volume[84696]: DEBUG cinder.utils [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:32:44 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=2,cinder:UPDATE=1 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:32:44 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=7,cinder:UPDATE=1 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:32:48 user cinder-volume[84696]: DEBUG cinder.utils [None req-d7e6b19d-3872-4fec-88e0-811907804f16 tempest-TestVolumeSwap-1714109745 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 74.438(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:32:48 user cinder-volume[84696]: DEBUG cinder.utils [None req-d7e6b19d-3872-4fec-88e0-811907804f16 tempest-TestVolumeSwap-1714109745 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:66000c93-eb12-4c23-8a69-71e1aa6b521f/luns/v_66000c93-eb12-4c23-8a69-71e1aa6b521f (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:32:55 user cinder-volume[84696]: DEBUG cinder.utils [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 73.140(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:32:55 user cinder-volume[84696]: DEBUG cinder.utils [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:33:04 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=2,cinder:UPDATE=2 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:33:14 user cinder-volume[84696]: DEBUG cinder.utils [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 140.858(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:33:14 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:33:14 user cinder-volume[84696]: INFO cinder.message.api [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Creating message record for request_id = req-e7b5301d-81ea-4ab5-9d69-766347595e11 Apr 17 15:33:14 user cinder-volume[84696]: DEBUG cinder.utils [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 141.048(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:33:14 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_62709a81-268b-4227-ba4a-070034f36414/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:33:14 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (35ddcbc2-13dd-4fbe-8faa-513f4b35e511) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 15:33:14 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (91b35ce4-f3c0-43ac-a18f-1a39667c87f7) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:33:14 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (91b35ce4-f3c0-43ac-a18f-1a39667c87f7) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:33:14 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Flow 'volume_create_manager' (5996bb83-4d49-4e7f-98f8-dbad127cce17) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 15:33:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 15:33:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 15:33:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 15:33:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 15:33:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 15:33:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 15:33:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 15:33:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 15:33:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 15:33:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 15:33:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 15:33:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 15:33:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 15:33:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 15:33:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 15:33:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 15:33:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 15:33:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 15:33:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 15:33:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 15:33:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 15:33:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 15:33:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 15:33:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 15:33:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:33:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 15:33:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 15:33:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 15:33:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 15:33:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 15:33:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 15:33:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 15:33:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 15:33:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 15:33:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 15:33:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 15:33:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 15:33:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 15:33:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 15:33:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 15:33:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 15:33:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 15:33:14 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 15:33:14 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=8,cinder:UPDATE=2,cinder:INSERT=1 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:33:14 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (90879f34-89e0-49ff-9cee-80b45a763611) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 15:33:14 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 15:33:14 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:28:12Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesActionsTest-Volume-620204537',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=62709a81-268b-4227-ba4a-070034f36414,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='a8b97ff6ed6047f0a33e74684a1c162b',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:30:54Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:30:54Z,use_quota=True,user_id='c993ceeb4a5c49eea14057e2125518ad',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 15:33:14 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:28:12Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesActionsTest-Volume-620204537',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=62709a81-268b-4227-ba4a-070034f36414,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='a8b97ff6ed6047f0a33e74684a1c162b',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:30:54Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:30:54Z,use_quota=True,user_id='c993ceeb4a5c49eea14057e2125518ad',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(62709a81-268b-4227-ba4a-070034f36414),volume_id=62709a81-268b-4227-ba4a-070034f36414,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': '62709a81-268b-4227-ba4a-070034f36414', 'volume_name': 'volume-62709a81-268b-4227-ba4a-070034f36414', 'volume_size': 1}} Apr 17 15:33:14 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': '62709a81-268b-4227-ba4a-070034f36414', 'volume': {'id': '62709a81-268b-4227-ba4a-070034f36414', '_name_id': None, 'ec2_id': None, 'user_id': 'c993ceeb4a5c49eea14057e2125518ad', 'project_id': 'a8b97ff6ed6047f0a33e74684a1c162b', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-VolumesActionsTest-Volume-620204537', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'admin_metadata': {}, 'glance_metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T15:28:12.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-62709a81-268b-4227-ba4a-070034f36414', 'name_id': '62709a81-268b-4227-ba4a-070034f36414', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumesActionsTest-Volume-620204537', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': 'a8b97ff6ed6047f0a33e74684a1c162b', 'qos_specs': None, 'reservations': ['1b325ba3-edbb-4c50-91e6-5be6258c7eee', '4eb6a10f-f088-4780-a8d5-294b078de0ce', 'ede206e9-4b4d-4f26-8a58-97a099b605d5', '2beaf0e1-5633-41fc-ad41-398657c8ddea'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'c993ceeb4a5c49eea14057e2125518ad', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumesActionsTest-Volume-620204537', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': 'a8b97ff6ed6047f0a33e74684a1c162b', 'qos_specs': None, 'reservations': ['1b325ba3-edbb-4c50-91e6-5be6258c7eee', '4eb6a10f-f088-4780-a8d5-294b078de0ce', 'ede206e9-4b4d-4f26-8a58-97a099b605d5', '2beaf0e1-5633-41fc-ad41-398657c8ddea'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'c993ceeb4a5c49eea14057e2125518ad', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': 'c993ceeb4a5c49eea14057e2125518ad', 'metadata': {}, 'qos_specs': None, 'retry': {'num_attempts': 2, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']}}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(62709a81-268b-4227-ba4a-070034f36414),volume_id=62709a81-268b-4227-ba4a-070034f36414,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:28:12Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesActionsTest-Volume-620204537',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=62709a81-268b-4227-ba4a-070034f36414,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='a8b97ff6ed6047f0a33e74684a1c162b',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:30:54Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:30:54Z,use_quota=True,user_id='c993ceeb4a5c49eea14057e2125518ad',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 15:33:14 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:28:12Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesActionsTest-Volume-620204537',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=62709a81-268b-4227-ba4a-070034f36414,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='a8b97ff6ed6047f0a33e74684a1c162b',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:30:54Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:30:54Z,use_quota=True,user_id='c993ceeb4a5c49eea14057e2125518ad',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:28:12Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesActionsTest-Volume-620204537',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=62709a81-268b-4227-ba4a-070034f36414,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='a8b97ff6ed6047f0a33e74684a1c162b',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:30:54Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:30:54Z,use_quota=True,user_id='c993ceeb4a5c49eea14057e2125518ad',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 15:33:14 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 15:33:14 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 15:33:14 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:33:14 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 15:33:14 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 15:33:14 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 15:33:14 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 15:33:14 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 15:33:14 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 15:33:14 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 15:33:14 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 15:33:14 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 15:33:14 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 15:33:14 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 15:33:14 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 15:33:14 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 15:33:14 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 15:33:14 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 15:33:14 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 15:33:14 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 15:33:14 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (90879f34-89e0-49ff-9cee-80b45a763611) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:33:14 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (90879f34-89e0-49ff-9cee-80b45a763611) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:33:14 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (25f6a879-3c9a-40e5-bf1b-dd26fbb692a4) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:33:14 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (25f6a879-3c9a-40e5-bf1b-dd26fbb692a4) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:33:14 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (d2d8b72d-5b36-4789-b831-089b51523205) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:33:14 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (d2d8b72d-5b36-4789-b831-089b51523205) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:33:14 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (f282ab50-6be5-4b8b-a8ff-74df520036f3) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:33:14 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Updating volume 62709a81-268b-4227-ba4a-070034f36414 with {'scheduled_at': datetime.datetime(2023, 4, 17, 15, 33, 14, 875393), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 15:33:14 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Volume 62709a81-268b-4227-ba4a-070034f36414: re-scheduling SchedulerAPI.create_volume attempt 2 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 15:33:14 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Volume 62709a81-268b-4227-ba4a-070034f36414: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 15:33:14 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] deleating volume v_62709a81-268b-4227-ba4a-070034f36414 {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 15:33:14 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] get snapshots for volume v_62709a81-268b-4227-ba4a-070034f36414 {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 15:33:14 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_62709a81-268b-4227-ba4a-070034f36414/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:33:14 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_62709a81-268b-4227-ba4a-070034f36414/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:33:15 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Volume reschedule parameters: True retry: {'num_attempts': 3, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 15:33:15 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Flow 'volume_create_manager' (e73c3c8f-9a3f-40bc-ae86-670b4fc8493c) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:33:15 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (c50526d0-27c3-4cfd-9573-b6633f14c23f) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:33:15 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (c50526d0-27c3-4cfd-9573-b6633f14c23f) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:28:12Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesActionsTest-Volume-620204537',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=62709a81-268b-4227-ba4a-070034f36414,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='a8b97ff6ed6047f0a33e74684a1c162b',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:33:15Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:33:15Z,use_quota=True,user_id='c993ceeb4a5c49eea14057e2125518ad',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:33:15 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (416c7a27-3bbf-4426-ba12-c01dcd02cce5) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:33:15 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (416c7a27-3bbf-4426-ba12-c01dcd02cce5) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:33:15 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (ac1057f3-d93d-42c7-b785-83e64bf096e3) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:33:15 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (ac1057f3-d93d-42c7-b785-83e64bf096e3) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': '62709a81-268b-4227-ba4a-070034f36414', 'volume_name': 'volume-62709a81-268b-4227-ba4a-070034f36414', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:33:15 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (96d31545-d396-468c-a380-730993851e93) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:33:15 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (96d31545-d396-468c-a380-730993851e93) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:33:15 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (adab2f28-31c5-4cfd-8a75-4410878080db) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:33:15 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Volume 62709a81-268b-4227-ba4a-070034f36414: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-62709a81-268b-4227-ba4a-070034f36414', 'volume_size': 1} Apr 17 15:33:15 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] creating volume v_62709a81-268b-4227-ba4a-070034f36414. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 15:33:15 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] create volume {'name': 'v_62709a81-268b-4227-ba4a-070034f36414', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 15:33:15 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:33:15 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:33:17 user cinder-volume[84696]: DEBUG cinder.utils [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.947(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:33:17 user cinder-volume[84696]: DEBUG cinder.utils [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_62709a81-268b-4227-ba4a-070034f36414/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:33:17 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 141.141(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:33:17 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:33:17 user cinder-volume[84696]: ERROR oslo_service.periodic_task [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Error during VolumeManager.publish_service_capabilities: TypeError: 'NoneType' object is not subscriptable Apr 17 15:33:17 user cinder-volume[84696]: ERROR oslo_service.periodic_task Traceback (most recent call last): Apr 17 15:33:17 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/usr/local/lib/python3.10/dist-packages/oslo_service/periodic_task.py", line 216, in run_periodic_tasks Apr 17 15:33:17 user cinder-volume[84696]: ERROR oslo_service.periodic_task task(self, context) Apr 17 15:33:17 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/manager.py", line 2876, in publish_service_capabilities Apr 17 15:33:17 user cinder-volume[84696]: ERROR oslo_service.periodic_task self._report_driver_status(context) Apr 17 15:33:17 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/manager.py", line 2755, in _report_driver_status Apr 17 15:33:17 user cinder-volume[84696]: ERROR oslo_service.periodic_task volume_stats = get_stats() Apr 17 15:33:17 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/usr/local/lib/python3.10/dist-packages/oslo_utils/timeutils.py", line 319, in wrapper Apr 17 15:33:17 user cinder-volume[84696]: ERROR oslo_service.periodic_task result = func(*args, **kwargs) Apr 17 15:33:17 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/manager.py", line 2753, in get_stats Apr 17 15:33:17 user cinder-volume[84696]: ERROR oslo_service.periodic_task return self.driver.get_volume_stats(refresh=True) Apr 17 15:33:17 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/driver.py", line 666, in get_volume_stats Apr 17 15:33:17 user cinder-volume[84696]: ERROR oslo_service.periodic_task self._update_volume_stats() Apr 17 15:33:17 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 726, in _update_volume_stats Apr 17 15:33:17 user cinder-volume[84696]: ERROR oslo_service.periodic_task pool_stats = self.ra.get_pool_stats() Apr 17 15:33:17 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 897, in get_pool_stats Apr 17 15:33:17 user cinder-volume[84696]: ERROR oslo_service.periodic_task if not resp["error"] and resp["code"] == 200: Apr 17 15:33:17 user cinder-volume[84696]: ERROR oslo_service.periodic_task TypeError: 'NoneType' object is not subscriptable Apr 17 15:33:17 user cinder-volume[84696]: ERROR oslo_service.periodic_task Apr 17 15:33:17 user cinder-volume[84696]: WARNING oslo.service.loopingcall [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Function 'cinder.service.Service.periodic_tasks' run outlasted interval by 81.15 sec Apr 17 15:33:17 user cinder-volume[84696]: DEBUG cinder.utils [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.768(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:33:17 user cinder-volume[84696]: DEBUG cinder.utils [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:33:17 user cinder-volume[84696]: DEBUG oslo_service.periodic_task [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Running periodic task VolumeManager.publish_service_capabilities {{(pid=84696) run_periodic_tasks /usr/local/lib/python3.10/dist-packages/oslo_service/periodic_task.py:210}} Apr 17 15:33:17 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Updating volume stats {{(pid=84696) _update_volume_stats /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:724}} Apr 17 15:33:17 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Get pool Pool-0 fsprops {{(pid=84696) get_pool_stats /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:894}} Apr 17 15:33:17 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:33:17 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:33:20 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 3.063(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:33:20 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:33:20 user cinder-volume[84696]: DEBUG cinder.utils [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.023(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:33:20 user cinder-volume[84696]: DEBUG cinder.utils [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_62709a81-268b-4227-ba4a-070034f36414/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:33:20 user cinder-volume[84696]: DEBUG cinder.utils [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.840(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:33:20 user cinder-volume[84696]: DEBUG cinder.utils [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:33:23 user cinder-volume[84696]: DEBUG cinder.utils [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 141.031(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:33:23 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_17f8d4f7-2dcd-4041-8d0b-5c8afebb75bf/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:33:24 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (ba0a21a5-fc9a-4815-9416-e53538e13902) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 15:33:24 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (bea28282-7b49-4c64-877a-2f9a3ec04a85) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:33:24 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (bea28282-7b49-4c64-877a-2f9a3ec04a85) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:33:24 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Flow 'volume_create_manager' (fa9685c1-a073-4d82-8bb0-ee053948ec78) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 15:33:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 15:33:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 15:33:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 15:33:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 15:33:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 15:33:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 15:33:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 15:33:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 15:33:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 15:33:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 15:33:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 15:33:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 15:33:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 15:33:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 15:33:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 15:33:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 15:33:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 15:33:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 15:33:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 15:33:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 15:33:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 15:33:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 15:33:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 15:33:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 15:33:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:33:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 15:33:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 15:33:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 15:33:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 15:33:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 15:33:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 15:33:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 15:33:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 15:33:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 15:33:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 15:33:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 15:33:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 15:33:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 15:33:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 15:33:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 15:33:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 15:33:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 15:33:24 user cinder-volume[84696]: DEBUG cinder.utils [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 140.877(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:33:24 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:33:24 user cinder-volume[84696]: INFO cinder.message.api [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Creating message record for request_id = req-051b1fdc-54da-4587-a61f-deb7faa24c2c Apr 17 15:33:24 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.160(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:33:24 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:33:24 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 15:33:24 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (eeba2ba2-6604-4eb9-8edd-18cf42c1cbbe) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 15:33:24 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 15:33:24 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:28:22Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-SnapshotMetadataTestJSON-Volume-902610845',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=17f8d4f7-2dcd-4041-8d0b-5c8afebb75bf,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='a4653bf4d2914c8d956e86ad3405051d',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:31:03Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:31:03Z,use_quota=True,user_id='5598ef2700af47d9bcf6fadea4460cb2',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 15:33:24 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:28:22Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-SnapshotMetadataTestJSON-Volume-902610845',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=17f8d4f7-2dcd-4041-8d0b-5c8afebb75bf,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='a4653bf4d2914c8d956e86ad3405051d',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:31:03Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:31:03Z,use_quota=True,user_id='5598ef2700af47d9bcf6fadea4460cb2',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(17f8d4f7-2dcd-4041-8d0b-5c8afebb75bf),volume_id=17f8d4f7-2dcd-4041-8d0b-5c8afebb75bf,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': '17f8d4f7-2dcd-4041-8d0b-5c8afebb75bf', 'volume_name': 'volume-17f8d4f7-2dcd-4041-8d0b-5c8afebb75bf', 'volume_size': 1}} Apr 17 15:33:24 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': '17f8d4f7-2dcd-4041-8d0b-5c8afebb75bf', 'volume': {'id': '17f8d4f7-2dcd-4041-8d0b-5c8afebb75bf', '_name_id': None, 'ec2_id': None, 'user_id': '5598ef2700af47d9bcf6fadea4460cb2', 'project_id': 'a4653bf4d2914c8d956e86ad3405051d', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-SnapshotMetadataTestJSON-Volume-902610845', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'admin_metadata': {}, 'glance_metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T15:28:22.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-17f8d4f7-2dcd-4041-8d0b-5c8afebb75bf', 'name_id': '17f8d4f7-2dcd-4041-8d0b-5c8afebb75bf', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-SnapshotMetadataTestJSON-Volume-902610845', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': 'a4653bf4d2914c8d956e86ad3405051d', 'qos_specs': None, 'reservations': ['ab8ffe08-13a2-4be5-8c18-55343d579bf3', 'd62e062b-7f8a-4b18-8a96-5ae0097970db', '0311c219-244b-45f9-903f-bdbceee93caf', 'f31b8283-7526-44ae-bd04-3726c40be823'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': '5598ef2700af47d9bcf6fadea4460cb2', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-SnapshotMetadataTestJSON-Volume-902610845', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': 'a4653bf4d2914c8d956e86ad3405051d', 'qos_specs': None, 'reservations': ['ab8ffe08-13a2-4be5-8c18-55343d579bf3', 'd62e062b-7f8a-4b18-8a96-5ae0097970db', '0311c219-244b-45f9-903f-bdbceee93caf', 'f31b8283-7526-44ae-bd04-3726c40be823'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': '5598ef2700af47d9bcf6fadea4460cb2', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': '5598ef2700af47d9bcf6fadea4460cb2', 'metadata': {}, 'qos_specs': None, 'retry': {'num_attempts': 2, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']}}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(17f8d4f7-2dcd-4041-8d0b-5c8afebb75bf),volume_id=17f8d4f7-2dcd-4041-8d0b-5c8afebb75bf,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:28:22Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-SnapshotMetadataTestJSON-Volume-902610845',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=17f8d4f7-2dcd-4041-8d0b-5c8afebb75bf,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='a4653bf4d2914c8d956e86ad3405051d',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:31:03Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:31:03Z,use_quota=True,user_id='5598ef2700af47d9bcf6fadea4460cb2',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 15:33:24 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:28:22Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-SnapshotMetadataTestJSON-Volume-902610845',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=17f8d4f7-2dcd-4041-8d0b-5c8afebb75bf,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='a4653bf4d2914c8d956e86ad3405051d',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:31:03Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:31:03Z,use_quota=True,user_id='5598ef2700af47d9bcf6fadea4460cb2',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:28:22Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-SnapshotMetadataTestJSON-Volume-902610845',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=17f8d4f7-2dcd-4041-8d0b-5c8afebb75bf,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='a4653bf4d2914c8d956e86ad3405051d',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:31:03Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:31:03Z,use_quota=True,user_id='5598ef2700af47d9bcf6fadea4460cb2',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 15:33:24 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 15:33:24 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 15:33:24 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:33:24 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 15:33:24 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 15:33:24 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 15:33:24 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 15:33:24 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 15:33:24 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 15:33:24 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 15:33:24 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 15:33:24 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 15:33:24 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 15:33:24 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 15:33:24 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 15:33:24 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 15:33:24 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 15:33:24 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 15:33:24 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 15:33:24 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 15:33:24 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (eeba2ba2-6604-4eb9-8edd-18cf42c1cbbe) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:33:24 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (eeba2ba2-6604-4eb9-8edd-18cf42c1cbbe) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:33:24 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (0fd09b44-7dcc-45d0-80e7-e05317bdc25c) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:33:24 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (0fd09b44-7dcc-45d0-80e7-e05317bdc25c) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:33:24 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (d6f3f11b-ca43-4d6d-930a-d0dab99eef4f) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:33:24 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (d6f3f11b-ca43-4d6d-930a-d0dab99eef4f) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:33:24 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (0cff7168-848c-4fc6-98f8-492aeec7e936) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:33:24 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Updating volume 17f8d4f7-2dcd-4041-8d0b-5c8afebb75bf with {'scheduled_at': datetime.datetime(2023, 4, 17, 15, 33, 24, 85330), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 15:33:24 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Volume 17f8d4f7-2dcd-4041-8d0b-5c8afebb75bf: re-scheduling SchedulerAPI.create_volume attempt 2 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 15:33:24 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Volume 17f8d4f7-2dcd-4041-8d0b-5c8afebb75bf: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 15:33:24 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] deleating volume v_17f8d4f7-2dcd-4041-8d0b-5c8afebb75bf {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 15:33:24 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] get snapshots for volume v_17f8d4f7-2dcd-4041-8d0b-5c8afebb75bf {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 15:33:24 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_17f8d4f7-2dcd-4041-8d0b-5c8afebb75bf/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:33:24 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_17f8d4f7-2dcd-4041-8d0b-5c8afebb75bf/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:33:24 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Volume reschedule parameters: True retry: {'num_attempts': 3, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 15:33:24 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Flow 'volume_create_manager' (cc63b5b8-0ad3-4d49-9a4f-0f06a1267879) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:33:24 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (24cc5e8e-cc38-409a-b469-bf86389089ce) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:33:24 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (24cc5e8e-cc38-409a-b469-bf86389089ce) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:28:22Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-SnapshotMetadataTestJSON-Volume-902610845',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=17f8d4f7-2dcd-4041-8d0b-5c8afebb75bf,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='a4653bf4d2914c8d956e86ad3405051d',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:33:24Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:33:24Z,use_quota=True,user_id='5598ef2700af47d9bcf6fadea4460cb2',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:33:24 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (68a53ea2-b4bc-426e-bf76-3d7cdf44638d) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:33:24 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (68a53ea2-b4bc-426e-bf76-3d7cdf44638d) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:33:24 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (ae62ee66-a4cf-459b-a80f-168a160768f7) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:33:24 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (ae62ee66-a4cf-459b-a80f-168a160768f7) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': '17f8d4f7-2dcd-4041-8d0b-5c8afebb75bf', 'volume_name': 'volume-17f8d4f7-2dcd-4041-8d0b-5c8afebb75bf', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:33:24 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (683ddc71-fc21-4551-b496-26f8bdc771ea) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:33:24 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (683ddc71-fc21-4551-b496-26f8bdc771ea) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:33:24 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (c135b8bf-a00b-4c1a-bc39-6239684fdf33) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:33:24 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Volume 17f8d4f7-2dcd-4041-8d0b-5c8afebb75bf: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-17f8d4f7-2dcd-4041-8d0b-5c8afebb75bf', 'volume_size': 1} Apr 17 15:33:24 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] creating volume v_17f8d4f7-2dcd-4041-8d0b-5c8afebb75bf. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 15:33:24 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] create volume {'name': 'v_17f8d4f7-2dcd-4041-8d0b-5c8afebb75bf', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 15:33:24 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:33:24 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:33:27 user cinder-volume[84696]: DEBUG cinder.utils [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.768(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:33:27 user cinder-volume[84696]: DEBUG cinder.utils [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:33:27 user cinder-volume[84696]: DEBUG cinder.utils [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.957(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:33:27 user cinder-volume[84696]: DEBUG cinder.utils [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_17f8d4f7-2dcd-4041-8d0b-5c8afebb75bf/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:33:27 user cinder-volume[84696]: DEBUG cinder.utils [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 11.983(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:33:27 user cinder-volume[84696]: DEBUG cinder.utils [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:33:27 user cinder-volume[84696]: DEBUG cinder.utils [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.168(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:33:27 user cinder-volume[84696]: DEBUG cinder.utils [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_62709a81-268b-4227-ba4a-070034f36414/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:33:30 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.279(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:33:30 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:33:30 user cinder-volume[84696]: DEBUG cinder.utils [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.842(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:33:30 user cinder-volume[84696]: DEBUG cinder.utils [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:33:30 user cinder-volume[84696]: DEBUG cinder.utils [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.028(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:33:30 user cinder-volume[84696]: DEBUG cinder.utils [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_17f8d4f7-2dcd-4041-8d0b-5c8afebb75bf/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:33:30 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Volume reschedule parameters: True retry: {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 15:33:30 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Flow 'volume_create_manager' (16942f14-554a-4c43-8393-9f00cb7324a1) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:33:30 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (f427bcb0-0446-4074-84df-e43cd57b3d83) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:33:30 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (f427bcb0-0446-4074-84df-e43cd57b3d83) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:33:31Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-ServersNegativeTestJSON-volume-1746588226',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=8cc35ab1-363f-4ad6-9a37-239d2b1ea996,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='d2dd48f279404859bebcc19bcc272915',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:33:31Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:33:31Z,use_quota=True,user_id='b0075b26d6044a50981d29df0d863959',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:33:30 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (0d5a3677-e116-474e-95e4-84189a7a0841) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:33:30 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (0d5a3677-e116-474e-95e4-84189a7a0841) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:33:30 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (c709a820-1c0d-4a42-b0d0-51cfb8321b74) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:33:30 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (c709a820-1c0d-4a42-b0d0-51cfb8321b74) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': '8cc35ab1-363f-4ad6-9a37-239d2b1ea996', 'volume_name': 'volume-8cc35ab1-363f-4ad6-9a37-239d2b1ea996', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:33:30 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (fed44b7b-4a47-4763-ac5e-6723c995f5c7) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:33:30 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (fed44b7b-4a47-4763-ac5e-6723c995f5c7) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:33:30 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (c9144115-8b7f-4295-815d-4c8e8626f5fe) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:33:30 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Volume 8cc35ab1-363f-4ad6-9a37-239d2b1ea996: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-8cc35ab1-363f-4ad6-9a37-239d2b1ea996', 'volume_size': 1} Apr 17 15:33:30 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] creating volume v_8cc35ab1-363f-4ad6-9a37-239d2b1ea996. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 15:33:30 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] create volume {'name': 'v_8cc35ab1-363f-4ad6-9a37-239d2b1ea996', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 15:33:30 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:33:30 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:33:33 user cinder-volume[84696]: DEBUG cinder.utils [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.440(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:33:33 user cinder-volume[84696]: DEBUG cinder.utils [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:33:36 user cinder-volume[84696]: DEBUG cinder.utils [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.195(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:33:36 user cinder-volume[84696]: DEBUG cinder.utils [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:33:36 user cinder-volume[84696]: DEBUG cinder.utils [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.515(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:33:36 user cinder-volume[84696]: DEBUG cinder.utils [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:33:36 user cinder-volume[84696]: DEBUG cinder.utils [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 140.313(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:33:36 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:33:36 user cinder-volume[84696]: INFO cinder.message.api [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Creating message record for request_id = req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 Apr 17 15:33:36 user cinder-volume[84696]: DEBUG cinder.utils [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.197(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:33:36 user cinder-volume[84696]: DEBUG cinder.utils [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_17f8d4f7-2dcd-4041-8d0b-5c8afebb75bf/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:33:36 user cinder-volume[84696]: DEBUG cinder.utils [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.013(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:33:36 user cinder-volume[84696]: DEBUG cinder.utils [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:33:36 user cinder-volume[84696]: DEBUG cinder.utils [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.409(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:33:36 user cinder-volume[84696]: DEBUG cinder.utils [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_62709a81-268b-4227-ba4a-070034f36414/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:33:36 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 15:33:36 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=17,cinder:UPDATE=5,cinder:INSERT=2 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:33:36 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (4f3cacdc-98a3-4a79-8d25-30f63ddc05a9) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 15:33:36 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 15:33:36 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:31:16Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesServicesTestJSON-Volume-968297362',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=5d9c86cd-3e22-4926-bfc9-e5a37f6ae76c,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='6cf93c7193f842d58d323e62e508a5eb',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:31:16Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:31:16Z,use_quota=True,user_id='225768ea457e484f8790d273f178b879',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 15:33:36 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:31:16Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesServicesTestJSON-Volume-968297362',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=5d9c86cd-3e22-4926-bfc9-e5a37f6ae76c,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='6cf93c7193f842d58d323e62e508a5eb',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:31:16Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:31:16Z,use_quota=True,user_id='225768ea457e484f8790d273f178b879',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(5d9c86cd-3e22-4926-bfc9-e5a37f6ae76c),volume_id=5d9c86cd-3e22-4926-bfc9-e5a37f6ae76c,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': '5d9c86cd-3e22-4926-bfc9-e5a37f6ae76c', 'volume_name': 'volume-5d9c86cd-3e22-4926-bfc9-e5a37f6ae76c', 'volume_size': 1}} Apr 17 15:33:36 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'retry': {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']}, 'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': '5d9c86cd-3e22-4926-bfc9-e5a37f6ae76c', 'volume': {'id': '5d9c86cd-3e22-4926-bfc9-e5a37f6ae76c', '_name_id': None, 'ec2_id': None, 'user_id': '225768ea457e484f8790d273f178b879', 'project_id': '6cf93c7193f842d58d323e62e508a5eb', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-VolumesServicesTestJSON-Volume-968297362', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T15:31:15.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-5d9c86cd-3e22-4926-bfc9-e5a37f6ae76c', 'name_id': '5d9c86cd-3e22-4926-bfc9-e5a37f6ae76c', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumesServicesTestJSON-Volume-968297362', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '6cf93c7193f842d58d323e62e508a5eb', 'qos_specs': None, 'reservations': ['ad0fd489-3554-44ca-a47d-2b78caa30877', 'ee3d5634-17e5-4946-81db-b7539ccc1b48', '991d4c78-00b3-4a69-98e2-174c64fa2675', '86cc929c-76c2-4c7f-a486-8495ac6da9a2'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': '225768ea457e484f8790d273f178b879', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumesServicesTestJSON-Volume-968297362', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '6cf93c7193f842d58d323e62e508a5eb', 'qos_specs': None, 'reservations': ['ad0fd489-3554-44ca-a47d-2b78caa30877', 'ee3d5634-17e5-4946-81db-b7539ccc1b48', '991d4c78-00b3-4a69-98e2-174c64fa2675', '86cc929c-76c2-4c7f-a486-8495ac6da9a2'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': '225768ea457e484f8790d273f178b879', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': '225768ea457e484f8790d273f178b879', 'metadata': {}, 'qos_specs': None}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(5d9c86cd-3e22-4926-bfc9-e5a37f6ae76c),volume_id=5d9c86cd-3e22-4926-bfc9-e5a37f6ae76c,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:31:16Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesServicesTestJSON-Volume-968297362',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=5d9c86cd-3e22-4926-bfc9-e5a37f6ae76c,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='6cf93c7193f842d58d323e62e508a5eb',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:31:16Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:31:16Z,use_quota=True,user_id='225768ea457e484f8790d273f178b879',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 15:33:36 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:31:16Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesServicesTestJSON-Volume-968297362',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=5d9c86cd-3e22-4926-bfc9-e5a37f6ae76c,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='6cf93c7193f842d58d323e62e508a5eb',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:31:16Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:31:16Z,use_quota=True,user_id='225768ea457e484f8790d273f178b879',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:31:16Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesServicesTestJSON-Volume-968297362',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=5d9c86cd-3e22-4926-bfc9-e5a37f6ae76c,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='6cf93c7193f842d58d323e62e508a5eb',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:31:16Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:31:16Z,use_quota=True,user_id='225768ea457e484f8790d273f178b879',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 15:33:36 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 15:33:36 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 15:33:36 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:33:36 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 15:33:36 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 15:33:36 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 15:33:36 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 15:33:36 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 15:33:36 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 15:33:36 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 15:33:36 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 15:33:36 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 15:33:36 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 15:33:36 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 15:33:36 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 15:33:36 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 15:33:36 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 15:33:36 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 15:33:36 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 15:33:36 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 15:33:36 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (4f3cacdc-98a3-4a79-8d25-30f63ddc05a9) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:33:36 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (4f3cacdc-98a3-4a79-8d25-30f63ddc05a9) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:33:36 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (15976f23-7d64-473d-9d4c-f1bba487221b) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:33:36 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (15976f23-7d64-473d-9d4c-f1bba487221b) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:33:36 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (f94ca729-d3ce-478f-ab08-2a1718eb461c) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:33:36 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (f94ca729-d3ce-478f-ab08-2a1718eb461c) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:33:36 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (29b5c233-605f-4eac-8f7f-55e5c3c0fb1a) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:33:36 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Updating volume 5d9c86cd-3e22-4926-bfc9-e5a37f6ae76c with {'scheduled_at': datetime.datetime(2023, 4, 17, 15, 33, 36, 384457), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 15:33:36 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Volume 5d9c86cd-3e22-4926-bfc9-e5a37f6ae76c: re-scheduling SchedulerAPI.create_volume attempt 1 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 15:33:36 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Volume 5d9c86cd-3e22-4926-bfc9-e5a37f6ae76c: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 15:33:36 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] deleating volume v_5d9c86cd-3e22-4926-bfc9-e5a37f6ae76c {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 15:33:36 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] get snapshots for volume v_5d9c86cd-3e22-4926-bfc9-e5a37f6ae76c {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 15:33:36 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_5d9c86cd-3e22-4926-bfc9-e5a37f6ae76c/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:33:36 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_5d9c86cd-3e22-4926-bfc9-e5a37f6ae76c/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:33:36 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Volume reschedule parameters: True retry: {'num_attempts': 2, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 15:33:36 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Flow 'volume_create_manager' (9a244f79-c148-4050-bb04-c14a9784aa3c) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:33:36 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (40e3868e-0e66-4d73-aa2b-c28b4c5350b0) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:33:36 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (40e3868e-0e66-4d73-aa2b-c28b4c5350b0) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:31:16Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesServicesTestJSON-Volume-968297362',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=5d9c86cd-3e22-4926-bfc9-e5a37f6ae76c,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='6cf93c7193f842d58d323e62e508a5eb',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:33:36Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:33:36Z,use_quota=True,user_id='225768ea457e484f8790d273f178b879',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:33:36 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (bab63c23-4055-4aca-85fa-c39397afb88d) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:33:36 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (bab63c23-4055-4aca-85fa-c39397afb88d) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:33:36 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (7573c4d3-2722-42ef-b4e2-12963bba93f3) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:33:36 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (7573c4d3-2722-42ef-b4e2-12963bba93f3) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': '5d9c86cd-3e22-4926-bfc9-e5a37f6ae76c', 'volume_name': 'volume-5d9c86cd-3e22-4926-bfc9-e5a37f6ae76c', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:33:36 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (7f7cc4e7-dd97-4df7-b262-cfd820193f94) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:33:36 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (7f7cc4e7-dd97-4df7-b262-cfd820193f94) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:33:36 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (8c4c5d14-1b5d-49aa-9ff9-d70176fe940b) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:33:36 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Volume 5d9c86cd-3e22-4926-bfc9-e5a37f6ae76c: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-5d9c86cd-3e22-4926-bfc9-e5a37f6ae76c', 'volume_size': 1} Apr 17 15:33:36 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] creating volume v_5d9c86cd-3e22-4926-bfc9-e5a37f6ae76c. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 15:33:36 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] create volume {'name': 'v_5d9c86cd-3e22-4926-bfc9-e5a37f6ae76c', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 15:33:36 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:33:36 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:33:39 user cinder-volume[84696]: DEBUG cinder.utils [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.721(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:33:39 user cinder-volume[84696]: DEBUG cinder.utils [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:33:39 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.497(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:33:39 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:33:39 user cinder-volume[84696]: DEBUG cinder.utils [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.932(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:33:39 user cinder-volume[84696]: DEBUG cinder.utils [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_5d9c86cd-3e22-4926-bfc9-e5a37f6ae76c/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:33:42 user cinder-volume[84696]: DEBUG cinder.utils [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.793(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:33:42 user cinder-volume[84696]: DEBUG cinder.utils [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:33:42 user cinder-volume[84696]: DEBUG cinder.utils [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 11.658(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:33:42 user cinder-volume[84696]: DEBUG cinder.utils [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:33:42 user cinder-volume[84696]: DEBUG cinder.utils [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.004(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:33:42 user cinder-volume[84696]: DEBUG cinder.utils [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_5d9c86cd-3e22-4926-bfc9-e5a37f6ae76c/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:33:45 user cinder-volume[84696]: DEBUG cinder.utils [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.200(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:33:45 user cinder-volume[84696]: DEBUG cinder.utils [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:33:45 user cinder-volume[84696]: DEBUG cinder.utils [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.388(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:33:45 user cinder-volume[84696]: DEBUG cinder.utils [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_17f8d4f7-2dcd-4041-8d0b-5c8afebb75bf/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:33:48 user cinder-volume[84696]: DEBUG cinder.utils [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 140.696(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:33:48 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_a4d96ab8-2072-4eff-b803-53b4f27d056f/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:33:48 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (d9af24a3-9896-49dc-ade6-3b00092bae74) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 15:33:48 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (ddc5e7b4-61a3-411a-89c1-aaa92116a739) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:33:48 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (ddc5e7b4-61a3-411a-89c1-aaa92116a739) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:33:48 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Flow 'volume_create_manager' (2c25ab92-a180-4932-b5b0-a57aae59fc6f) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 15:33:48 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 15:33:48 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 15:33:48 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 15:33:48 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 15:33:48 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 15:33:48 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 15:33:48 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 15:33:48 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 15:33:48 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 15:33:48 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 15:33:48 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 15:33:48 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 15:33:48 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 15:33:48 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 15:33:48 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 15:33:48 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 15:33:48 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 15:33:48 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 15:33:48 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 15:33:48 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 15:33:48 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 15:33:48 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 15:33:48 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 15:33:48 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 15:33:48 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:33:48 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 15:33:48 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 15:33:48 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 15:33:48 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 15:33:48 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 15:33:48 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 15:33:48 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 15:33:48 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 15:33:48 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 15:33:48 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 15:33:48 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 15:33:48 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 15:33:48 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 15:33:48 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 15:33:48 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 15:33:48 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 15:33:48 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 15:33:48 user cinder-volume[84696]: DEBUG cinder.utils [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 140.015(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:33:48 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:33:48 user cinder-volume[84696]: INFO cinder.message.api [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Creating message record for request_id = req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df Apr 17 15:33:48 user cinder-volume[84696]: DEBUG cinder.utils [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 11.960(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:33:48 user cinder-volume[84696]: DEBUG cinder.utils [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:33:48 user cinder-volume[84696]: DEBUG cinder.utils [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.169(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:33:48 user cinder-volume[84696]: DEBUG cinder.utils [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_5d9c86cd-3e22-4926-bfc9-e5a37f6ae76c/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:33:48 user cinder-volume[84696]: DEBUG cinder.utils [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 140.548(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:33:48 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:33:48 user cinder-volume[84696]: INFO cinder.message.api [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Creating message record for request_id = req-6e1f5cff-98a0-49d3-b32d-29a667939689 Apr 17 15:33:48 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 15:33:48 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 15:33:48 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (5387c66d-2398-4e22-85d4-97447502a627) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 15:33:48 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 15:33:48 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:31:28Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesActionsTest-Volume-2081546358',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=44f909cf-7774-41f6-9f86-805456609ff8,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='a8b97ff6ed6047f0a33e74684a1c162b',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:31:28Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:31:28Z,use_quota=True,user_id='c993ceeb4a5c49eea14057e2125518ad',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 15:33:48 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:31:28Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesActionsTest-Volume-2081546358',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=44f909cf-7774-41f6-9f86-805456609ff8,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='a8b97ff6ed6047f0a33e74684a1c162b',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:31:28Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:31:28Z,use_quota=True,user_id='c993ceeb4a5c49eea14057e2125518ad',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(44f909cf-7774-41f6-9f86-805456609ff8),volume_id=44f909cf-7774-41f6-9f86-805456609ff8,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': '44f909cf-7774-41f6-9f86-805456609ff8', 'volume_name': 'volume-44f909cf-7774-41f6-9f86-805456609ff8', 'volume_size': 1}} Apr 17 15:33:48 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'retry': {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']}, 'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': '44f909cf-7774-41f6-9f86-805456609ff8', 'volume': {'id': '44f909cf-7774-41f6-9f86-805456609ff8', '_name_id': None, 'ec2_id': None, 'user_id': 'c993ceeb4a5c49eea14057e2125518ad', 'project_id': 'a8b97ff6ed6047f0a33e74684a1c162b', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-VolumesActionsTest-Volume-2081546358', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T15:31:28.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-44f909cf-7774-41f6-9f86-805456609ff8', 'name_id': '44f909cf-7774-41f6-9f86-805456609ff8', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumesActionsTest-Volume-2081546358', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': 'a8b97ff6ed6047f0a33e74684a1c162b', 'qos_specs': None, 'reservations': ['85f3837d-d106-4fab-b4df-d6deba5e2524', 'a6a77e64-4343-4b2d-a087-e1d5e603fbe7', '46947bf4-8977-46e0-91f8-7f7221eb5f15', '82ecd437-b8ec-41ab-ad06-ff748c2a9c71'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'c993ceeb4a5c49eea14057e2125518ad', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumesActionsTest-Volume-2081546358', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': 'a8b97ff6ed6047f0a33e74684a1c162b', 'qos_specs': None, 'reservations': ['85f3837d-d106-4fab-b4df-d6deba5e2524', 'a6a77e64-4343-4b2d-a087-e1d5e603fbe7', '46947bf4-8977-46e0-91f8-7f7221eb5f15', '82ecd437-b8ec-41ab-ad06-ff748c2a9c71'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'c993ceeb4a5c49eea14057e2125518ad', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': 'c993ceeb4a5c49eea14057e2125518ad', 'metadata': {}, 'qos_specs': None}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(44f909cf-7774-41f6-9f86-805456609ff8),volume_id=44f909cf-7774-41f6-9f86-805456609ff8,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:31:28Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesActionsTest-Volume-2081546358',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=44f909cf-7774-41f6-9f86-805456609ff8,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='a8b97ff6ed6047f0a33e74684a1c162b',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:31:28Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:31:28Z,use_quota=True,user_id='c993ceeb4a5c49eea14057e2125518ad',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 15:33:48 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:31:28Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesActionsTest-Volume-2081546358',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=44f909cf-7774-41f6-9f86-805456609ff8,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='a8b97ff6ed6047f0a33e74684a1c162b',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:31:28Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:31:28Z,use_quota=True,user_id='c993ceeb4a5c49eea14057e2125518ad',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:31:28Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesActionsTest-Volume-2081546358',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=44f909cf-7774-41f6-9f86-805456609ff8,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='a8b97ff6ed6047f0a33e74684a1c162b',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:31:28Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:31:28Z,use_quota=True,user_id='c993ceeb4a5c49eea14057e2125518ad',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 15:33:48 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 15:33:48 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 15:33:48 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:33:48 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 15:33:48 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 15:33:48 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 15:33:48 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 15:33:48 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 15:33:48 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 15:33:48 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 15:33:48 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 15:33:48 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 15:33:48 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 15:33:48 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 15:33:48 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 15:33:48 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 15:33:48 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 15:33:48 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 15:33:48 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 15:33:48 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 15:33:48 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (5387c66d-2398-4e22-85d4-97447502a627) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:33:48 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (5387c66d-2398-4e22-85d4-97447502a627) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:33:48 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (384faa1c-3604-4481-a6db-06d70874262f) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:33:48 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (384faa1c-3604-4481-a6db-06d70874262f) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:33:48 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (0c6ccd05-560d-49f4-9081-54365613f3a8) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:33:48 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (0c6ccd05-560d-49f4-9081-54365613f3a8) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:33:48 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (5ca814ee-ed16-440a-807b-02886c96097c) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:33:48 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Updating volume 44f909cf-7774-41f6-9f86-805456609ff8 with {'scheduled_at': datetime.datetime(2023, 4, 17, 15, 33, 48, 652417), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 15:33:48 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=20,cinder:UPDATE=7,cinder:INSERT=3 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:33:48 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (15948cbe-9b14-4690-a310-1c0808bdb5c1) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 15:33:48 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 15:33:48 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:29:10Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesAdminNegativeTest-volume-1635380936',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=a4d96ab8-2072-4eff-b803-53b4f27d056f,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='b334c5bc33f143c484518d9bd0e750dd',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:31:28Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:31:28Z,use_quota=True,user_id='3dfadfd30c394b46abe75ec930007850',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 15:33:48 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:29:10Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesAdminNegativeTest-volume-1635380936',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=a4d96ab8-2072-4eff-b803-53b4f27d056f,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='b334c5bc33f143c484518d9bd0e750dd',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:31:28Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:31:28Z,use_quota=True,user_id='3dfadfd30c394b46abe75ec930007850',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(a4d96ab8-2072-4eff-b803-53b4f27d056f),volume_id=a4d96ab8-2072-4eff-b803-53b4f27d056f,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': 'a4d96ab8-2072-4eff-b803-53b4f27d056f', 'volume_name': 'volume-a4d96ab8-2072-4eff-b803-53b4f27d056f', 'volume_size': 1}} Apr 17 15:33:48 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': 'a4d96ab8-2072-4eff-b803-53b4f27d056f', 'volume': {'id': 'a4d96ab8-2072-4eff-b803-53b4f27d056f', '_name_id': None, 'ec2_id': None, 'user_id': '3dfadfd30c394b46abe75ec930007850', 'project_id': 'b334c5bc33f143c484518d9bd0e750dd', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-VolumesAdminNegativeTest-volume-1635380936', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'admin_metadata': {}, 'glance_metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T15:29:10.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-a4d96ab8-2072-4eff-b803-53b4f27d056f', 'name_id': 'a4d96ab8-2072-4eff-b803-53b4f27d056f', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumesAdminNegativeTest-volume-1635380936', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': 'b334c5bc33f143c484518d9bd0e750dd', 'qos_specs': None, 'reservations': ['09f94177-ab67-4a35-91b5-ec04b542f529', '7b3828ae-e615-4ad0-8847-f1bf1e17c182', '8e9d3d0e-b4d4-4bfc-b097-6eaec633b485', 'b33f40af-c5d7-4f4f-b730-d04956abccb8'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': '3dfadfd30c394b46abe75ec930007850', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumesAdminNegativeTest-volume-1635380936', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': 'b334c5bc33f143c484518d9bd0e750dd', 'qos_specs': None, 'reservations': ['09f94177-ab67-4a35-91b5-ec04b542f529', '7b3828ae-e615-4ad0-8847-f1bf1e17c182', '8e9d3d0e-b4d4-4bfc-b097-6eaec633b485', 'b33f40af-c5d7-4f4f-b730-d04956abccb8'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': '3dfadfd30c394b46abe75ec930007850', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': '3dfadfd30c394b46abe75ec930007850', 'metadata': {}, 'qos_specs': None, 'retry': {'num_attempts': 2, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']}}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(a4d96ab8-2072-4eff-b803-53b4f27d056f),volume_id=a4d96ab8-2072-4eff-b803-53b4f27d056f,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:29:10Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesAdminNegativeTest-volume-1635380936',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=a4d96ab8-2072-4eff-b803-53b4f27d056f,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='b334c5bc33f143c484518d9bd0e750dd',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:31:28Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:31:28Z,use_quota=True,user_id='3dfadfd30c394b46abe75ec930007850',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 15:33:48 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:29:10Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesAdminNegativeTest-volume-1635380936',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=a4d96ab8-2072-4eff-b803-53b4f27d056f,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='b334c5bc33f143c484518d9bd0e750dd',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:31:28Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:31:28Z,use_quota=True,user_id='3dfadfd30c394b46abe75ec930007850',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:29:10Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesAdminNegativeTest-volume-1635380936',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=a4d96ab8-2072-4eff-b803-53b4f27d056f,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='b334c5bc33f143c484518d9bd0e750dd',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:31:28Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:31:28Z,use_quota=True,user_id='3dfadfd30c394b46abe75ec930007850',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 15:33:48 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 15:33:48 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 15:33:48 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:33:48 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 15:33:48 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 15:33:48 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 15:33:48 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 15:33:48 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 15:33:48 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 15:33:48 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 15:33:48 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 15:33:48 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 15:33:48 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 15:33:48 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 15:33:48 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 15:33:48 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 15:33:48 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 15:33:48 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 15:33:48 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 15:33:48 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 15:33:48 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (15948cbe-9b14-4690-a310-1c0808bdb5c1) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:33:48 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (15948cbe-9b14-4690-a310-1c0808bdb5c1) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:33:48 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (1d412cb4-8814-423b-b191-1190fc67df04) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:33:48 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (1d412cb4-8814-423b-b191-1190fc67df04) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:33:48 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (5317e8e4-e74e-4485-ba6d-c2a70b449951) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:33:48 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (5317e8e4-e74e-4485-ba6d-c2a70b449951) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:33:48 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (c9447cd5-5af6-4746-9bd5-3d7def3fb398) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:33:48 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Updating volume a4d96ab8-2072-4eff-b803-53b4f27d056f with {'scheduled_at': datetime.datetime(2023, 4, 17, 15, 33, 48, 702053), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 15:33:48 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Volume 44f909cf-7774-41f6-9f86-805456609ff8: re-scheduling SchedulerAPI.create_volume attempt 1 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 15:33:48 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Volume a4d96ab8-2072-4eff-b803-53b4f27d056f: re-scheduling SchedulerAPI.create_volume attempt 2 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 15:33:48 user cinder-volume[84696]: DEBUG cinder.coordination [None req-3175029a-5606-46a3-86f4-a16d583ad0ce tempest-TestStampPattern-173447269 None] Acquiring lock "cinder-a9c177a1-61f0-4201-bb8d-7f18c9af396c-delete_volume" by "delete_volume" {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:192}} Apr 17 15:33:48 user cinder-volume[84696]: DEBUG cinder.coordination [None req-3175029a-5606-46a3-86f4-a16d583ad0ce tempest-TestStampPattern-173447269 None] Lock "cinder-a9c177a1-61f0-4201-bb8d-7f18c9af396c-delete_volume" acquired by "delete_volume" :: waited 0.018s {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:196}} Apr 17 15:33:48 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Volume 44f909cf-7774-41f6-9f86-805456609ff8: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 15:33:48 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] deleating volume v_44f909cf-7774-41f6-9f86-805456609ff8 {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 15:33:48 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] get snapshots for volume v_44f909cf-7774-41f6-9f86-805456609ff8 {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 15:33:48 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_44f909cf-7774-41f6-9f86-805456609ff8/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:33:48 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_44f909cf-7774-41f6-9f86-805456609ff8/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:33:48 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Volume a4d96ab8-2072-4eff-b803-53b4f27d056f: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 15:33:48 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] deleating volume v_a4d96ab8-2072-4eff-b803-53b4f27d056f {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 15:33:48 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] get snapshots for volume v_a4d96ab8-2072-4eff-b803-53b4f27d056f {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 15:33:48 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_a4d96ab8-2072-4eff-b803-53b4f27d056f/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:33:48 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_a4d96ab8-2072-4eff-b803-53b4f27d056f/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:33:48 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-3175029a-5606-46a3-86f4-a16d583ad0ce tempest-TestStampPattern-173447269 None] remove_export for volume: a9c177a1-61f0-4201-bb8d-7f18c9af396c. {{(pid=84696) remove_export /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:718}} Apr 17 15:33:48 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-3175029a-5606-46a3-86f4-a16d583ad0ce tempest-TestStampPattern-173447269 None] remove export {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:912}} Apr 17 15:33:48 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-3175029a-5606-46a3-86f4-a16d583ad0ce tempest-TestStampPattern-173447269 None] detach volume:Volume(_name_id=None,admin_metadata={},attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:27:45Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TestStampPattern-volume-809461214',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=a9c177a1-61f0-4201-bb8d-7f18c9af396c,launched_at=2023-04-17T15:27:47Z,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='c9ca6aeb2f184cd58cf4b4dde051bdbb',provider_auth='CHAP spyZYatj 2VbFJ3RAf0EXlL',provider_geometry=None,provider_id=None,provider_location='172.16.0.220:3260,1 iqn.2010-10.org.openstack:a9c177a1-61f0-4201-bb8d-7f18c9af396c 0',replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:27:45Z,service_uuid='e628352e-c4cf-40eb-8389-cce40663026a',shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='deleting',terminated_at=2023-04-17T15:33:49Z,updated_at=2023-04-17T15:33:49Z,use_quota=True,user_id='795213c1e99c46a7b6dd5512b1e9f48c',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c) from target:iqn.2010-10.org.openstack:a9c177a1-61f0-4201-bb8d-7f18c9af396c. {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:913}} Apr 17 15:33:48 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-3175029a-5606-46a3-86f4-a16d583ad0ce tempest-TestStampPattern-173447269 None] detach volume v_a9c177a1-61f0-4201-bb8d-7f18c9af396c from target iqn.2010-10.org.openstack:a9c177a1-61f0-4201-bb8d-7f18c9af396c {{(pid=84696) detach_target_vol /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:606}} Apr 17 15:33:48 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-3175029a-5606-46a3-86f4-a16d583ad0ce tempest-TestStampPattern-173447269 None] Sending pool request DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:a9c177a1-61f0-4201-bb8d-7f18c9af396c/luns/v_a9c177a1-61f0-4201-bb8d-7f18c9af396c {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:33:48 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-3175029a-5606-46a3-86f4-a16d583ad0ce tempest-TestStampPattern-173447269 None] Sending DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:a9c177a1-61f0-4201-bb8d-7f18c9af396c/luns/v_a9c177a1-61f0-4201-bb8d-7f18c9af396c {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:33:48 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Volume reschedule parameters: True retry: {'num_attempts': 2, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 15:33:48 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Flow 'volume_create_manager' (04029fc4-38ce-4344-87e4-a6a557fe365d) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:33:48 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (9c5b4cb2-aa1c-4d03-af48-60f8ec135096) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:33:49 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (9c5b4cb2-aa1c-4d03-af48-60f8ec135096) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:31:28Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesActionsTest-Volume-2081546358',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=44f909cf-7774-41f6-9f86-805456609ff8,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='a8b97ff6ed6047f0a33e74684a1c162b',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:33:49Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:33:49Z,use_quota=True,user_id='c993ceeb4a5c49eea14057e2125518ad',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:33:49 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (e4df08b7-3778-4055-9368-a4b9d604e358) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:33:49 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (e4df08b7-3778-4055-9368-a4b9d604e358) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:33:49 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (1f705a08-fad7-4ef4-9ad1-6ba66457dd2e) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:33:49 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (1f705a08-fad7-4ef4-9ad1-6ba66457dd2e) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': '44f909cf-7774-41f6-9f86-805456609ff8', 'volume_name': 'volume-44f909cf-7774-41f6-9f86-805456609ff8', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:33:49 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (d56250f2-e591-4e75-bd2e-b9424716da79) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:33:49 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (d56250f2-e591-4e75-bd2e-b9424716da79) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:33:49 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (715f24c3-60e8-426b-9530-51df436971f2) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:33:49 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Volume 44f909cf-7774-41f6-9f86-805456609ff8: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-44f909cf-7774-41f6-9f86-805456609ff8', 'volume_size': 1} Apr 17 15:33:49 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] creating volume v_44f909cf-7774-41f6-9f86-805456609ff8. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 15:33:49 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] create volume {'name': 'v_44f909cf-7774-41f6-9f86-805456609ff8', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 15:33:49 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:33:49 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:33:49 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Volume reschedule parameters: True retry: {'num_attempts': 3, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 15:33:49 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Flow 'volume_create_manager' (120a1031-6402-4316-aa77-94af172d19ce) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:33:49 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (a3ffb6ea-004c-40fb-81a9-76ef1088252c) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:33:49 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (a3ffb6ea-004c-40fb-81a9-76ef1088252c) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:29:10Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesAdminNegativeTest-volume-1635380936',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=a4d96ab8-2072-4eff-b803-53b4f27d056f,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='b334c5bc33f143c484518d9bd0e750dd',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:33:49Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:33:49Z,use_quota=True,user_id='3dfadfd30c394b46abe75ec930007850',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:33:49 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (3998f8e0-3ba1-4802-8f74-76b867b943f7) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:33:49 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (3998f8e0-3ba1-4802-8f74-76b867b943f7) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:33:49 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (abb0c222-81ba-470f-82c8-ace0482c0fe5) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:33:49 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (abb0c222-81ba-470f-82c8-ace0482c0fe5) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': 'a4d96ab8-2072-4eff-b803-53b4f27d056f', 'volume_name': 'volume-a4d96ab8-2072-4eff-b803-53b4f27d056f', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:33:49 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (b04db691-f082-4c0f-8173-5261e8d24964) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:33:49 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (b04db691-f082-4c0f-8173-5261e8d24964) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:33:49 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (15b48948-992e-4957-a0c9-08bfade09bc4) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:33:49 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Volume a4d96ab8-2072-4eff-b803-53b4f27d056f: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-a4d96ab8-2072-4eff-b803-53b4f27d056f', 'volume_size': 1} Apr 17 15:33:49 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] creating volume v_a4d96ab8-2072-4eff-b803-53b4f27d056f. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 15:33:49 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] create volume {'name': 'v_a4d96ab8-2072-4eff-b803-53b4f27d056f', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 15:33:49 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:33:49 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:33:51 user cinder-volume[84696]: DEBUG cinder.utils [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.606(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:33:51 user cinder-volume[84696]: DEBUG cinder.utils [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:33:51 user cinder-volume[84696]: DEBUG cinder.utils [None req-3175029a-5606-46a3-86f4-a16d583ad0ce tempest-TestStampPattern-173447269 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.863(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:33:51 user cinder-volume[84696]: DEBUG cinder.utils [None req-3175029a-5606-46a3-86f4-a16d583ad0ce tempest-TestStampPattern-173447269 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:a9c177a1-61f0-4201-bb8d-7f18c9af396c/luns/v_a9c177a1-61f0-4201-bb8d-7f18c9af396c (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:33:51 user cinder-volume[84696]: DEBUG cinder.utils [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.532(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:33:51 user cinder-volume[84696]: DEBUG cinder.utils [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:33:51 user cinder-volume[84696]: DEBUG cinder.utils [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.905(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:33:51 user cinder-volume[84696]: DEBUG cinder.utils [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_44f909cf-7774-41f6-9f86-805456609ff8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:33:51 user cinder-volume[84696]: DEBUG cinder.utils [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.896(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:33:51 user cinder-volume[84696]: DEBUG cinder.utils [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_a4d96ab8-2072-4eff-b803-53b4f27d056f/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:33:51 user cinder-volume[84696]: DEBUG cinder.utils [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 20.883(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:33:51 user cinder-volume[84696]: DEBUG cinder.utils [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:33:54 user cinder-volume[84696]: DEBUG cinder.utils [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.970(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:33:54 user cinder-volume[84696]: DEBUG cinder.utils [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_44f909cf-7774-41f6-9f86-805456609ff8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:33:54 user cinder-volume[84696]: DEBUG cinder.utils [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.962(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:33:54 user cinder-volume[84696]: DEBUG cinder.utils [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_a4d96ab8-2072-4eff-b803-53b4f27d056f/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:33:54 user cinder-volume[84696]: DEBUG cinder.utils [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.679(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:33:54 user cinder-volume[84696]: DEBUG cinder.utils [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:33:54 user cinder-volume[84696]: DEBUG cinder.utils [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.817(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:33:54 user cinder-volume[84696]: DEBUG cinder.utils [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_62709a81-268b-4227-ba4a-070034f36414/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:33:54 user cinder-volume[84696]: DEBUG cinder.utils [None req-d7e6b19d-3872-4fec-88e0-811907804f16 tempest-TestVolumeSwap-1714109745 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 140.237(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:33:54 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-d7e6b19d-3872-4fec-88e0-811907804f16 tempest-TestVolumeSwap-1714109745 None] Geting None from DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:66000c93-eb12-4c23-8a69-71e1aa6b521f/luns/v_66000c93-eb12-4c23-8a69-71e1aa6b521f {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:33:54 user cinder-volume[84696]: DEBUG cinder.utils [None req-3175029a-5606-46a3-86f4-a16d583ad0ce tempest-TestStampPattern-173447269 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.939(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:33:54 user cinder-volume[84696]: DEBUG cinder.utils [None req-3175029a-5606-46a3-86f4-a16d583ad0ce tempest-TestStampPattern-173447269 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:a9c177a1-61f0-4201-bb8d-7f18c9af396c/luns/v_a9c177a1-61f0-4201-bb8d-7f18c9af396c (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:33:54 user cinder-volume[84696]: DEBUG cinder.utils [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.638(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:33:54 user cinder-volume[84696]: DEBUG cinder.utils [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:33:54 user cinder-volume[84696]: DEBUG cinder.utils [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.609(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:33:54 user cinder-volume[84696]: DEBUG cinder.utils [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:33:54 user cinder-volume[84696]: DEBUG cinder.coordination [None req-d7e6b19d-3872-4fec-88e0-811907804f16 tempest-TestVolumeSwap-1714109745 None] Lock "cinder-66000c93-eb12-4c23-8a69-71e1aa6b521f-delete_volume" released by "delete_volume" :: held 140.288s {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:207}} Apr 17 15:33:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-d7e6b19d-3872-4fec-88e0-811907804f16 tempest-TestVolumeSwap-1714109745 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 15:33:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 15:33:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 15:33:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 15:33:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 15:33:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 15:33:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 15:33:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 15:33:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 203, in wrapper Apr 17 15:33:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server skip_clean = func(self, context, volume, *args, **kwargs) Apr 17 15:33:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 15:33:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 15:33:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/coordination.py", line 200, in _synchronized Apr 17 15:33:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return f(*a, **k) Apr 17 15:33:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 15:33:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 15:33:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 15:33:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 15:33:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 1035, in delete_volume Apr 17 15:33:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 15:33:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 15:33:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 15:33:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 15:33:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 15:33:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 1004, in delete_volume Apr 17 15:33:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.driver.remove_export(context, volume) Apr 17 15:33:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 720, in remove_export Apr 17 15:33:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self._remove_target_volume(volume) Apr 17 15:33:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 918, in _remove_target_volume Apr 17 15:33:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.detach_target_vol(target_name, jcom.vname(volume.id)) Apr 17 15:33:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 612, in detach_target_vol Apr 17 15:33:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if resp["code"] in (200, 201, 204): Apr 17 15:33:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 15:33:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 15:33:57 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.927(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:33:57 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:33:57 user cinder-volume[84696]: DEBUG cinder.utils [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.154(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:33:57 user cinder-volume[84696]: DEBUG cinder.utils [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:33:57 user cinder-volume[84696]: DEBUG cinder.utils [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.364(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:33:57 user cinder-volume[84696]: DEBUG cinder.utils [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_5d9c86cd-3e22-4926-bfc9-e5a37f6ae76c/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:34:00 user cinder-volume[84696]: DEBUG cinder.utils [None req-3175029a-5606-46a3-86f4-a16d583ad0ce tempest-TestStampPattern-173447269 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.073(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:34:00 user cinder-volume[84696]: DEBUG cinder.utils [None req-3175029a-5606-46a3-86f4-a16d583ad0ce tempest-TestStampPattern-173447269 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:a9c177a1-61f0-4201-bb8d-7f18c9af396c/luns/v_a9c177a1-61f0-4201-bb8d-7f18c9af396c (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:34:00 user cinder-volume[84696]: DEBUG cinder.utils [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 138.934(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:34:00 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:34:00 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (57e44147-da06-48f9-9d75-0dc7f5ae6de7) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 15:34:00 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 15:34:00 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:31:42Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesCloneTest-Volume-297264340',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=0fed6b21-0520-4554-bee4-9540a03f9af0,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='82fce1724afa40dd914e0791c3ee69a3',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:31:42Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:31:42Z,use_quota=True,user_id='773ce6c2a0f844edaf666e190504e258',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 15:34:00 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:31:42Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesCloneTest-Volume-297264340',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=0fed6b21-0520-4554-bee4-9540a03f9af0,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='82fce1724afa40dd914e0791c3ee69a3',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:31:42Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:31:42Z,use_quota=True,user_id='773ce6c2a0f844edaf666e190504e258',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=b3f0dec1-7758-44a8-82e0-76ada8837428,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(0fed6b21-0520-4554-bee4-9540a03f9af0),volume_id=0fed6b21-0520-4554-bee4-9540a03f9af0,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'image', 'volume_id': '0fed6b21-0520-4554-bee4-9540a03f9af0', 'volume_name': 'volume-0fed6b21-0520-4554-bee4-9540a03f9af0', 'volume_size': 1, 'image_id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'image_location': (None, None), 'image_meta': {'name': 'cirros-0.5.2-x86_64-disk', 'disk_format': 'qcow2', 'container_format': 'bare', 'visibility': 'public', 'size': 16300544, 'virtual_size': 117440512, 'status': 'active', 'checksum': 'b874c39491a2377b8490f5f1e89761a4', 'protected': False, 'min_ram': 0, 'min_disk': 0, 'owner': 'a43c211b4ed542538799349489b1d884', 'os_hidden': False, 'os_hash_algo': 'sha512', 'os_hash_value': '6b813aa46bb90b4da216a4d19376593fa3f4fc7e617f03a92b7fe11e9a3981cbe8f0959dbebe36225e5f53dc4492341a4863cac4ed1ee0909f3fc78ef9c3e869', 'id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'created_at': datetime.datetime(2023, 4, 17, 15, 19, 53, tzinfo=datetime.timezone.utc), 'updated_at': datetime.datetime(2023, 4, 17, 15, 19, 55, tzinfo=datetime.timezone.utc), 'tags': [], 'file': '/v2/images/b3f0dec1-7758-44a8-82e0-76ada8837428/file', 'properties': {'hw_rng_model': 'virtio', 'owner_specified.openstack.md5': '', 'owner_specified.openstack.object': 'images/cirros-0.5.2-x86_64-disk', 'owner_specified.openstack.sha256': ''}}, 'image_service': }} Apr 17 15:34:00 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'retry': {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']}, 'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'snapshot_id': None, 'source_volid': None, 'volume_id': '0fed6b21-0520-4554-bee4-9540a03f9af0', 'volume': {'id': '0fed6b21-0520-4554-bee4-9540a03f9af0', '_name_id': None, 'ec2_id': None, 'user_id': '773ce6c2a0f844edaf666e190504e258', 'project_id': '82fce1724afa40dd914e0791c3ee69a3', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-VolumesCloneTest-Volume-297264340', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T15:31:41.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-0fed6b21-0520-4554-bee4-9540a03f9af0', 'name_id': '0fed6b21-0520-4554-bee4-9540a03f9af0', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumesCloneTest-Volume-297264340', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '82fce1724afa40dd914e0791c3ee69a3', 'qos_specs': None, 'reservations': ['1379a6d8-9976-4d94-a18a-ef0001e5de6b', 'ebe18ce3-cd83-4706-9c9a-b20a8ec73b8f', '3321e2fc-3bcb-4c90-bfc9-3a5a122ad7a3', '0805cd4d-24d4-47d8-b230-5b3c0a1d66aa'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': '773ce6c2a0f844edaf666e190504e258', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumesCloneTest-Volume-297264340', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '82fce1724afa40dd914e0791c3ee69a3', 'qos_specs': None, 'reservations': ['1379a6d8-9976-4d94-a18a-ef0001e5de6b', 'ebe18ce3-cd83-4706-9c9a-b20a8ec73b8f', '3321e2fc-3bcb-4c90-bfc9-3a5a122ad7a3', '0805cd4d-24d4-47d8-b230-5b3c0a1d66aa'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': '773ce6c2a0f844edaf666e190504e258', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': '773ce6c2a0f844edaf666e190504e258', 'metadata': {}, 'qos_specs': None}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=b3f0dec1-7758-44a8-82e0-76ada8837428,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(0fed6b21-0520-4554-bee4-9540a03f9af0),volume_id=0fed6b21-0520-4554-bee4-9540a03f9af0,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:31:42Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesCloneTest-Volume-297264340',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=0fed6b21-0520-4554-bee4-9540a03f9af0,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='82fce1724afa40dd914e0791c3ee69a3',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:31:42Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:31:42Z,use_quota=True,user_id='773ce6c2a0f844edaf666e190504e258',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 15:34:00 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:31:42Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesCloneTest-Volume-297264340',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=0fed6b21-0520-4554-bee4-9540a03f9af0,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='82fce1724afa40dd914e0791c3ee69a3',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:31:42Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:31:42Z,use_quota=True,user_id='773ce6c2a0f844edaf666e190504e258',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:31:42Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesCloneTest-Volume-297264340',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=0fed6b21-0520-4554-bee4-9540a03f9af0,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='82fce1724afa40dd914e0791c3ee69a3',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:31:42Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:31:42Z,use_quota=True,user_id='773ce6c2a0f844edaf666e190504e258',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 15:34:00 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 15:34:00 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 15:34:00 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:34:00 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 15:34:00 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1237, in execute Apr 17 15:34:00 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_from_image(context, Apr 17 15:34:00 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/utils.py", line 670, in _wrapper Apr 17 15:34:00 user cinder-volume[84696]: ERROR cinder.volume.manager return r(f, *args, **kwargs) Apr 17 15:34:00 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/tenacity/__init__.py", line 423, in __call__ Apr 17 15:34:00 user cinder-volume[84696]: ERROR cinder.volume.manager do = self.iter(retry_state=retry_state) Apr 17 15:34:00 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/tenacity/__init__.py", line 360, in iter Apr 17 15:34:00 user cinder-volume[84696]: ERROR cinder.volume.manager return fut.result() Apr 17 15:34:00 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/lib/python3.10/concurrent/futures/_base.py", line 451, in result Apr 17 15:34:00 user cinder-volume[84696]: ERROR cinder.volume.manager return self.__get_result() Apr 17 15:34:00 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result Apr 17 15:34:00 user cinder-volume[84696]: ERROR cinder.volume.manager raise self._exception Apr 17 15:34:00 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/tenacity/__init__.py", line 426, in __call__ Apr 17 15:34:00 user cinder-volume[84696]: ERROR cinder.volume.manager result = fn(*args, **kwargs) Apr 17 15:34:00 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1125, in _create_from_image Apr 17 15:34:00 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_from_image_cache_or_download( Apr 17 15:34:00 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 944, in _create_from_image_cache_or_download Apr 17 15:34:00 user cinder-volume[84696]: ERROR cinder.volume.manager model_update, cloned = self._create_from_image_cache( Apr 17 15:34:00 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 843, in _create_from_image_cache Apr 17 15:34:00 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_from_source_volume( Apr 17 15:34:00 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 659, in _create_from_source_volume Apr 17 15:34:00 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self.driver.create_cloned_volume(volume, srcvol_ref) Apr 17 15:34:00 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 525, in create_cloned_volume Apr 17 15:34:00 user cinder-volume[84696]: ERROR cinder.volume.manager self._clone_object(vname, cvname) Apr 17 15:34:00 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 461, in _clone_object Apr 17 15:34:00 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_snapshot(oname, coname) Apr 17 15:34:00 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 639, in create_snapshot Apr 17 15:34:00 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201, 204): Apr 17 15:34:00 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 15:34:00 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 15:34:00 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (57e44147-da06-48f9-9d75-0dc7f5ae6de7) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:34:00 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (57e44147-da06-48f9-9d75-0dc7f5ae6de7) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:34:00 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (d60fd341-b3cb-4d3a-9e50-9730bbd40470) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:34:00 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (d60fd341-b3cb-4d3a-9e50-9730bbd40470) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:34:00 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (5fbc725d-6cae-4136-9318-996fe6a0b29d) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:34:00 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (5fbc725d-6cae-4136-9318-996fe6a0b29d) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:34:00 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (975708e3-2525-423b-9a96-cdb2cd196821) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:34:00 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Updating volume 0fed6b21-0520-4554-bee4-9540a03f9af0 with {'scheduled_at': datetime.datetime(2023, 4, 17, 15, 34, 0, 905167), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 15:34:00 user cinder-volume[84696]: DEBUG cinder.utils [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 11.788(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:34:00 user cinder-volume[84696]: DEBUG cinder.utils [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:34:00 user cinder-volume[84696]: DEBUG cinder.utils [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 11.866(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:34:00 user cinder-volume[84696]: DEBUG cinder.utils [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:34:00 user cinder-volume[84696]: DEBUG cinder.utils [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.152(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:34:00 user cinder-volume[84696]: DEBUG cinder.utils [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_a4d96ab8-2072-4eff-b803-53b4f27d056f/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:34:00 user cinder-volume[84696]: DEBUG cinder.utils [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.163(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:34:00 user cinder-volume[84696]: DEBUG cinder.utils [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_44f909cf-7774-41f6-9f86-805456609ff8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:34:00 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Volume 0fed6b21-0520-4554-bee4-9540a03f9af0: re-scheduling SchedulerAPI.create_volume attempt 1 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 15:34:00 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Volume 0fed6b21-0520-4554-bee4-9540a03f9af0: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 15:34:00 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] deleating volume v_0fed6b21-0520-4554-bee4-9540a03f9af0 {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 15:34:00 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] get snapshots for volume v_0fed6b21-0520-4554-bee4-9540a03f9af0 {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 15:34:00 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_0fed6b21-0520-4554-bee4-9540a03f9af0/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:34:00 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_0fed6b21-0520-4554-bee4-9540a03f9af0/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:34:01 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Volume reschedule parameters: True retry: {'num_attempts': 2, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 15:34:01 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Flow 'volume_create_manager' (455eb968-f753-40eb-a170-f452ecb3538e) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:34:01 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (2c43f562-4ae1-47c3-bf56-e2d22afd7ee1) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:34:01 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (2c43f562-4ae1-47c3-bf56-e2d22afd7ee1) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:31:42Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesCloneTest-Volume-297264340',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=0fed6b21-0520-4554-bee4-9540a03f9af0,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='82fce1724afa40dd914e0791c3ee69a3',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:34:01Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:34:01Z,use_quota=True,user_id='773ce6c2a0f844edaf666e190504e258',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:34:01 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (3ce8ae22-70a5-4d60-94e6-f9407d0cd7fd) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:34:01 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (3ce8ae22-70a5-4d60-94e6-f9407d0cd7fd) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:34:01 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (46b5f298-21b2-4289-8c51-0a8dca8c9bbf) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:34:01 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (46b5f298-21b2-4289-8c51-0a8dca8c9bbf) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'image', 'volume_id': '0fed6b21-0520-4554-bee4-9540a03f9af0', 'volume_name': 'volume-0fed6b21-0520-4554-bee4-9540a03f9af0', 'volume_size': 1, 'image_id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'image_location': (None, None), 'image_meta': {'name': 'cirros-0.5.2-x86_64-disk', 'disk_format': 'qcow2', 'container_format': 'bare', 'visibility': 'public', 'size': 16300544, 'virtual_size': 117440512, 'status': 'active', 'checksum': 'b874c39491a2377b8490f5f1e89761a4', 'protected': False, 'min_ram': 0, 'min_disk': 0, 'owner': 'a43c211b4ed542538799349489b1d884', 'os_hidden': False, 'os_hash_algo': 'sha512', 'os_hash_value': '6b813aa46bb90b4da216a4d19376593fa3f4fc7e617f03a92b7fe11e9a3981cbe8f0959dbebe36225e5f53dc4492341a4863cac4ed1ee0909f3fc78ef9c3e869', 'id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'created_at': datetime.datetime(2023, 4, 17, 15, 19, 53, tzinfo=datetime.timezone.utc), 'updated_at': datetime.datetime(2023, 4, 17, 15, 19, 55, tzinfo=datetime.timezone.utc), 'tags': [], 'file': '/v2/images/b3f0dec1-7758-44a8-82e0-76ada8837428/file', 'properties': {'hw_rng_model': 'virtio', 'owner_specified.openstack.md5': '', 'owner_specified.openstack.object': 'images/cirros-0.5.2-x86_64-disk', 'owner_specified.openstack.sha256': ''}}, 'image_service': }' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:34:01 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (41eef73b-9547-438d-8266-ee4f295287e9) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:34:01 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (41eef73b-9547-438d-8266-ee4f295287e9) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:34:01 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (98df3471-98bf-4d2e-ad67-b0292122b87f) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:34:01 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Volume 0fed6b21-0520-4554-bee4-9540a03f9af0: being created as image with specification: {'status': 'creating', 'volume_name': 'volume-0fed6b21-0520-4554-bee4-9540a03f9af0', 'volume_size': 1, 'image_id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'image_location': (None, None), 'image_meta': {'name': 'cirros-0.5.2-x86_64-disk', 'disk_format': 'qcow2', 'container_format': 'bare', 'visibility': 'public', 'size': 16300544, 'virtual_size': 117440512, 'status': 'active', 'checksum': 'b874c39491a2377b8490f5f1e89761a4', 'protected': False, 'min_ram': 0, 'min_disk': 0, 'owner': 'a43c211b4ed542538799349489b1d884', 'os_hidden': False, 'os_hash_algo': 'sha512', 'os_hash_value': '6b813aa46bb90b4da216a4d19376593fa3f4fc7e617f03a92b7fe11e9a3981cbe8f0959dbebe36225e5f53dc4492341a4863cac4ed1ee0909f3fc78ef9c3e869', 'id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'created_at': datetime.datetime(2023, 4, 17, 15, 19, 53, tzinfo=datetime.timezone.utc), 'updated_at': datetime.datetime(2023, 4, 17, 15, 19, 55, tzinfo=datetime.timezone.utc), 'tags': [], 'file': '/v2/images/b3f0dec1-7758-44a8-82e0-76ada8837428/file', 'properties': {'hw_rng_model': 'virtio', 'owner_specified.openstack.md5': '', 'owner_specified.openstack.object': 'images/cirros-0.5.2-x86_64-disk', 'owner_specified.openstack.sha256': ''}}, 'image_service': } Apr 17 15:34:01 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Cloning 0fed6b21-0520-4554-bee4-9540a03f9af0 from image b3f0dec1-7758-44a8-82e0-76ada8837428 at location (None, None). {{(pid=84696) _create_from_image /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1071}} Apr 17 15:34:01 user cinder-volume[84696]: DEBUG cinder.coordination [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Acquiring lock "cinder-b3f0dec1-7758-44a8-82e0-76ada8837428" by "_prepare_image_cache_entry" {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:192}} Apr 17 15:34:01 user cinder-volume[84696]: DEBUG cinder.coordination [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Lock "cinder-b3f0dec1-7758-44a8-82e0-76ada8837428" acquired by "_prepare_image_cache_entry" :: waited 0.008s {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:196}} Apr 17 15:34:01 user cinder-volume[84696]: DEBUG cinder.image.cache [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Found image-volume cache entry: {'id': 1, 'image_id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'volume_id': '006f20ae-5306-44fe-afdf-e433f79284e8', 'host': 'user@joviandss-2#joviandss-2', 'size': 1, 'image_updated_at': datetime.datetime(2023, 4, 17, 15, 19, 55), 'last_used': datetime.datetime(2023, 4, 17, 15, 34, 1, 213728)}. {{(pid=84696) get_entry /opt/stack/cinder/cinder/image/cache.py:76}} Apr 17 15:34:01 user cinder-volume[84696]: DEBUG cinder.image.cache [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Image-volume cache entry image_update_at = 2023-04-17 15:19:55+00:00, requested image updated_at = 2023-04-17 15:19:55+00:00. {{(pid=84696) _should_update_entry /opt/stack/cinder/cinder/image/cache.py:259}} Apr 17 15:34:01 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Found cache entry for image = b3f0dec1-7758-44a8-82e0-76ada8837428 on host user@joviandss-2#joviandss-2. {{(pid=84696) _prepare_image_cache_entry /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:886}} Apr 17 15:34:01 user cinder-volume[84696]: DEBUG cinder.coordination [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Lock "cinder-b3f0dec1-7758-44a8-82e0-76ada8837428" released by "_prepare_image_cache_entry" :: held 0.013s {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:207}} Apr 17 15:34:01 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Attempting to retrieve cache entry for image = b3f0dec1-7758-44a8-82e0-76ada8837428 on host user@joviandss-2#joviandss-2. {{(pid=84696) _create_from_image_cache /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:827}} Apr 17 15:34:01 user cinder-volume[84696]: DEBUG cinder.image.cache [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Found image-volume cache entry: {'id': 1, 'image_id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'volume_id': '006f20ae-5306-44fe-afdf-e433f79284e8', 'host': 'user@joviandss-2#joviandss-2', 'size': 1, 'image_updated_at': datetime.datetime(2023, 4, 17, 15, 19, 55), 'last_used': datetime.datetime(2023, 4, 17, 15, 34, 1, 226835)}. {{(pid=84696) get_entry /opt/stack/cinder/cinder/image/cache.py:76}} Apr 17 15:34:01 user cinder-volume[84696]: DEBUG cinder.image.cache [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Image-volume cache entry image_update_at = 2023-04-17 15:19:55+00:00, requested image updated_at = 2023-04-17 15:19:55+00:00. {{(pid=84696) _should_update_entry /opt/stack/cinder/cinder/image/cache.py:259}} Apr 17 15:34:01 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Creating from source image-volume 006f20ae-5306-44fe-afdf-e433f79284e8 {{(pid=84696) _create_from_image_cache /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:841}} Apr 17 15:34:01 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] cloned volume 006f20ae-5306-44fe-afdf-e433f79284e8 to 0fed6b21-0520-4554-bee4-9540a03f9af0 {{(pid=84696) create_cloned_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:521}} Apr 17 15:34:01 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] cloning v_006f20ae-5306-44fe-afdf-e433f79284e8 to v_0fed6b21-0520-4554-bee4-9540a03f9af0 {{(pid=84696) _clone_object /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:456}} Apr 17 15:34:01 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] create snapshot v_0fed6b21-0520-4554-bee4-9540a03f9af0 {{(pid=84696) create_snapshot /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:635}} Apr 17 15:34:01 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:34:01 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:34:03 user cinder-volume[84696]: DEBUG cinder.utils [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.632(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:34:03 user cinder-volume[84696]: DEBUG cinder.utils [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:34:03 user cinder-volume[84696]: DEBUG cinder.utils [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 3.000(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:34:03 user cinder-volume[84696]: DEBUG cinder.utils [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_0fed6b21-0520-4554-bee4-9540a03f9af0/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:34:03 user cinder-volume[84696]: DEBUG cinder.utils [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.687(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:34:03 user cinder-volume[84696]: DEBUG cinder.utils [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:34:03 user cinder-volume[84696]: DEBUG cinder.utils [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.823(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:34:03 user cinder-volume[84696]: DEBUG cinder.utils [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_17f8d4f7-2dcd-4041-8d0b-5c8afebb75bf/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:34:07 user cinder-volume[84696]: DEBUG cinder.utils [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.070(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:34:07 user cinder-volume[84696]: DEBUG cinder.utils [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_0fed6b21-0520-4554-bee4-9540a03f9af0/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:34:07 user cinder-volume[84696]: DEBUG cinder.utils [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.756(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:34:07 user cinder-volume[84696]: DEBUG cinder.utils [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:34:10 user cinder-volume[84696]: DEBUG cinder.utils [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.304(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:34:10 user cinder-volume[84696]: DEBUG cinder.utils [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:34:10 user cinder-volume[84696]: DEBUG cinder.utils [None req-3175029a-5606-46a3-86f4-a16d583ad0ce tempest-TestStampPattern-173447269 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.290(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:34:10 user cinder-volume[84696]: DEBUG cinder.utils [None req-3175029a-5606-46a3-86f4-a16d583ad0ce tempest-TestStampPattern-173447269 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:a9c177a1-61f0-4201-bb8d-7f18c9af396c/luns/v_a9c177a1-61f0-4201-bb8d-7f18c9af396c (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:34:10 user cinder-volume[84696]: DEBUG cinder.utils [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 20.959(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:34:10 user cinder-volume[84696]: DEBUG cinder.utils [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:34:10 user cinder-volume[84696]: DEBUG cinder.utils [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.037(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:34:10 user cinder-volume[84696]: DEBUG cinder.utils [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:34:10 user cinder-volume[84696]: DEBUG cinder.utils [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.322(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:34:10 user cinder-volume[84696]: DEBUG cinder.utils [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_a4d96ab8-2072-4eff-b803-53b4f27d056f/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:34:10 user cinder-volume[84696]: DEBUG cinder.utils [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.333(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:34:10 user cinder-volume[84696]: DEBUG cinder.utils [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_44f909cf-7774-41f6-9f86-805456609ff8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:34:11 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=31,cinder:UPDATE=12,cinder:INSERT=4 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:34:13 user cinder-volume[84696]: DEBUG cinder.utils [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 11.899(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:34:13 user cinder-volume[84696]: DEBUG cinder.utils [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:34:13 user cinder-volume[84696]: DEBUG cinder.utils [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.217(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:34:13 user cinder-volume[84696]: DEBUG cinder.utils [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_0fed6b21-0520-4554-bee4-9540a03f9af0/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:34:16 user cinder-volume[84696]: DEBUG cinder.utils [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.793(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:34:16 user cinder-volume[84696]: DEBUG cinder.utils [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_5d9c86cd-3e22-4926-bfc9-e5a37f6ae76c/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:34:16 user cinder-volume[84696]: DEBUG cinder.utils [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.586(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:34:16 user cinder-volume[84696]: DEBUG cinder.utils [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:34:16 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Volume reschedule parameters: True retry: {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 15:34:17 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Flow 'volume_create_manager' (a8614e59-5bc3-4dab-90a0-4251e85b09b8) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:34:17 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (9df5963a-0c49-4b10-8145-f7009247aea2) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:34:17 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (9df5963a-0c49-4b10-8145-f7009247aea2) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:34:17Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TaggedBootDevicesTest-volume-1099737906',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=e80c78b2-20c9-4f9b-8898-6ab131d1eb06,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='75ab8873f7064d3fbe954746c071cb72',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:34:17Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:34:17Z,use_quota=True,user_id='39328f4ed6da4a85ac312bc5637b1500',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:34:17 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (6298f3cf-bb1b-429d-82fc-ec346f35e33b) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:34:17 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (6298f3cf-bb1b-429d-82fc-ec346f35e33b) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:34:17 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (58293bf3-a01b-4124-8ed5-33388987ea91) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:34:17 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (58293bf3-a01b-4124-8ed5-33388987ea91) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'image', 'volume_id': 'e80c78b2-20c9-4f9b-8898-6ab131d1eb06', 'volume_name': 'volume-e80c78b2-20c9-4f9b-8898-6ab131d1eb06', 'volume_size': 1, 'image_id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'image_location': (None, None), 'image_meta': {'name': 'cirros-0.5.2-x86_64-disk', 'disk_format': 'qcow2', 'container_format': 'bare', 'visibility': 'public', 'size': 16300544, 'virtual_size': 117440512, 'status': 'active', 'checksum': 'b874c39491a2377b8490f5f1e89761a4', 'protected': False, 'min_ram': 0, 'min_disk': 0, 'owner': 'a43c211b4ed542538799349489b1d884', 'os_hidden': False, 'os_hash_algo': 'sha512', 'os_hash_value': '6b813aa46bb90b4da216a4d19376593fa3f4fc7e617f03a92b7fe11e9a3981cbe8f0959dbebe36225e5f53dc4492341a4863cac4ed1ee0909f3fc78ef9c3e869', 'id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'created_at': datetime.datetime(2023, 4, 17, 15, 19, 53, tzinfo=datetime.timezone.utc), 'updated_at': datetime.datetime(2023, 4, 17, 15, 19, 55, tzinfo=datetime.timezone.utc), 'tags': [], 'file': '/v2/images/b3f0dec1-7758-44a8-82e0-76ada8837428/file', 'properties': {'hw_rng_model': 'virtio', 'owner_specified.openstack.md5': '', 'owner_specified.openstack.object': 'images/cirros-0.5.2-x86_64-disk', 'owner_specified.openstack.sha256': ''}}, 'image_service': }' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:34:17 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (4bdc302e-f359-4099-99a8-482da233d7cd) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:34:17 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (4bdc302e-f359-4099-99a8-482da233d7cd) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:34:17 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (b1232b89-540b-4b2f-8c7c-9165c17366a0) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:34:17 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Volume e80c78b2-20c9-4f9b-8898-6ab131d1eb06: being created as image with specification: {'status': 'creating', 'volume_name': 'volume-e80c78b2-20c9-4f9b-8898-6ab131d1eb06', 'volume_size': 1, 'image_id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'image_location': (None, None), 'image_meta': {'name': 'cirros-0.5.2-x86_64-disk', 'disk_format': 'qcow2', 'container_format': 'bare', 'visibility': 'public', 'size': 16300544, 'virtual_size': 117440512, 'status': 'active', 'checksum': 'b874c39491a2377b8490f5f1e89761a4', 'protected': False, 'min_ram': 0, 'min_disk': 0, 'owner': 'a43c211b4ed542538799349489b1d884', 'os_hidden': False, 'os_hash_algo': 'sha512', 'os_hash_value': '6b813aa46bb90b4da216a4d19376593fa3f4fc7e617f03a92b7fe11e9a3981cbe8f0959dbebe36225e5f53dc4492341a4863cac4ed1ee0909f3fc78ef9c3e869', 'id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'created_at': datetime.datetime(2023, 4, 17, 15, 19, 53, tzinfo=datetime.timezone.utc), 'updated_at': datetime.datetime(2023, 4, 17, 15, 19, 55, tzinfo=datetime.timezone.utc), 'tags': [], 'file': '/v2/images/b3f0dec1-7758-44a8-82e0-76ada8837428/file', 'properties': {'hw_rng_model': 'virtio', 'owner_specified.openstack.md5': '', 'owner_specified.openstack.object': 'images/cirros-0.5.2-x86_64-disk', 'owner_specified.openstack.sha256': ''}}, 'image_service': } Apr 17 15:34:17 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Cloning e80c78b2-20c9-4f9b-8898-6ab131d1eb06 from image b3f0dec1-7758-44a8-82e0-76ada8837428 at location (None, None). {{(pid=84696) _create_from_image /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1071}} Apr 17 15:34:17 user cinder-volume[84696]: DEBUG cinder.coordination [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Acquiring lock "cinder-b3f0dec1-7758-44a8-82e0-76ada8837428" by "_prepare_image_cache_entry" {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:192}} Apr 17 15:34:17 user cinder-volume[84696]: DEBUG cinder.coordination [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Lock "cinder-b3f0dec1-7758-44a8-82e0-76ada8837428" acquired by "_prepare_image_cache_entry" :: waited 0.010s {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:196}} Apr 17 15:34:17 user cinder-volume[84696]: DEBUG cinder.image.cache [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Found image-volume cache entry: {'id': 1, 'image_id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'volume_id': '006f20ae-5306-44fe-afdf-e433f79284e8', 'host': 'user@joviandss-2#joviandss-2', 'size': 1, 'image_updated_at': datetime.datetime(2023, 4, 17, 15, 19, 55), 'last_used': datetime.datetime(2023, 4, 17, 15, 34, 17, 185425)}. {{(pid=84696) get_entry /opt/stack/cinder/cinder/image/cache.py:76}} Apr 17 15:34:17 user cinder-volume[84696]: DEBUG cinder.image.cache [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Image-volume cache entry image_update_at = 2023-04-17 15:19:55+00:00, requested image updated_at = 2023-04-17 15:19:55+00:00. {{(pid=84696) _should_update_entry /opt/stack/cinder/cinder/image/cache.py:259}} Apr 17 15:34:17 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Found cache entry for image = b3f0dec1-7758-44a8-82e0-76ada8837428 on host user@joviandss-2#joviandss-2. {{(pid=84696) _prepare_image_cache_entry /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:886}} Apr 17 15:34:17 user cinder-volume[84696]: DEBUG cinder.coordination [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Lock "cinder-b3f0dec1-7758-44a8-82e0-76ada8837428" released by "_prepare_image_cache_entry" :: held 0.014s {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:207}} Apr 17 15:34:17 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Attempting to retrieve cache entry for image = b3f0dec1-7758-44a8-82e0-76ada8837428 on host user@joviandss-2#joviandss-2. {{(pid=84696) _create_from_image_cache /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:827}} Apr 17 15:34:17 user cinder-volume[84696]: DEBUG cinder.image.cache [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Found image-volume cache entry: {'id': 1, 'image_id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'volume_id': '006f20ae-5306-44fe-afdf-e433f79284e8', 'host': 'user@joviandss-2#joviandss-2', 'size': 1, 'image_updated_at': datetime.datetime(2023, 4, 17, 15, 19, 55), 'last_used': datetime.datetime(2023, 4, 17, 15, 34, 17, 198672)}. {{(pid=84696) get_entry /opt/stack/cinder/cinder/image/cache.py:76}} Apr 17 15:34:17 user cinder-volume[84696]: DEBUG cinder.image.cache [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Image-volume cache entry image_update_at = 2023-04-17 15:19:55+00:00, requested image updated_at = 2023-04-17 15:19:55+00:00. {{(pid=84696) _should_update_entry /opt/stack/cinder/cinder/image/cache.py:259}} Apr 17 15:34:17 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Creating from source image-volume 006f20ae-5306-44fe-afdf-e433f79284e8 {{(pid=84696) _create_from_image_cache /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:841}} Apr 17 15:34:17 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] cloned volume 006f20ae-5306-44fe-afdf-e433f79284e8 to e80c78b2-20c9-4f9b-8898-6ab131d1eb06 {{(pid=84696) create_cloned_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:521}} Apr 17 15:34:17 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] cloning v_006f20ae-5306-44fe-afdf-e433f79284e8 to v_e80c78b2-20c9-4f9b-8898-6ab131d1eb06 {{(pid=84696) _clone_object /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:456}} Apr 17 15:34:17 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] create snapshot v_e80c78b2-20c9-4f9b-8898-6ab131d1eb06 {{(pid=84696) create_snapshot /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:635}} Apr 17 15:34:17 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:34:17 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:34:19 user cinder-volume[84696]: DEBUG cinder.utils [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.077(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:34:19 user cinder-volume[84696]: DEBUG cinder.utils [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:34:22 user cinder-volume[84696]: DEBUG cinder.utils [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.115(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:34:22 user cinder-volume[84696]: DEBUG cinder.utils [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:34:22 user cinder-volume[84696]: DEBUG cinder.utils [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.151(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:34:22 user cinder-volume[84696]: DEBUG cinder.utils [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:34:22 user cinder-volume[84696]: DEBUG cinder.utils [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.433(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:34:22 user cinder-volume[84696]: DEBUG cinder.utils [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_0fed6b21-0520-4554-bee4-9540a03f9af0/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:34:24 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:UPDATE=6,cinder:SELECT=29,cinder:DELETE=1,cinder:INSERT=1 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:34:28 user cinder-volume[84696]: DEBUG cinder.utils [None req-3175029a-5606-46a3-86f4-a16d583ad0ce tempest-TestStampPattern-173447269 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.721(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:34:28 user cinder-volume[84696]: DEBUG cinder.utils [None req-3175029a-5606-46a3-86f4-a16d583ad0ce tempest-TestStampPattern-173447269 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:a9c177a1-61f0-4201-bb8d-7f18c9af396c/luns/v_a9c177a1-61f0-4201-bb8d-7f18c9af396c (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:34:28 user cinder-volume[84696]: DEBUG cinder.utils [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.392(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:34:28 user cinder-volume[84696]: DEBUG cinder.utils [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:34:28 user cinder-volume[84696]: DEBUG cinder.utils [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 73.423(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:34:28 user cinder-volume[84696]: DEBUG cinder.utils [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:34:28 user cinder-volume[84696]: DEBUG cinder.utils [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 11.298(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:34:28 user cinder-volume[84696]: DEBUG cinder.utils [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:34:28 user cinder-volume[84696]: DEBUG cinder.utils [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.767(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:34:28 user cinder-volume[84696]: DEBUG cinder.utils [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_44f909cf-7774-41f6-9f86-805456609ff8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:34:28 user cinder-volume[84696]: DEBUG cinder.utils [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.758(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:34:28 user cinder-volume[84696]: DEBUG cinder.utils [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_a4d96ab8-2072-4eff-b803-53b4f27d056f/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:34:28 user cinder-volume[84696]: DEBUG cinder.utils [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.474(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:34:28 user cinder-volume[84696]: DEBUG cinder.utils [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:34:28 user cinder-volume[84696]: DEBUG cinder.utils [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 73.612(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:34:28 user cinder-volume[84696]: DEBUG cinder.utils [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_62709a81-268b-4227-ba4a-070034f36414/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:34:31 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 73.719(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:34:31 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:34:38 user cinder-volume[84696]: DEBUG cinder.utils [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 74.324(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:34:38 user cinder-volume[84696]: DEBUG cinder.utils [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:34:38 user cinder-volume[84696]: DEBUG cinder.utils [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 74.511(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:34:38 user cinder-volume[84696]: DEBUG cinder.utils [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_17f8d4f7-2dcd-4041-8d0b-5c8afebb75bf/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:34:38 user cinder-volume[84696]: DEBUG cinder.utils [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.411(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:34:38 user cinder-volume[84696]: DEBUG cinder.utils [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:34:38 user cinder-volume[84696]: DEBUG cinder.utils [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 37.378(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:34:38 user cinder-volume[84696]: DEBUG cinder.utils [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:34:38 user cinder-volume[84696]: DEBUG cinder.utils [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 37.694(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:34:38 user cinder-volume[84696]: DEBUG cinder.utils [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_0fed6b21-0520-4554-bee4-9540a03f9af0/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:34:44 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=14,cinder:UPDATE=4 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:34:44 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Volume reschedule parameters: True retry: {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 15:34:44 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Flow 'volume_create_manager' (d55eec9d-6854-427d-a3dd-46f6883c264e) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:34:44 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (b724dd98-84e2-44ad-809d-9f3d9f9cffb0) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:34:44 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (b724dd98-84e2-44ad-809d-9f3d9f9cffb0) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:34:44Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesSnapshotNegativeTestJSON-Volume-379445405',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=29d9e55e-0b8b-4f70-b5c4-830d39f10fcd,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='0b21bda55be444fc9f1a9c8c37b09224',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:34:45Z,service_uuid=None,shared_targets=True,size=2,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:34:45Z,use_quota=True,user_id='df17d7972cc649eb829bde41b8a718e4',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:34:44 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (8dd354fa-c2fd-47a3-b436-20c24e43b89c) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:34:44 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (8dd354fa-c2fd-47a3-b436-20c24e43b89c) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:34:44 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (329f0f77-6eed-4521-be8e-00d969b90d4c) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:34:44 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (329f0f77-6eed-4521-be8e-00d969b90d4c) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': '29d9e55e-0b8b-4f70-b5c4-830d39f10fcd', 'volume_name': 'volume-29d9e55e-0b8b-4f70-b5c4-830d39f10fcd', 'volume_size': 2}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:34:44 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (a1b2df09-56f1-4ec9-8694-d8d098d9c67e) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:34:44 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (a1b2df09-56f1-4ec9-8694-d8d098d9c67e) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:34:44 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (50a8e45b-fa50-4f45-8385-11804bfc7a3e) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:34:44 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Volume 29d9e55e-0b8b-4f70-b5c4-830d39f10fcd: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-29d9e55e-0b8b-4f70-b5c4-830d39f10fcd', 'volume_size': 2} Apr 17 15:34:44 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] creating volume v_29d9e55e-0b8b-4f70-b5c4-830d39f10fcd. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 15:34:44 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] create volume {'name': 'v_29d9e55e-0b8b-4f70-b5c4-830d39f10fcd', 'size': '2147483648', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 15:34:44 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:34:44 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:34:45 user cinder-volume[84696]: DEBUG cinder.utils [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 74.377(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:34:45 user cinder-volume[84696]: DEBUG cinder.utils [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:34:45 user cinder-volume[84696]: DEBUG cinder.utils [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 0.466(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:34:45 user cinder-volume[84696]: DEBUG cinder.utils [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:34:45 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Volume reschedule parameters: True retry: {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 15:34:45 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Flow 'volume_create_manager' (70c615e4-6344-4f7d-9e21-81127ba4e18f) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:34:45 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (a08f7252-02d4-4246-8712-cb78ca4dbe35) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:34:45 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (a08f7252-02d4-4246-8712-cb78ca4dbe35) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:34:45Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesActionsTest-Volume-737622743',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=44073502-102d-452a-8244-69a8233b9060,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='a8b97ff6ed6047f0a33e74684a1c162b',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:34:45Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:34:45Z,use_quota=True,user_id='c993ceeb4a5c49eea14057e2125518ad',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:34:45 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (dcd2578c-e54b-4460-9110-7a625f5c8f06) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:34:45 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (dcd2578c-e54b-4460-9110-7a625f5c8f06) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:34:45 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (56df3c26-a1d6-42f1-a5e2-343041d15337) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:34:45 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (56df3c26-a1d6-42f1-a5e2-343041d15337) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': '44073502-102d-452a-8244-69a8233b9060', 'volume_name': 'volume-44073502-102d-452a-8244-69a8233b9060', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:34:45 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (c484ab16-8f47-4695-9188-3ab5b6d4ceaa) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:34:45 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (c484ab16-8f47-4695-9188-3ab5b6d4ceaa) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:34:45 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (1fa3fdbc-1dbf-4f77-9f17-81cf9b9780d1) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:34:45 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Volume 44073502-102d-452a-8244-69a8233b9060: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-44073502-102d-452a-8244-69a8233b9060', 'volume_size': 1} Apr 17 15:34:45 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] creating volume v_44073502-102d-452a-8244-69a8233b9060. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 15:34:45 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] create volume {'name': 'v_44073502-102d-452a-8244-69a8233b9060', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 15:34:45 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:34:45 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:34:48 user cinder-volume[84696]: DEBUG cinder.utils [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 3.533(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:34:48 user cinder-volume[84696]: DEBUG cinder.utils [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:34:48 user cinder-volume[84696]: DEBUG cinder.utils [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.815(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:34:48 user cinder-volume[84696]: DEBUG cinder.utils [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:34:51 user cinder-volume[84696]: DEBUG cinder.utils [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 74.865(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:34:51 user cinder-volume[84696]: DEBUG cinder.utils [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_5d9c86cd-3e22-4926-bfc9-e5a37f6ae76c/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:34:51 user cinder-volume[84696]: DEBUG cinder.utils [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 74.660(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:34:51 user cinder-volume[84696]: DEBUG cinder.utils [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:34:51 user cinder-volume[84696]: DEBUG cinder.utils [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.890(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:34:51 user cinder-volume[84696]: DEBUG cinder.utils [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:34:54 user cinder-volume[84696]: DEBUG cinder.utils [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 9.677(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:34:54 user cinder-volume[84696]: DEBUG cinder.utils [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:34:57 user cinder-volume[84696]: DEBUG cinder.utils [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 40.221(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:34:57 user cinder-volume[84696]: DEBUG cinder.utils [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:34:57 user cinder-volume[84696]: DEBUG cinder.utils [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.031(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:34:57 user cinder-volume[84696]: DEBUG cinder.utils [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:34:57 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Volume reschedule parameters: True retry: {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 15:34:57 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Flow 'volume_create_manager' (fe34318b-61e5-4779-b002-9463edfd5e99) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:34:57 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (c781a08d-c532-4244-8543-10941f333b67) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:34:57 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (c781a08d-c532-4244-8543-10941f333b67) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:34:57Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesCloneTest-Volume-2030679513',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=153b4df8-1f37-4118-8aa6-1c155ea5700c,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='82fce1724afa40dd914e0791c3ee69a3',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:34:58Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:34:58Z,use_quota=True,user_id='773ce6c2a0f844edaf666e190504e258',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:34:57 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (dc6d0e22-ed05-4c9f-aa05-9065a5afc1a6) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:34:57 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (dc6d0e22-ed05-4c9f-aa05-9065a5afc1a6) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:34:57 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (0b7ec015-7c9a-4fa0-adc5-8827b35235f9) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:34:57 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (0b7ec015-7c9a-4fa0-adc5-8827b35235f9) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': '153b4df8-1f37-4118-8aa6-1c155ea5700c', 'volume_name': 'volume-153b4df8-1f37-4118-8aa6-1c155ea5700c', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:34:57 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (1a455c89-502c-4392-bdb8-2b40a1b1f1eb) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:34:57 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (1a455c89-502c-4392-bdb8-2b40a1b1f1eb) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:34:57 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (02a1a044-9922-44cb-ba7e-18400be516a1) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:34:57 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Volume 153b4df8-1f37-4118-8aa6-1c155ea5700c: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-153b4df8-1f37-4118-8aa6-1c155ea5700c', 'volume_size': 1} Apr 17 15:34:57 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] creating volume v_153b4df8-1f37-4118-8aa6-1c155ea5700c. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 15:34:57 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] create volume {'name': 'v_153b4df8-1f37-4118-8aa6-1c155ea5700c', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 15:34:57 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:34:57 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:35:00 user cinder-volume[84696]: DEBUG cinder.utils [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.846(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:35:00 user cinder-volume[84696]: DEBUG cinder.utils [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:35:03 user cinder-volume[84696]: DEBUG cinder.utils [None req-3175029a-5606-46a3-86f4-a16d583ad0ce tempest-TestStampPattern-173447269 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 74.793(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:35:03 user cinder-volume[84696]: DEBUG cinder.utils [None req-3175029a-5606-46a3-86f4-a16d583ad0ce tempest-TestStampPattern-173447269 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:a9c177a1-61f0-4201-bb8d-7f18c9af396c/luns/v_a9c177a1-61f0-4201-bb8d-7f18c9af396c (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:35:03 user cinder-volume[84696]: DEBUG cinder.utils [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 74.835(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:35:03 user cinder-volume[84696]: DEBUG cinder.utils [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_44f909cf-7774-41f6-9f86-805456609ff8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:35:03 user cinder-volume[84696]: DEBUG cinder.utils [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 74.465(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:35:03 user cinder-volume[84696]: DEBUG cinder.utils [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:35:03 user cinder-volume[84696]: DEBUG cinder.utils [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 74.828(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:35:03 user cinder-volume[84696]: DEBUG cinder.utils [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_a4d96ab8-2072-4eff-b803-53b4f27d056f/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:35:03 user cinder-volume[84696]: DEBUG cinder.utils [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 74.545(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:35:03 user cinder-volume[84696]: DEBUG cinder.utils [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:35:03 user cinder-volume[84696]: DEBUG cinder.utils [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.926(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:35:03 user cinder-volume[84696]: DEBUG cinder.utils [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:35:03 user cinder-volume[84696]: DEBUG cinder.utils [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 18.902(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:35:03 user cinder-volume[84696]: DEBUG cinder.utils [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:35:06 user cinder-volume[84696]: DEBUG cinder.utils [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.245(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:35:06 user cinder-volume[84696]: DEBUG cinder.utils [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:35:09 user cinder-volume[84696]: DEBUG cinder.utils [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.063(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:35:09 user cinder-volume[84696]: DEBUG cinder.utils [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:35:12 user cinder-volume[84696]: DEBUG cinder.utils [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 71.862(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:35:12 user cinder-volume[84696]: DEBUG cinder.utils [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_0fed6b21-0520-4554-bee4-9540a03f9af0/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:35:12 user cinder-volume[84696]: DEBUG cinder.utils [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 71.549(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:35:12 user cinder-volume[84696]: DEBUG cinder.utils [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:35:14 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=14,cinder:UPDATE=3 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:35:14 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=11,cinder:UPDATE=3 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:35:19 user cinder-volume[84696]: DEBUG cinder.utils [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 34.989(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:35:19 user cinder-volume[84696]: DEBUG cinder.utils [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:35:19 user cinder-volume[84696]: DEBUG cinder.utils [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 22.017(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:35:19 user cinder-volume[84696]: DEBUG cinder.utils [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:35:22 user cinder-volume[84696]: DEBUG cinder.utils [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 37.341(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:35:22 user cinder-volume[84696]: DEBUG cinder.utils [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:35:29 user cinder-volume[84696]: DEBUG cinder.utils [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 72.513(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:35:29 user cinder-volume[84696]: DEBUG cinder.utils [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:35:33 user cinder-volume[84696]: DEBUG cinder.utils [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 138.755(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:35:33 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_62709a81-268b-4227-ba4a-070034f36414/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:35:33 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (f282ab50-6be5-4b8b-a8ff-74df520036f3) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 15:35:33 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (2da3cb5d-0cee-45d1-9121-1f4d3236e757) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:35:33 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (2da3cb5d-0cee-45d1-9121-1f4d3236e757) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:35:33 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Flow 'volume_create_manager' (45c1607c-735d-41db-8ea0-22e6846280ce) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 15:35:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 15:35:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 15:35:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 15:35:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 15:35:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 15:35:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 15:35:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 15:35:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 15:35:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 15:35:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 15:35:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 15:35:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 15:35:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 15:35:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 15:35:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 15:35:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 15:35:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 15:35:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 15:35:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 15:35:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 15:35:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 15:35:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 15:35:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 15:35:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 15:35:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:35:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 15:35:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 15:35:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 15:35:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 15:35:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 15:35:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 15:35:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 15:35:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 15:35:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 15:35:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 15:35:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 15:35:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 15:35:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 15:35:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 15:35:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 15:35:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 15:35:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 15:35:33 user cinder-volume[84696]: DEBUG cinder.utils [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 138.590(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:35:33 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:35:33 user cinder-volume[84696]: INFO cinder.message.api [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Creating message record for request_id = req-e7b5301d-81ea-4ab5-9d69-766347595e11 Apr 17 15:35:33 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 15:35:33 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (adab2f28-31c5-4cfd-8a75-4410878080db) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 15:35:33 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 15:35:33 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:28:12Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesActionsTest-Volume-620204537',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=62709a81-268b-4227-ba4a-070034f36414,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='a8b97ff6ed6047f0a33e74684a1c162b',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:33:15Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:33:15Z,use_quota=True,user_id='c993ceeb4a5c49eea14057e2125518ad',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 15:35:33 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:28:12Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesActionsTest-Volume-620204537',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=62709a81-268b-4227-ba4a-070034f36414,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='a8b97ff6ed6047f0a33e74684a1c162b',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:33:15Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:33:15Z,use_quota=True,user_id='c993ceeb4a5c49eea14057e2125518ad',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(62709a81-268b-4227-ba4a-070034f36414),volume_id=62709a81-268b-4227-ba4a-070034f36414,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': '62709a81-268b-4227-ba4a-070034f36414', 'volume_name': 'volume-62709a81-268b-4227-ba4a-070034f36414', 'volume_size': 1}} Apr 17 15:35:33 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': '62709a81-268b-4227-ba4a-070034f36414', 'volume': {'id': '62709a81-268b-4227-ba4a-070034f36414', '_name_id': None, 'ec2_id': None, 'user_id': 'c993ceeb4a5c49eea14057e2125518ad', 'project_id': 'a8b97ff6ed6047f0a33e74684a1c162b', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-VolumesActionsTest-Volume-620204537', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'admin_metadata': {}, 'glance_metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T15:28:12.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-62709a81-268b-4227-ba4a-070034f36414', 'name_id': '62709a81-268b-4227-ba4a-070034f36414', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumesActionsTest-Volume-620204537', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': 'a8b97ff6ed6047f0a33e74684a1c162b', 'qos_specs': None, 'reservations': ['1b325ba3-edbb-4c50-91e6-5be6258c7eee', '4eb6a10f-f088-4780-a8d5-294b078de0ce', 'ede206e9-4b4d-4f26-8a58-97a099b605d5', '2beaf0e1-5633-41fc-ad41-398657c8ddea'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'c993ceeb4a5c49eea14057e2125518ad', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumesActionsTest-Volume-620204537', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': 'a8b97ff6ed6047f0a33e74684a1c162b', 'qos_specs': None, 'reservations': ['1b325ba3-edbb-4c50-91e6-5be6258c7eee', '4eb6a10f-f088-4780-a8d5-294b078de0ce', 'ede206e9-4b4d-4f26-8a58-97a099b605d5', '2beaf0e1-5633-41fc-ad41-398657c8ddea'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'c993ceeb4a5c49eea14057e2125518ad', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': 'c993ceeb4a5c49eea14057e2125518ad', 'metadata': {}, 'qos_specs': None, 'retry': {'num_attempts': 3, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']}}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(62709a81-268b-4227-ba4a-070034f36414),volume_id=62709a81-268b-4227-ba4a-070034f36414,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:28:12Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesActionsTest-Volume-620204537',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=62709a81-268b-4227-ba4a-070034f36414,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='a8b97ff6ed6047f0a33e74684a1c162b',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:33:15Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:33:15Z,use_quota=True,user_id='c993ceeb4a5c49eea14057e2125518ad',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 15:35:33 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:28:12Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesActionsTest-Volume-620204537',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=62709a81-268b-4227-ba4a-070034f36414,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='a8b97ff6ed6047f0a33e74684a1c162b',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:33:15Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:33:15Z,use_quota=True,user_id='c993ceeb4a5c49eea14057e2125518ad',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:28:12Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesActionsTest-Volume-620204537',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=62709a81-268b-4227-ba4a-070034f36414,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='a8b97ff6ed6047f0a33e74684a1c162b',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:33:15Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:33:15Z,use_quota=True,user_id='c993ceeb4a5c49eea14057e2125518ad',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 15:35:33 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 15:35:33 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 15:35:33 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:35:33 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 15:35:33 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 15:35:33 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 15:35:33 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 15:35:33 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 15:35:33 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 15:35:33 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 15:35:33 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 15:35:33 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 15:35:33 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 15:35:33 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 15:35:33 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 15:35:33 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 15:35:33 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 15:35:33 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 15:35:33 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 15:35:33 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 15:35:33 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (adab2f28-31c5-4cfd-8a75-4410878080db) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:35:33 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (adab2f28-31c5-4cfd-8a75-4410878080db) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:35:33 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (96d31545-d396-468c-a380-730993851e93) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:35:33 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (96d31545-d396-468c-a380-730993851e93) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:35:33 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (ac1057f3-d93d-42c7-b785-83e64bf096e3) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:35:33 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (ac1057f3-d93d-42c7-b785-83e64bf096e3) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:35:33 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (416c7a27-3bbf-4426-ba12-c01dcd02cce5) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:35:33 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Updating volume 62709a81-268b-4227-ba4a-070034f36414 with {'scheduled_at': datetime.datetime(2023, 4, 17, 15, 35, 33, 742598), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 15:35:33 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Volume 62709a81-268b-4227-ba4a-070034f36414: re-scheduling SchedulerAPI.create_volume attempt 3 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 15:35:33 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Volume 62709a81-268b-4227-ba4a-070034f36414: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 15:35:33 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] deleating volume v_62709a81-268b-4227-ba4a-070034f36414 {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 15:35:33 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] get snapshots for volume v_62709a81-268b-4227-ba4a-070034f36414 {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 15:35:33 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_62709a81-268b-4227-ba4a-070034f36414/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:35:33 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_62709a81-268b-4227-ba4a-070034f36414/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:35:36 user cinder-volume[84696]: DEBUG cinder.utils [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.962(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:35:36 user cinder-volume[84696]: DEBUG cinder.utils [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_62709a81-268b-4227-ba4a-070034f36414/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:35:36 user cinder-volume[84696]: DEBUG cinder.utils [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.072(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:35:36 user cinder-volume[84696]: DEBUG cinder.utils [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:35:36 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 138.874(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:35:36 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:35:36 user cinder-volume[84696]: ERROR oslo_service.periodic_task [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Error during VolumeManager.publish_service_capabilities: TypeError: 'NoneType' object is not subscriptable Apr 17 15:35:36 user cinder-volume[84696]: ERROR oslo_service.periodic_task Traceback (most recent call last): Apr 17 15:35:36 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/usr/local/lib/python3.10/dist-packages/oslo_service/periodic_task.py", line 216, in run_periodic_tasks Apr 17 15:35:36 user cinder-volume[84696]: ERROR oslo_service.periodic_task task(self, context) Apr 17 15:35:36 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/manager.py", line 2876, in publish_service_capabilities Apr 17 15:35:36 user cinder-volume[84696]: ERROR oslo_service.periodic_task self._report_driver_status(context) Apr 17 15:35:36 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/manager.py", line 2755, in _report_driver_status Apr 17 15:35:36 user cinder-volume[84696]: ERROR oslo_service.periodic_task volume_stats = get_stats() Apr 17 15:35:36 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/usr/local/lib/python3.10/dist-packages/oslo_utils/timeutils.py", line 319, in wrapper Apr 17 15:35:36 user cinder-volume[84696]: ERROR oslo_service.periodic_task result = func(*args, **kwargs) Apr 17 15:35:36 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/manager.py", line 2753, in get_stats Apr 17 15:35:36 user cinder-volume[84696]: ERROR oslo_service.periodic_task return self.driver.get_volume_stats(refresh=True) Apr 17 15:35:36 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/driver.py", line 666, in get_volume_stats Apr 17 15:35:36 user cinder-volume[84696]: ERROR oslo_service.periodic_task self._update_volume_stats() Apr 17 15:35:36 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 726, in _update_volume_stats Apr 17 15:35:36 user cinder-volume[84696]: ERROR oslo_service.periodic_task pool_stats = self.ra.get_pool_stats() Apr 17 15:35:36 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 897, in get_pool_stats Apr 17 15:35:36 user cinder-volume[84696]: ERROR oslo_service.periodic_task if not resp["error"] and resp["code"] == 200: Apr 17 15:35:36 user cinder-volume[84696]: ERROR oslo_service.periodic_task TypeError: 'NoneType' object is not subscriptable Apr 17 15:35:36 user cinder-volume[84696]: ERROR oslo_service.periodic_task Apr 17 15:35:36 user cinder-volume[84696]: WARNING oslo.service.loopingcall [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Function 'cinder.service.Service.periodic_tasks' run outlasted interval by 78.88 sec Apr 17 15:35:36 user cinder-volume[84696]: DEBUG oslo_service.periodic_task [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Running periodic task VolumeManager.publish_service_capabilities {{(pid=84696) run_periodic_tasks /usr/local/lib/python3.10/dist-packages/oslo_service/periodic_task.py:210}} Apr 17 15:35:36 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Updating volume stats {{(pid=84696) _update_volume_stats /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:724}} Apr 17 15:35:36 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Get pool Pool-0 fsprops {{(pid=84696) get_pool_stats /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:894}} Apr 17 15:35:36 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:35:36 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:35:39 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 3.064(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:35:39 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:35:39 user cinder-volume[84696]: DEBUG cinder.utils [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.036(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:35:39 user cinder-volume[84696]: DEBUG cinder.utils [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_62709a81-268b-4227-ba4a-070034f36414/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:35:42 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.136(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:35:42 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:35:42 user cinder-volume[84696]: DEBUG cinder.utils [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 138.763(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:35:42 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_17f8d4f7-2dcd-4041-8d0b-5c8afebb75bf/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:35:42 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (0cff7168-848c-4fc6-98f8-492aeec7e936) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 15:35:42 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (a8345d2b-66ce-4522-b887-89959f362043) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:35:42 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (a8345d2b-66ce-4522-b887-89959f362043) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:35:42 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Flow 'volume_create_manager' (49907c15-6e1c-41b2-acbc-bfe4f1435174) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 15:35:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 15:35:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 15:35:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 15:35:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 15:35:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 15:35:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 15:35:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 15:35:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 15:35:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 15:35:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 15:35:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 15:35:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 15:35:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 15:35:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 15:35:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 15:35:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 15:35:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 15:35:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 15:35:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 15:35:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 15:35:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 15:35:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 15:35:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 15:35:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 15:35:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:35:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 15:35:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 15:35:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 15:35:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 15:35:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 15:35:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 15:35:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 15:35:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 15:35:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 15:35:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 15:35:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 15:35:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 15:35:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 15:35:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 15:35:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 15:35:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 15:35:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 15:35:42 user cinder-volume[84696]: DEBUG cinder.utils [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 138.595(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:35:42 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:35:42 user cinder-volume[84696]: INFO cinder.message.api [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Creating message record for request_id = req-051b1fdc-54da-4587-a61f-deb7faa24c2c Apr 17 15:35:42 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 15:35:42 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (c135b8bf-a00b-4c1a-bc39-6239684fdf33) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 15:35:42 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 15:35:42 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:28:22Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-SnapshotMetadataTestJSON-Volume-902610845',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=17f8d4f7-2dcd-4041-8d0b-5c8afebb75bf,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='a4653bf4d2914c8d956e86ad3405051d',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:33:24Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:33:24Z,use_quota=True,user_id='5598ef2700af47d9bcf6fadea4460cb2',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 15:35:42 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:28:22Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-SnapshotMetadataTestJSON-Volume-902610845',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=17f8d4f7-2dcd-4041-8d0b-5c8afebb75bf,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='a4653bf4d2914c8d956e86ad3405051d',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:33:24Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:33:24Z,use_quota=True,user_id='5598ef2700af47d9bcf6fadea4460cb2',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(17f8d4f7-2dcd-4041-8d0b-5c8afebb75bf),volume_id=17f8d4f7-2dcd-4041-8d0b-5c8afebb75bf,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': '17f8d4f7-2dcd-4041-8d0b-5c8afebb75bf', 'volume_name': 'volume-17f8d4f7-2dcd-4041-8d0b-5c8afebb75bf', 'volume_size': 1}} Apr 17 15:35:42 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': '17f8d4f7-2dcd-4041-8d0b-5c8afebb75bf', 'volume': {'id': '17f8d4f7-2dcd-4041-8d0b-5c8afebb75bf', '_name_id': None, 'ec2_id': None, 'user_id': '5598ef2700af47d9bcf6fadea4460cb2', 'project_id': 'a4653bf4d2914c8d956e86ad3405051d', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-SnapshotMetadataTestJSON-Volume-902610845', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'admin_metadata': {}, 'glance_metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T15:28:22.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-17f8d4f7-2dcd-4041-8d0b-5c8afebb75bf', 'name_id': '17f8d4f7-2dcd-4041-8d0b-5c8afebb75bf', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-SnapshotMetadataTestJSON-Volume-902610845', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': 'a4653bf4d2914c8d956e86ad3405051d', 'qos_specs': None, 'reservations': ['ab8ffe08-13a2-4be5-8c18-55343d579bf3', 'd62e062b-7f8a-4b18-8a96-5ae0097970db', '0311c219-244b-45f9-903f-bdbceee93caf', 'f31b8283-7526-44ae-bd04-3726c40be823'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': '5598ef2700af47d9bcf6fadea4460cb2', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-SnapshotMetadataTestJSON-Volume-902610845', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': 'a4653bf4d2914c8d956e86ad3405051d', 'qos_specs': None, 'reservations': ['ab8ffe08-13a2-4be5-8c18-55343d579bf3', 'd62e062b-7f8a-4b18-8a96-5ae0097970db', '0311c219-244b-45f9-903f-bdbceee93caf', 'f31b8283-7526-44ae-bd04-3726c40be823'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': '5598ef2700af47d9bcf6fadea4460cb2', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': '5598ef2700af47d9bcf6fadea4460cb2', 'metadata': {}, 'qos_specs': None, 'retry': {'num_attempts': 3, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']}}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(17f8d4f7-2dcd-4041-8d0b-5c8afebb75bf),volume_id=17f8d4f7-2dcd-4041-8d0b-5c8afebb75bf,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:28:22Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-SnapshotMetadataTestJSON-Volume-902610845',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=17f8d4f7-2dcd-4041-8d0b-5c8afebb75bf,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='a4653bf4d2914c8d956e86ad3405051d',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:33:24Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:33:24Z,use_quota=True,user_id='5598ef2700af47d9bcf6fadea4460cb2',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 15:35:42 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:28:22Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-SnapshotMetadataTestJSON-Volume-902610845',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=17f8d4f7-2dcd-4041-8d0b-5c8afebb75bf,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='a4653bf4d2914c8d956e86ad3405051d',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:33:24Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:33:24Z,use_quota=True,user_id='5598ef2700af47d9bcf6fadea4460cb2',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:28:22Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-SnapshotMetadataTestJSON-Volume-902610845',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=17f8d4f7-2dcd-4041-8d0b-5c8afebb75bf,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='a4653bf4d2914c8d956e86ad3405051d',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:33:24Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:33:24Z,use_quota=True,user_id='5598ef2700af47d9bcf6fadea4460cb2',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 15:35:42 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 15:35:42 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 15:35:42 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:35:42 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 15:35:42 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 15:35:42 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 15:35:42 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 15:35:42 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 15:35:42 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 15:35:42 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 15:35:42 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 15:35:42 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 15:35:42 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 15:35:42 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 15:35:42 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 15:35:42 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 15:35:42 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 15:35:42 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 15:35:42 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 15:35:42 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 15:35:42 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (c135b8bf-a00b-4c1a-bc39-6239684fdf33) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:35:42 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (c135b8bf-a00b-4c1a-bc39-6239684fdf33) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:35:42 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (683ddc71-fc21-4551-b496-26f8bdc771ea) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:35:42 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (683ddc71-fc21-4551-b496-26f8bdc771ea) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:35:42 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (ae62ee66-a4cf-459b-a80f-168a160768f7) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:35:42 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (ae62ee66-a4cf-459b-a80f-168a160768f7) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:35:42 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (68a53ea2-b4bc-426e-bf76-3d7cdf44638d) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:35:42 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Updating volume 17f8d4f7-2dcd-4041-8d0b-5c8afebb75bf with {'scheduled_at': datetime.datetime(2023, 4, 17, 15, 35, 42, 982782), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 15:35:43 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Volume 17f8d4f7-2dcd-4041-8d0b-5c8afebb75bf: re-scheduling SchedulerAPI.create_volume attempt 3 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 15:35:43 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Volume 17f8d4f7-2dcd-4041-8d0b-5c8afebb75bf: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 15:35:43 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] deleating volume v_17f8d4f7-2dcd-4041-8d0b-5c8afebb75bf {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 15:35:43 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] get snapshots for volume v_17f8d4f7-2dcd-4041-8d0b-5c8afebb75bf {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 15:35:43 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_17f8d4f7-2dcd-4041-8d0b-5c8afebb75bf/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:35:43 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_17f8d4f7-2dcd-4041-8d0b-5c8afebb75bf/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:35:45 user cinder-volume[84696]: DEBUG cinder.utils [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.905(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:35:45 user cinder-volume[84696]: DEBUG cinder.utils [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_17f8d4f7-2dcd-4041-8d0b-5c8afebb75bf/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:35:45 user cinder-volume[84696]: DEBUG cinder.utils [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.181(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:35:45 user cinder-volume[84696]: DEBUG cinder.utils [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_62709a81-268b-4227-ba4a-070034f36414/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:35:49 user cinder-volume[84696]: DEBUG cinder.utils [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.977(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:35:49 user cinder-volume[84696]: DEBUG cinder.utils [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_17f8d4f7-2dcd-4041-8d0b-5c8afebb75bf/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:35:49 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.281(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:35:49 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:35:52 user cinder-volume[84696]: DEBUG cinder.utils [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 141.320(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:35:52 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:35:52 user cinder-volume[84696]: INFO cinder.message.api [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Creating message record for request_id = req-822bc81b-4af7-4475-af70-370e1870fe50 Apr 17 15:35:52 user cinder-volume[84696]: DEBUG cinder.utils [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 67.409(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:35:52 user cinder-volume[84696]: DEBUG cinder.utils [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:35:52 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 15:35:52 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=11,cinder:UPDATE=4,cinder:INSERT=2 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:35:52 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=8,cinder:UPDATE=4,cinder:INSERT=3 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:35:52 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (c9144115-8b7f-4295-815d-4c8e8626f5fe) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 15:35:52 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 15:35:52 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:33:31Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-ServersNegativeTestJSON-volume-1746588226',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=8cc35ab1-363f-4ad6-9a37-239d2b1ea996,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='d2dd48f279404859bebcc19bcc272915',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:33:31Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:33:31Z,use_quota=True,user_id='b0075b26d6044a50981d29df0d863959',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 15:35:52 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:33:31Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-ServersNegativeTestJSON-volume-1746588226',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=8cc35ab1-363f-4ad6-9a37-239d2b1ea996,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='d2dd48f279404859bebcc19bcc272915',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:33:31Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:33:31Z,use_quota=True,user_id='b0075b26d6044a50981d29df0d863959',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(8cc35ab1-363f-4ad6-9a37-239d2b1ea996),volume_id=8cc35ab1-363f-4ad6-9a37-239d2b1ea996,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': '8cc35ab1-363f-4ad6-9a37-239d2b1ea996', 'volume_name': 'volume-8cc35ab1-363f-4ad6-9a37-239d2b1ea996', 'volume_size': 1}} Apr 17 15:35:52 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'retry': {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']}, 'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': '8cc35ab1-363f-4ad6-9a37-239d2b1ea996', 'volume': {'id': '8cc35ab1-363f-4ad6-9a37-239d2b1ea996', '_name_id': None, 'ec2_id': None, 'user_id': 'b0075b26d6044a50981d29df0d863959', 'project_id': 'd2dd48f279404859bebcc19bcc272915', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-ServersNegativeTestJSON-volume-1746588226', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T15:33:30.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-8cc35ab1-363f-4ad6-9a37-239d2b1ea996', 'name_id': '8cc35ab1-363f-4ad6-9a37-239d2b1ea996', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-ServersNegativeTestJSON-volume-1746588226', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': 'd2dd48f279404859bebcc19bcc272915', 'qos_specs': None, 'reservations': ['1161f3e2-1648-485c-8f2d-e7d723e7cefb', 'b3435402-b7a6-4cc8-ada5-e3f921682979', '131f82e8-6bdb-4b90-85b1-3d76ab20e6b8', '5d42d0e7-f4e0-4d14-83c2-a0df94279daf'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'b0075b26d6044a50981d29df0d863959', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-ServersNegativeTestJSON-volume-1746588226', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': 'd2dd48f279404859bebcc19bcc272915', 'qos_specs': None, 'reservations': ['1161f3e2-1648-485c-8f2d-e7d723e7cefb', 'b3435402-b7a6-4cc8-ada5-e3f921682979', '131f82e8-6bdb-4b90-85b1-3d76ab20e6b8', '5d42d0e7-f4e0-4d14-83c2-a0df94279daf'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'b0075b26d6044a50981d29df0d863959', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': 'b0075b26d6044a50981d29df0d863959', 'metadata': {}, 'qos_specs': None}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(8cc35ab1-363f-4ad6-9a37-239d2b1ea996),volume_id=8cc35ab1-363f-4ad6-9a37-239d2b1ea996,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:33:31Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-ServersNegativeTestJSON-volume-1746588226',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=8cc35ab1-363f-4ad6-9a37-239d2b1ea996,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='d2dd48f279404859bebcc19bcc272915',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:33:31Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:33:31Z,use_quota=True,user_id='b0075b26d6044a50981d29df0d863959',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 15:35:52 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:33:31Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-ServersNegativeTestJSON-volume-1746588226',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=8cc35ab1-363f-4ad6-9a37-239d2b1ea996,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='d2dd48f279404859bebcc19bcc272915',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:33:31Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:33:31Z,use_quota=True,user_id='b0075b26d6044a50981d29df0d863959',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:33:31Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-ServersNegativeTestJSON-volume-1746588226',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=8cc35ab1-363f-4ad6-9a37-239d2b1ea996,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='d2dd48f279404859bebcc19bcc272915',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:33:31Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:33:31Z,use_quota=True,user_id='b0075b26d6044a50981d29df0d863959',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 15:35:52 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 15:35:52 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 15:35:52 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:35:52 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 15:35:52 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 15:35:52 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 15:35:52 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 15:35:52 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 15:35:52 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 15:35:52 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 15:35:52 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 15:35:52 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 15:35:52 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 15:35:52 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 15:35:52 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 15:35:52 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 15:35:52 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 15:35:52 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 15:35:52 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 15:35:52 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 15:35:52 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (c9144115-8b7f-4295-815d-4c8e8626f5fe) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:35:52 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (c9144115-8b7f-4295-815d-4c8e8626f5fe) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:35:52 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (fed44b7b-4a47-4763-ac5e-6723c995f5c7) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:35:52 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (fed44b7b-4a47-4763-ac5e-6723c995f5c7) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:35:52 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (c709a820-1c0d-4a42-b0d0-51cfb8321b74) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:35:52 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (c709a820-1c0d-4a42-b0d0-51cfb8321b74) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:35:52 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (0d5a3677-e116-474e-95e4-84189a7a0841) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:35:52 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Updating volume 8cc35ab1-363f-4ad6-9a37-239d2b1ea996 with {'scheduled_at': datetime.datetime(2023, 4, 17, 15, 35, 52, 155948), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 15:35:52 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Volume 8cc35ab1-363f-4ad6-9a37-239d2b1ea996: re-scheduling SchedulerAPI.create_volume attempt 1 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 15:35:52 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Volume 8cc35ab1-363f-4ad6-9a37-239d2b1ea996: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 15:35:52 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] deleating volume v_8cc35ab1-363f-4ad6-9a37-239d2b1ea996 {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 15:35:52 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] get snapshots for volume v_8cc35ab1-363f-4ad6-9a37-239d2b1ea996 {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 15:35:52 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_8cc35ab1-363f-4ad6-9a37-239d2b1ea996/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:35:52 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_8cc35ab1-363f-4ad6-9a37-239d2b1ea996/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:35:52 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Volume reschedule parameters: True retry: {'num_attempts': 2, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 15:35:52 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Flow 'volume_create_manager' (63832f38-dba4-49a1-beef-e4709ad0af78) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:35:52 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (b4343f33-09c3-438a-97a1-2c34cd07593b) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:35:52 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (b4343f33-09c3-438a-97a1-2c34cd07593b) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:33:31Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-ServersNegativeTestJSON-volume-1746588226',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=8cc35ab1-363f-4ad6-9a37-239d2b1ea996,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='d2dd48f279404859bebcc19bcc272915',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:35:52Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:35:52Z,use_quota=True,user_id='b0075b26d6044a50981d29df0d863959',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:35:52 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (c5b6afb4-5dd5-4a89-816d-25f24aac2e91) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:35:52 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (c5b6afb4-5dd5-4a89-816d-25f24aac2e91) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:35:52 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (60711e10-2763-486c-821b-597051575b6c) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:35:52 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (60711e10-2763-486c-821b-597051575b6c) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': '8cc35ab1-363f-4ad6-9a37-239d2b1ea996', 'volume_name': 'volume-8cc35ab1-363f-4ad6-9a37-239d2b1ea996', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:35:52 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (778d0fe4-4c8f-4209-a406-e7fa04393194) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:35:52 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (778d0fe4-4c8f-4209-a406-e7fa04393194) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:35:52 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (40958e61-dda1-4e8e-858f-047b135f1e2b) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:35:52 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Volume 8cc35ab1-363f-4ad6-9a37-239d2b1ea996: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-8cc35ab1-363f-4ad6-9a37-239d2b1ea996', 'volume_size': 1} Apr 17 15:35:52 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] creating volume v_8cc35ab1-363f-4ad6-9a37-239d2b1ea996. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 15:35:52 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] create volume {'name': 'v_8cc35ab1-363f-4ad6-9a37-239d2b1ea996', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 15:35:52 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:35:52 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:35:55 user cinder-volume[84696]: DEBUG cinder.utils [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.394(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:35:55 user cinder-volume[84696]: DEBUG cinder.utils [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_62709a81-268b-4227-ba4a-070034f36414/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:35:55 user cinder-volume[84696]: DEBUG cinder.utils [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.984(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:35:55 user cinder-volume[84696]: DEBUG cinder.utils [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_8cc35ab1-363f-4ad6-9a37-239d2b1ea996/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:35:55 user cinder-volume[84696]: DEBUG cinder.utils [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.123(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:35:55 user cinder-volume[84696]: DEBUG cinder.utils [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_17f8d4f7-2dcd-4041-8d0b-5c8afebb75bf/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:35:55 user cinder-volume[84696]: DEBUG cinder.utils [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 69.761(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:35:55 user cinder-volume[84696]: DEBUG cinder.utils [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:35:55 user cinder-volume[84696]: DEBUG cinder.utils [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.819(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:35:55 user cinder-volume[84696]: DEBUG cinder.utils [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:35:58 user cinder-volume[84696]: DEBUG cinder.utils [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 141.809(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:35:58 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_5d9c86cd-3e22-4926-bfc9-e5a37f6ae76c/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:35:58 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (29b5c233-605f-4eac-8f7f-55e5c3c0fb1a) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 15:35:58 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (5c41ff22-6093-4dcb-8723-b5c4d8323022) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:35:58 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (5c41ff22-6093-4dcb-8723-b5c4d8323022) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:35:58 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Flow 'volume_create_manager' (3cc5893e-f96c-4dae-87bb-39268ed40abb) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 15:35:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 15:35:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 15:35:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 15:35:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 15:35:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 15:35:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 15:35:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 15:35:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 15:35:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 15:35:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 15:35:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 15:35:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 15:35:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 15:35:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 15:35:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 15:35:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 15:35:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 15:35:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 15:35:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 15:35:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 15:35:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 15:35:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 15:35:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 15:35:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 15:35:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:35:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 15:35:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 15:35:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 15:35:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 15:35:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 15:35:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 15:35:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 15:35:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 15:35:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 15:35:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 15:35:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 15:35:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 15:35:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 15:35:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 15:35:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 15:35:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 15:35:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 15:35:58 user cinder-volume[84696]: DEBUG cinder.utils [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.072(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:35:58 user cinder-volume[84696]: DEBUG cinder.utils [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_8cc35ab1-363f-4ad6-9a37-239d2b1ea996/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:35:58 user cinder-volume[84696]: DEBUG cinder.utils [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 141.621(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:35:58 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:35:58 user cinder-volume[84696]: INFO cinder.message.api [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Creating message record for request_id = req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 Apr 17 15:35:58 user cinder-volume[84696]: DEBUG cinder.utils [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.911(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:35:58 user cinder-volume[84696]: DEBUG cinder.utils [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:35:58 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.520(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:35:58 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:35:58 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 15:35:58 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (8c4c5d14-1b5d-49aa-9ff9-d70176fe940b) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 15:35:58 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 15:35:58 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:31:16Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesServicesTestJSON-Volume-968297362',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=5d9c86cd-3e22-4926-bfc9-e5a37f6ae76c,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='6cf93c7193f842d58d323e62e508a5eb',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:33:36Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:33:36Z,use_quota=True,user_id='225768ea457e484f8790d273f178b879',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 15:35:58 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:31:16Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesServicesTestJSON-Volume-968297362',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=5d9c86cd-3e22-4926-bfc9-e5a37f6ae76c,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='6cf93c7193f842d58d323e62e508a5eb',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:33:36Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:33:36Z,use_quota=True,user_id='225768ea457e484f8790d273f178b879',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(5d9c86cd-3e22-4926-bfc9-e5a37f6ae76c),volume_id=5d9c86cd-3e22-4926-bfc9-e5a37f6ae76c,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': '5d9c86cd-3e22-4926-bfc9-e5a37f6ae76c', 'volume_name': 'volume-5d9c86cd-3e22-4926-bfc9-e5a37f6ae76c', 'volume_size': 1}} Apr 17 15:35:58 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': '5d9c86cd-3e22-4926-bfc9-e5a37f6ae76c', 'volume': {'id': '5d9c86cd-3e22-4926-bfc9-e5a37f6ae76c', '_name_id': None, 'ec2_id': None, 'user_id': '225768ea457e484f8790d273f178b879', 'project_id': '6cf93c7193f842d58d323e62e508a5eb', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-VolumesServicesTestJSON-Volume-968297362', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'admin_metadata': {}, 'glance_metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T15:31:15.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-5d9c86cd-3e22-4926-bfc9-e5a37f6ae76c', 'name_id': '5d9c86cd-3e22-4926-bfc9-e5a37f6ae76c', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumesServicesTestJSON-Volume-968297362', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '6cf93c7193f842d58d323e62e508a5eb', 'qos_specs': None, 'reservations': ['ad0fd489-3554-44ca-a47d-2b78caa30877', 'ee3d5634-17e5-4946-81db-b7539ccc1b48', '991d4c78-00b3-4a69-98e2-174c64fa2675', '86cc929c-76c2-4c7f-a486-8495ac6da9a2'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': '225768ea457e484f8790d273f178b879', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumesServicesTestJSON-Volume-968297362', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '6cf93c7193f842d58d323e62e508a5eb', 'qos_specs': None, 'reservations': ['ad0fd489-3554-44ca-a47d-2b78caa30877', 'ee3d5634-17e5-4946-81db-b7539ccc1b48', '991d4c78-00b3-4a69-98e2-174c64fa2675', '86cc929c-76c2-4c7f-a486-8495ac6da9a2'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': '225768ea457e484f8790d273f178b879', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': '225768ea457e484f8790d273f178b879', 'metadata': {}, 'qos_specs': None, 'retry': {'num_attempts': 2, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']}}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(5d9c86cd-3e22-4926-bfc9-e5a37f6ae76c),volume_id=5d9c86cd-3e22-4926-bfc9-e5a37f6ae76c,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:31:16Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesServicesTestJSON-Volume-968297362',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=5d9c86cd-3e22-4926-bfc9-e5a37f6ae76c,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='6cf93c7193f842d58d323e62e508a5eb',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:33:36Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:33:36Z,use_quota=True,user_id='225768ea457e484f8790d273f178b879',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 15:35:58 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:31:16Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesServicesTestJSON-Volume-968297362',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=5d9c86cd-3e22-4926-bfc9-e5a37f6ae76c,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='6cf93c7193f842d58d323e62e508a5eb',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:33:36Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:33:36Z,use_quota=True,user_id='225768ea457e484f8790d273f178b879',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:31:16Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesServicesTestJSON-Volume-968297362',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=5d9c86cd-3e22-4926-bfc9-e5a37f6ae76c,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='6cf93c7193f842d58d323e62e508a5eb',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:33:36Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:33:36Z,use_quota=True,user_id='225768ea457e484f8790d273f178b879',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 15:35:58 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 15:35:58 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 15:35:58 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:35:58 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 15:35:58 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 15:35:58 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 15:35:58 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 15:35:58 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 15:35:58 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 15:35:58 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 15:35:58 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 15:35:58 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 15:35:58 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 15:35:58 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 15:35:58 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 15:35:58 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 15:35:58 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 15:35:58 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 15:35:58 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 15:35:58 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 15:35:58 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (8c4c5d14-1b5d-49aa-9ff9-d70176fe940b) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:35:58 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (8c4c5d14-1b5d-49aa-9ff9-d70176fe940b) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:35:58 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (7f7cc4e7-dd97-4df7-b262-cfd820193f94) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:35:58 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (7f7cc4e7-dd97-4df7-b262-cfd820193f94) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:35:58 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (7573c4d3-2722-42ef-b4e2-12963bba93f3) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:35:58 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (7573c4d3-2722-42ef-b4e2-12963bba93f3) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:35:58 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (bab63c23-4055-4aca-85fa-c39397afb88d) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:35:58 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Updating volume 5d9c86cd-3e22-4926-bfc9-e5a37f6ae76c with {'scheduled_at': datetime.datetime(2023, 4, 17, 15, 35, 58, 321421), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 15:35:58 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Volume 5d9c86cd-3e22-4926-bfc9-e5a37f6ae76c: re-scheduling SchedulerAPI.create_volume attempt 2 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 15:35:58 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Volume 5d9c86cd-3e22-4926-bfc9-e5a37f6ae76c: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 15:35:58 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] deleating volume v_5d9c86cd-3e22-4926-bfc9-e5a37f6ae76c {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 15:35:58 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] get snapshots for volume v_5d9c86cd-3e22-4926-bfc9-e5a37f6ae76c {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 15:35:58 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_5d9c86cd-3e22-4926-bfc9-e5a37f6ae76c/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:35:58 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_5d9c86cd-3e22-4926-bfc9-e5a37f6ae76c/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:35:58 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Volume reschedule parameters: True retry: {'num_attempts': 3, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 15:35:58 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Flow 'volume_create_manager' (1de8cec6-3135-4047-a7e1-e72cadc5bdd2) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:35:58 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (3d64220f-2ad6-4808-add0-ce29ee542b1c) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:35:58 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (3d64220f-2ad6-4808-add0-ce29ee542b1c) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:31:16Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesServicesTestJSON-Volume-968297362',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=5d9c86cd-3e22-4926-bfc9-e5a37f6ae76c,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='6cf93c7193f842d58d323e62e508a5eb',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:35:59Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:35:59Z,use_quota=True,user_id='225768ea457e484f8790d273f178b879',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:35:58 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (b04285b7-aa2c-4e25-91ee-7eaa61ab5ad3) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:35:58 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (b04285b7-aa2c-4e25-91ee-7eaa61ab5ad3) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:35:58 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (e8f61d79-7eed-4da9-b850-506b5b1f71c0) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:35:58 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (e8f61d79-7eed-4da9-b850-506b5b1f71c0) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': '5d9c86cd-3e22-4926-bfc9-e5a37f6ae76c', 'volume_name': 'volume-5d9c86cd-3e22-4926-bfc9-e5a37f6ae76c', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:35:58 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (81b3d3e2-9250-4276-b1d9-4a599db0b361) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:35:58 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (81b3d3e2-9250-4276-b1d9-4a599db0b361) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:35:58 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (4e33be93-f26b-41f2-aa05-c7fee5b48dfd) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:35:58 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Volume 5d9c86cd-3e22-4926-bfc9-e5a37f6ae76c: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-5d9c86cd-3e22-4926-bfc9-e5a37f6ae76c', 'volume_size': 1} Apr 17 15:35:58 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] creating volume v_5d9c86cd-3e22-4926-bfc9-e5a37f6ae76c. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 15:35:58 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] create volume {'name': 'v_5d9c86cd-3e22-4926-bfc9-e5a37f6ae76c', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 15:35:58 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:35:58 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:36:01 user cinder-volume[84696]: DEBUG cinder.utils [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.958(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:36:01 user cinder-volume[84696]: DEBUG cinder.utils [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_5d9c86cd-3e22-4926-bfc9-e5a37f6ae76c/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:36:01 user cinder-volume[84696]: DEBUG cinder.utils [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.695(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:36:01 user cinder-volume[84696]: DEBUG cinder.utils [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:36:04 user cinder-volume[84696]: DEBUG cinder.utils [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.030(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:36:04 user cinder-volume[84696]: DEBUG cinder.utils [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_5d9c86cd-3e22-4926-bfc9-e5a37f6ae76c/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:36:04 user cinder-volume[84696]: DEBUG cinder.utils [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.338(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:36:04 user cinder-volume[84696]: DEBUG cinder.utils [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_17f8d4f7-2dcd-4041-8d0b-5c8afebb75bf/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:36:04 user cinder-volume[84696]: DEBUG cinder.utils [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.034(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:36:04 user cinder-volume[84696]: DEBUG cinder.utils [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:36:04 user cinder-volume[84696]: DEBUG cinder.utils [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.203(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:36:04 user cinder-volume[84696]: DEBUG cinder.utils [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_8cc35ab1-363f-4ad6-9a37-239d2b1ea996/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:36:04 user cinder-volume[84696]: DEBUG cinder.utils [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.770(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:36:04 user cinder-volume[84696]: DEBUG cinder.utils [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:36:10 user cinder-volume[84696]: DEBUG cinder.utils [None req-3175029a-5606-46a3-86f4-a16d583ad0ce tempest-TestStampPattern-173447269 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 141.867(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:36:10 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-3175029a-5606-46a3-86f4-a16d583ad0ce tempest-TestStampPattern-173447269 None] Geting None from DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:a9c177a1-61f0-4201-bb8d-7f18c9af396c/luns/v_a9c177a1-61f0-4201-bb8d-7f18c9af396c {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:36:10 user cinder-volume[84696]: DEBUG cinder.utils [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.308(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:36:10 user cinder-volume[84696]: DEBUG cinder.utils [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_5d9c86cd-3e22-4926-bfc9-e5a37f6ae76c/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:36:10 user cinder-volume[84696]: DEBUG cinder.utils [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.043(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:36:10 user cinder-volume[84696]: DEBUG cinder.utils [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:36:10 user cinder-volume[84696]: DEBUG cinder.utils [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 141.618(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:36:10 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:36:10 user cinder-volume[84696]: INFO cinder.message.api [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Creating message record for request_id = req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df Apr 17 15:36:10 user cinder-volume[84696]: DEBUG cinder.utils [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 141.906(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:36:10 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_a4d96ab8-2072-4eff-b803-53b4f27d056f/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:36:10 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (c9447cd5-5af6-4746-9bd5-3d7def3fb398) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 15:36:10 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (1c905cf7-91a9-416f-9ba6-b7270ad88166) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:36:10 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (1c905cf7-91a9-416f-9ba6-b7270ad88166) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:36:10 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Flow 'volume_create_manager' (317bb325-927f-4b35-a3ee-ea16297f0cc5) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 15:36:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 15:36:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 15:36:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 15:36:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 15:36:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 15:36:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 15:36:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 15:36:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 15:36:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 15:36:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 15:36:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 15:36:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 15:36:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 15:36:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 15:36:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 15:36:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 15:36:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 15:36:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 15:36:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 15:36:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 15:36:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 15:36:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 15:36:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 15:36:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 15:36:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:36:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 15:36:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 15:36:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 15:36:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 15:36:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 15:36:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 15:36:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 15:36:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 15:36:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 15:36:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 15:36:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 15:36:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 15:36:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 15:36:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 15:36:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 15:36:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 15:36:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 15:36:10 user cinder-volume[84696]: DEBUG cinder.utils [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 73.020(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:36:10 user cinder-volume[84696]: DEBUG cinder.utils [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:36:10 user cinder-volume[84696]: DEBUG cinder.utils [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 141.938(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:36:10 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_44f909cf-7774-41f6-9f86-805456609ff8/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:36:10 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (5ca814ee-ed16-440a-807b-02886c96097c) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 15:36:10 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (8b0cf505-b2d1-4665-83d7-5f45705099b6) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:36:10 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (8b0cf505-b2d1-4665-83d7-5f45705099b6) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:36:10 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Flow 'volume_create_manager' (e20bcd95-ec7f-4e4f-b424-798fc06d1894) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 15:36:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 15:36:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 15:36:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 15:36:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 15:36:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 15:36:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 15:36:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 15:36:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 15:36:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 15:36:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 15:36:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 15:36:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 15:36:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 15:36:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 15:36:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 15:36:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 15:36:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 15:36:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 15:36:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 15:36:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 15:36:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 15:36:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 15:36:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 15:36:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 15:36:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:36:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 15:36:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 15:36:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 15:36:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 15:36:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 15:36:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 15:36:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 15:36:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 15:36:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 15:36:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 15:36:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 15:36:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 15:36:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 15:36:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 15:36:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 15:36:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 15:36:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 15:36:10 user cinder-volume[84696]: DEBUG cinder.utils [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 141.584(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:36:10 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:36:10 user cinder-volume[84696]: INFO cinder.message.api [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Creating message record for request_id = req-6e1f5cff-98a0-49d3-b32d-29a667939689 Apr 17 15:36:10 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 15:36:10 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 15:36:10 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (15b48948-992e-4957-a0c9-08bfade09bc4) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 15:36:10 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 15:36:10 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:29:10Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesAdminNegativeTest-volume-1635380936',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=a4d96ab8-2072-4eff-b803-53b4f27d056f,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='b334c5bc33f143c484518d9bd0e750dd',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:33:49Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:33:49Z,use_quota=True,user_id='3dfadfd30c394b46abe75ec930007850',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 15:36:10 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:29:10Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesAdminNegativeTest-volume-1635380936',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=a4d96ab8-2072-4eff-b803-53b4f27d056f,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='b334c5bc33f143c484518d9bd0e750dd',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:33:49Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:33:49Z,use_quota=True,user_id='3dfadfd30c394b46abe75ec930007850',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(a4d96ab8-2072-4eff-b803-53b4f27d056f),volume_id=a4d96ab8-2072-4eff-b803-53b4f27d056f,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': 'a4d96ab8-2072-4eff-b803-53b4f27d056f', 'volume_name': 'volume-a4d96ab8-2072-4eff-b803-53b4f27d056f', 'volume_size': 1}} Apr 17 15:36:10 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': 'a4d96ab8-2072-4eff-b803-53b4f27d056f', 'volume': {'id': 'a4d96ab8-2072-4eff-b803-53b4f27d056f', '_name_id': None, 'ec2_id': None, 'user_id': '3dfadfd30c394b46abe75ec930007850', 'project_id': 'b334c5bc33f143c484518d9bd0e750dd', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-VolumesAdminNegativeTest-volume-1635380936', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'admin_metadata': {}, 'glance_metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T15:29:10.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-a4d96ab8-2072-4eff-b803-53b4f27d056f', 'name_id': 'a4d96ab8-2072-4eff-b803-53b4f27d056f', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumesAdminNegativeTest-volume-1635380936', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': 'b334c5bc33f143c484518d9bd0e750dd', 'qos_specs': None, 'reservations': ['09f94177-ab67-4a35-91b5-ec04b542f529', '7b3828ae-e615-4ad0-8847-f1bf1e17c182', '8e9d3d0e-b4d4-4bfc-b097-6eaec633b485', 'b33f40af-c5d7-4f4f-b730-d04956abccb8'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': '3dfadfd30c394b46abe75ec930007850', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumesAdminNegativeTest-volume-1635380936', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': 'b334c5bc33f143c484518d9bd0e750dd', 'qos_specs': None, 'reservations': ['09f94177-ab67-4a35-91b5-ec04b542f529', '7b3828ae-e615-4ad0-8847-f1bf1e17c182', '8e9d3d0e-b4d4-4bfc-b097-6eaec633b485', 'b33f40af-c5d7-4f4f-b730-d04956abccb8'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': '3dfadfd30c394b46abe75ec930007850', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': '3dfadfd30c394b46abe75ec930007850', 'metadata': {}, 'qos_specs': None, 'retry': {'num_attempts': 3, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']}}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(a4d96ab8-2072-4eff-b803-53b4f27d056f),volume_id=a4d96ab8-2072-4eff-b803-53b4f27d056f,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:29:10Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesAdminNegativeTest-volume-1635380936',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=a4d96ab8-2072-4eff-b803-53b4f27d056f,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='b334c5bc33f143c484518d9bd0e750dd',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:33:49Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:33:49Z,use_quota=True,user_id='3dfadfd30c394b46abe75ec930007850',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 15:36:10 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:29:10Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesAdminNegativeTest-volume-1635380936',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=a4d96ab8-2072-4eff-b803-53b4f27d056f,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='b334c5bc33f143c484518d9bd0e750dd',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:33:49Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:33:49Z,use_quota=True,user_id='3dfadfd30c394b46abe75ec930007850',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:29:10Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesAdminNegativeTest-volume-1635380936',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=a4d96ab8-2072-4eff-b803-53b4f27d056f,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='b334c5bc33f143c484518d9bd0e750dd',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:33:49Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:33:49Z,use_quota=True,user_id='3dfadfd30c394b46abe75ec930007850',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 15:36:10 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 15:36:10 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 15:36:10 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:36:10 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 15:36:10 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 15:36:10 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 15:36:10 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 15:36:10 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 15:36:10 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 15:36:10 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 15:36:10 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 15:36:10 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 15:36:10 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 15:36:10 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 15:36:10 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 15:36:10 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 15:36:10 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 15:36:10 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 15:36:10 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 15:36:10 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 15:36:10 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (15b48948-992e-4957-a0c9-08bfade09bc4) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:36:10 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (15b48948-992e-4957-a0c9-08bfade09bc4) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:36:10 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (b04db691-f082-4c0f-8173-5261e8d24964) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:36:10 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (b04db691-f082-4c0f-8173-5261e8d24964) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:36:10 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (abb0c222-81ba-470f-82c8-ace0482c0fe5) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:36:10 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (abb0c222-81ba-470f-82c8-ace0482c0fe5) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:36:10 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (3998f8e0-3ba1-4802-8f74-76b867b943f7) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:36:10 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Updating volume a4d96ab8-2072-4eff-b803-53b4f27d056f with {'scheduled_at': datetime.datetime(2023, 4, 17, 15, 36, 10, 771904), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 15:36:10 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (715f24c3-60e8-426b-9530-51df436971f2) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 15:36:10 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 15:36:10 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:31:28Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesActionsTest-Volume-2081546358',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=44f909cf-7774-41f6-9f86-805456609ff8,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='a8b97ff6ed6047f0a33e74684a1c162b',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:33:49Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:33:49Z,use_quota=True,user_id='c993ceeb4a5c49eea14057e2125518ad',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 15:36:10 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:31:28Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesActionsTest-Volume-2081546358',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=44f909cf-7774-41f6-9f86-805456609ff8,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='a8b97ff6ed6047f0a33e74684a1c162b',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:33:49Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:33:49Z,use_quota=True,user_id='c993ceeb4a5c49eea14057e2125518ad',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(44f909cf-7774-41f6-9f86-805456609ff8),volume_id=44f909cf-7774-41f6-9f86-805456609ff8,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': '44f909cf-7774-41f6-9f86-805456609ff8', 'volume_name': 'volume-44f909cf-7774-41f6-9f86-805456609ff8', 'volume_size': 1}} Apr 17 15:36:10 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': '44f909cf-7774-41f6-9f86-805456609ff8', 'volume': {'id': '44f909cf-7774-41f6-9f86-805456609ff8', '_name_id': None, 'ec2_id': None, 'user_id': 'c993ceeb4a5c49eea14057e2125518ad', 'project_id': 'a8b97ff6ed6047f0a33e74684a1c162b', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-VolumesActionsTest-Volume-2081546358', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'admin_metadata': {}, 'glance_metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T15:31:28.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-44f909cf-7774-41f6-9f86-805456609ff8', 'name_id': '44f909cf-7774-41f6-9f86-805456609ff8', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumesActionsTest-Volume-2081546358', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': 'a8b97ff6ed6047f0a33e74684a1c162b', 'qos_specs': None, 'reservations': ['85f3837d-d106-4fab-b4df-d6deba5e2524', 'a6a77e64-4343-4b2d-a087-e1d5e603fbe7', '46947bf4-8977-46e0-91f8-7f7221eb5f15', '82ecd437-b8ec-41ab-ad06-ff748c2a9c71'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'c993ceeb4a5c49eea14057e2125518ad', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumesActionsTest-Volume-2081546358', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': 'a8b97ff6ed6047f0a33e74684a1c162b', 'qos_specs': None, 'reservations': ['85f3837d-d106-4fab-b4df-d6deba5e2524', 'a6a77e64-4343-4b2d-a087-e1d5e603fbe7', '46947bf4-8977-46e0-91f8-7f7221eb5f15', '82ecd437-b8ec-41ab-ad06-ff748c2a9c71'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'c993ceeb4a5c49eea14057e2125518ad', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': 'c993ceeb4a5c49eea14057e2125518ad', 'metadata': {}, 'qos_specs': None, 'retry': {'num_attempts': 2, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']}}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(44f909cf-7774-41f6-9f86-805456609ff8),volume_id=44f909cf-7774-41f6-9f86-805456609ff8,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:31:28Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesActionsTest-Volume-2081546358',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=44f909cf-7774-41f6-9f86-805456609ff8,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='a8b97ff6ed6047f0a33e74684a1c162b',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:33:49Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:33:49Z,use_quota=True,user_id='c993ceeb4a5c49eea14057e2125518ad',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 15:36:10 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:31:28Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesActionsTest-Volume-2081546358',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=44f909cf-7774-41f6-9f86-805456609ff8,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='a8b97ff6ed6047f0a33e74684a1c162b',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:33:49Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:33:49Z,use_quota=True,user_id='c993ceeb4a5c49eea14057e2125518ad',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:31:28Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesActionsTest-Volume-2081546358',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=44f909cf-7774-41f6-9f86-805456609ff8,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='a8b97ff6ed6047f0a33e74684a1c162b',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:33:49Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:33:49Z,use_quota=True,user_id='c993ceeb4a5c49eea14057e2125518ad',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 15:36:10 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 15:36:10 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 15:36:10 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:36:10 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 15:36:10 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 15:36:10 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 15:36:10 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 15:36:10 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 15:36:10 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 15:36:10 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 15:36:10 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 15:36:10 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 15:36:10 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 15:36:10 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 15:36:10 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 15:36:10 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 15:36:10 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 15:36:10 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 15:36:10 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 15:36:10 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 15:36:10 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (715f24c3-60e8-426b-9530-51df436971f2) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:36:10 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (715f24c3-60e8-426b-9530-51df436971f2) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:36:10 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (d56250f2-e591-4e75-bd2e-b9424716da79) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:36:10 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (d56250f2-e591-4e75-bd2e-b9424716da79) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:36:10 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (1f705a08-fad7-4ef4-9ad1-6ba66457dd2e) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:36:10 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (1f705a08-fad7-4ef4-9ad1-6ba66457dd2e) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:36:10 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (e4df08b7-3778-4055-9368-a4b9d604e358) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:36:10 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Updating volume 44f909cf-7774-41f6-9f86-805456609ff8 with {'scheduled_at': datetime.datetime(2023, 4, 17, 15, 36, 10, 814994), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 15:36:10 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Volume a4d96ab8-2072-4eff-b803-53b4f27d056f: re-scheduling SchedulerAPI.create_volume attempt 3 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 15:36:10 user cinder-volume[84696]: DEBUG cinder.coordination [None req-3175029a-5606-46a3-86f4-a16d583ad0ce tempest-TestStampPattern-173447269 None] Lock "cinder-a9c177a1-61f0-4201-bb8d-7f18c9af396c-delete_volume" released by "delete_volume" :: held 142.089s {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:207}} Apr 17 15:36:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-3175029a-5606-46a3-86f4-a16d583ad0ce tempest-TestStampPattern-173447269 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 15:36:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 15:36:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 15:36:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 15:36:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 15:36:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 15:36:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 15:36:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 15:36:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 203, in wrapper Apr 17 15:36:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server skip_clean = func(self, context, volume, *args, **kwargs) Apr 17 15:36:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 15:36:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 15:36:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/coordination.py", line 200, in _synchronized Apr 17 15:36:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return f(*a, **k) Apr 17 15:36:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 15:36:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 15:36:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 15:36:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 15:36:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 1035, in delete_volume Apr 17 15:36:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 15:36:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 15:36:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 15:36:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 15:36:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 15:36:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 1004, in delete_volume Apr 17 15:36:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.driver.remove_export(context, volume) Apr 17 15:36:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 720, in remove_export Apr 17 15:36:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self._remove_target_volume(volume) Apr 17 15:36:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 918, in _remove_target_volume Apr 17 15:36:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.detach_target_vol(target_name, jcom.vname(volume.id)) Apr 17 15:36:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 612, in detach_target_vol Apr 17 15:36:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if resp["code"] in (200, 201, 204): Apr 17 15:36:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 15:36:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 15:36:10 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Volume 44f909cf-7774-41f6-9f86-805456609ff8: re-scheduling SchedulerAPI.create_volume attempt 2 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 15:36:10 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Volume a4d96ab8-2072-4eff-b803-53b4f27d056f: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 15:36:10 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] deleating volume v_a4d96ab8-2072-4eff-b803-53b4f27d056f {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 15:36:10 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] get snapshots for volume v_a4d96ab8-2072-4eff-b803-53b4f27d056f {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 15:36:10 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_a4d96ab8-2072-4eff-b803-53b4f27d056f/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:36:10 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_a4d96ab8-2072-4eff-b803-53b4f27d056f/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:36:10 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Volume 44f909cf-7774-41f6-9f86-805456609ff8: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 15:36:10 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] deleating volume v_44f909cf-7774-41f6-9f86-805456609ff8 {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 15:36:10 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] get snapshots for volume v_44f909cf-7774-41f6-9f86-805456609ff8 {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 15:36:10 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_44f909cf-7774-41f6-9f86-805456609ff8/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:36:10 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_44f909cf-7774-41f6-9f86-805456609ff8/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:36:11 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Volume reschedule parameters: True retry: {'num_attempts': 3, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 15:36:11 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Flow 'volume_create_manager' (d15433fb-3b6e-47b8-8df1-aa4f8b82cd32) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:36:11 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (a2daede2-93f4-4c4f-9805-dbb1f304ff7f) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:36:11 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (a2daede2-93f4-4c4f-9805-dbb1f304ff7f) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:31:28Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesActionsTest-Volume-2081546358',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=44f909cf-7774-41f6-9f86-805456609ff8,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='a8b97ff6ed6047f0a33e74684a1c162b',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:36:11Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:36:11Z,use_quota=True,user_id='c993ceeb4a5c49eea14057e2125518ad',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:36:11 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (ad21cf54-42e6-4d2d-bbc8-2aa4dc0604d7) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:36:11 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (ad21cf54-42e6-4d2d-bbc8-2aa4dc0604d7) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:36:11 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (9ffe98ef-6fc5-4654-9da2-bf6ae717aa79) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:36:11 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (9ffe98ef-6fc5-4654-9da2-bf6ae717aa79) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': '44f909cf-7774-41f6-9f86-805456609ff8', 'volume_name': 'volume-44f909cf-7774-41f6-9f86-805456609ff8', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:36:11 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (6b4d6194-64a0-455e-8780-79eb13cfd925) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:36:11 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (6b4d6194-64a0-455e-8780-79eb13cfd925) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:36:11 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (b391e404-ecbf-4536-85c7-738e3109e043) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:36:11 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Volume 44f909cf-7774-41f6-9f86-805456609ff8: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-44f909cf-7774-41f6-9f86-805456609ff8', 'volume_size': 1} Apr 17 15:36:11 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] creating volume v_44f909cf-7774-41f6-9f86-805456609ff8. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 15:36:11 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] create volume {'name': 'v_44f909cf-7774-41f6-9f86-805456609ff8', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 15:36:11 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:36:11 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:36:13 user cinder-volume[84696]: DEBUG cinder.utils [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.627(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:36:13 user cinder-volume[84696]: DEBUG cinder.utils [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:36:13 user cinder-volume[84696]: DEBUG cinder.utils [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.545(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:36:13 user cinder-volume[84696]: DEBUG cinder.utils [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_8cc35ab1-363f-4ad6-9a37-239d2b1ea996/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:36:13 user cinder-volume[84696]: DEBUG cinder.utils [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.856(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:36:13 user cinder-volume[84696]: DEBUG cinder.utils [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_44f909cf-7774-41f6-9f86-805456609ff8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:36:13 user cinder-volume[84696]: DEBUG cinder.utils [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.379(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:36:13 user cinder-volume[84696]: DEBUG cinder.utils [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:36:13 user cinder-volume[84696]: DEBUG cinder.utils [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.959(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:36:13 user cinder-volume[84696]: DEBUG cinder.utils [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_62709a81-268b-4227-ba4a-070034f36414/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:36:13 user cinder-volume[84696]: DEBUG cinder.utils [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.871(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:36:13 user cinder-volume[84696]: DEBUG cinder.utils [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_a4d96ab8-2072-4eff-b803-53b4f27d056f/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:36:16 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 40.056(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:36:16 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:36:16 user cinder-volume[84696]: DEBUG cinder.utils [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.938(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:36:16 user cinder-volume[84696]: DEBUG cinder.utils [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_a4d96ab8-2072-4eff-b803-53b4f27d056f/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:36:16 user cinder-volume[84696]: DEBUG cinder.utils [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.700(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:36:16 user cinder-volume[84696]: DEBUG cinder.utils [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:36:16 user cinder-volume[84696]: DEBUG cinder.utils [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.928(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:36:16 user cinder-volume[84696]: DEBUG cinder.utils [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_44f909cf-7774-41f6-9f86-805456609ff8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:36:19 user cinder-volume[84696]: DEBUG cinder.utils [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 138.934(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:36:19 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_0fed6b21-0520-4554-bee4-9540a03f9af0/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:36:19 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (975708e3-2525-423b-9a96-cdb2cd196821) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 15:36:19 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (e4c7a40d-0db8-4b52-973c-6b0be4467495) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:36:19 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (e4c7a40d-0db8-4b52-973c-6b0be4467495) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:36:19 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Flow 'volume_create_manager' (9a8b737c-677b-49d2-acfc-3ad07a3df649) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 15:36:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 15:36:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 15:36:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 15:36:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 15:36:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 15:36:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 15:36:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 15:36:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 15:36:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 15:36:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 15:36:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 15:36:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 15:36:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 15:36:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 15:36:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 15:36:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 15:36:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 15:36:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 15:36:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 15:36:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 15:36:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 15:36:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 15:36:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 15:36:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 15:36:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:36:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 15:36:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1237, in execute Apr 17 15:36:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_from_image(context, Apr 17 15:36:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/utils.py", line 670, in _wrapper Apr 17 15:36:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return r(f, *args, **kwargs) Apr 17 15:36:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/tenacity/__init__.py", line 423, in __call__ Apr 17 15:36:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server do = self.iter(retry_state=retry_state) Apr 17 15:36:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/tenacity/__init__.py", line 360, in iter Apr 17 15:36:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return fut.result() Apr 17 15:36:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/lib/python3.10/concurrent/futures/_base.py", line 451, in result Apr 17 15:36:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self.__get_result() Apr 17 15:36:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result Apr 17 15:36:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self._exception Apr 17 15:36:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/tenacity/__init__.py", line 426, in __call__ Apr 17 15:36:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = fn(*args, **kwargs) Apr 17 15:36:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1125, in _create_from_image Apr 17 15:36:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_from_image_cache_or_download( Apr 17 15:36:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 944, in _create_from_image_cache_or_download Apr 17 15:36:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update, cloned = self._create_from_image_cache( Apr 17 15:36:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 843, in _create_from_image_cache Apr 17 15:36:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_from_source_volume( Apr 17 15:36:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 659, in _create_from_source_volume Apr 17 15:36:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self.driver.create_cloned_volume(volume, srcvol_ref) Apr 17 15:36:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 525, in create_cloned_volume Apr 17 15:36:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self._clone_object(vname, cvname) Apr 17 15:36:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 461, in _clone_object Apr 17 15:36:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_snapshot(oname, coname) Apr 17 15:36:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 639, in create_snapshot Apr 17 15:36:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201, 204): Apr 17 15:36:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 15:36:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 15:36:19 user cinder-volume[84696]: DEBUG cinder.utils [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 138.638(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:36:19 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:36:19 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (98df3471-98bf-4d2e-ad67-b0292122b87f) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 15:36:19 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 15:36:19 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:31:42Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesCloneTest-Volume-297264340',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=0fed6b21-0520-4554-bee4-9540a03f9af0,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='82fce1724afa40dd914e0791c3ee69a3',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:34:01Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:34:01Z,use_quota=True,user_id='773ce6c2a0f844edaf666e190504e258',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 15:36:19 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:31:42Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesCloneTest-Volume-297264340',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=0fed6b21-0520-4554-bee4-9540a03f9af0,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='82fce1724afa40dd914e0791c3ee69a3',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:34:01Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:34:01Z,use_quota=True,user_id='773ce6c2a0f844edaf666e190504e258',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=b3f0dec1-7758-44a8-82e0-76ada8837428,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(0fed6b21-0520-4554-bee4-9540a03f9af0),volume_id=0fed6b21-0520-4554-bee4-9540a03f9af0,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'image', 'volume_id': '0fed6b21-0520-4554-bee4-9540a03f9af0', 'volume_name': 'volume-0fed6b21-0520-4554-bee4-9540a03f9af0', 'volume_size': 1, 'image_id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'image_location': (None, None), 'image_meta': {'name': 'cirros-0.5.2-x86_64-disk', 'disk_format': 'qcow2', 'container_format': 'bare', 'visibility': 'public', 'size': 16300544, 'virtual_size': 117440512, 'status': 'active', 'checksum': 'b874c39491a2377b8490f5f1e89761a4', 'protected': False, 'min_ram': 0, 'min_disk': 0, 'owner': 'a43c211b4ed542538799349489b1d884', 'os_hidden': False, 'os_hash_algo': 'sha512', 'os_hash_value': '6b813aa46bb90b4da216a4d19376593fa3f4fc7e617f03a92b7fe11e9a3981cbe8f0959dbebe36225e5f53dc4492341a4863cac4ed1ee0909f3fc78ef9c3e869', 'id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'created_at': datetime.datetime(2023, 4, 17, 15, 19, 53, tzinfo=datetime.timezone.utc), 'updated_at': datetime.datetime(2023, 4, 17, 15, 19, 55, tzinfo=datetime.timezone.utc), 'tags': [], 'file': '/v2/images/b3f0dec1-7758-44a8-82e0-76ada8837428/file', 'properties': {'hw_rng_model': 'virtio', 'owner_specified.openstack.md5': '', 'owner_specified.openstack.object': 'images/cirros-0.5.2-x86_64-disk', 'owner_specified.openstack.sha256': ''}}, 'image_service': }} Apr 17 15:36:19 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'snapshot_id': None, 'source_volid': None, 'volume_id': '0fed6b21-0520-4554-bee4-9540a03f9af0', 'volume': {'id': '0fed6b21-0520-4554-bee4-9540a03f9af0', '_name_id': None, 'ec2_id': None, 'user_id': '773ce6c2a0f844edaf666e190504e258', 'project_id': '82fce1724afa40dd914e0791c3ee69a3', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-VolumesCloneTest-Volume-297264340', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'admin_metadata': {}, 'glance_metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T15:31:41.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-0fed6b21-0520-4554-bee4-9540a03f9af0', 'name_id': '0fed6b21-0520-4554-bee4-9540a03f9af0', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumesCloneTest-Volume-297264340', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '82fce1724afa40dd914e0791c3ee69a3', 'qos_specs': None, 'reservations': ['1379a6d8-9976-4d94-a18a-ef0001e5de6b', 'ebe18ce3-cd83-4706-9c9a-b20a8ec73b8f', '3321e2fc-3bcb-4c90-bfc9-3a5a122ad7a3', '0805cd4d-24d4-47d8-b230-5b3c0a1d66aa'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': '773ce6c2a0f844edaf666e190504e258', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumesCloneTest-Volume-297264340', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '82fce1724afa40dd914e0791c3ee69a3', 'qos_specs': None, 'reservations': ['1379a6d8-9976-4d94-a18a-ef0001e5de6b', 'ebe18ce3-cd83-4706-9c9a-b20a8ec73b8f', '3321e2fc-3bcb-4c90-bfc9-3a5a122ad7a3', '0805cd4d-24d4-47d8-b230-5b3c0a1d66aa'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': '773ce6c2a0f844edaf666e190504e258', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': '773ce6c2a0f844edaf666e190504e258', 'metadata': {}, 'qos_specs': None, 'retry': {'num_attempts': 2, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']}}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=b3f0dec1-7758-44a8-82e0-76ada8837428,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(0fed6b21-0520-4554-bee4-9540a03f9af0),volume_id=0fed6b21-0520-4554-bee4-9540a03f9af0,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:31:42Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesCloneTest-Volume-297264340',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=0fed6b21-0520-4554-bee4-9540a03f9af0,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='82fce1724afa40dd914e0791c3ee69a3',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:34:01Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:34:01Z,use_quota=True,user_id='773ce6c2a0f844edaf666e190504e258',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 15:36:19 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:31:42Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesCloneTest-Volume-297264340',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=0fed6b21-0520-4554-bee4-9540a03f9af0,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='82fce1724afa40dd914e0791c3ee69a3',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:34:01Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:34:01Z,use_quota=True,user_id='773ce6c2a0f844edaf666e190504e258',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:31:42Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesCloneTest-Volume-297264340',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=0fed6b21-0520-4554-bee4-9540a03f9af0,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='82fce1724afa40dd914e0791c3ee69a3',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:34:01Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:34:01Z,use_quota=True,user_id='773ce6c2a0f844edaf666e190504e258',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 15:36:19 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 15:36:19 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 15:36:19 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:36:19 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 15:36:19 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1237, in execute Apr 17 15:36:19 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_from_image(context, Apr 17 15:36:19 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/utils.py", line 670, in _wrapper Apr 17 15:36:19 user cinder-volume[84696]: ERROR cinder.volume.manager return r(f, *args, **kwargs) Apr 17 15:36:19 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/tenacity/__init__.py", line 423, in __call__ Apr 17 15:36:19 user cinder-volume[84696]: ERROR cinder.volume.manager do = self.iter(retry_state=retry_state) Apr 17 15:36:19 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/tenacity/__init__.py", line 360, in iter Apr 17 15:36:19 user cinder-volume[84696]: ERROR cinder.volume.manager return fut.result() Apr 17 15:36:19 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/lib/python3.10/concurrent/futures/_base.py", line 451, in result Apr 17 15:36:19 user cinder-volume[84696]: ERROR cinder.volume.manager return self.__get_result() Apr 17 15:36:19 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result Apr 17 15:36:19 user cinder-volume[84696]: ERROR cinder.volume.manager raise self._exception Apr 17 15:36:19 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/tenacity/__init__.py", line 426, in __call__ Apr 17 15:36:19 user cinder-volume[84696]: ERROR cinder.volume.manager result = fn(*args, **kwargs) Apr 17 15:36:19 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1125, in _create_from_image Apr 17 15:36:19 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_from_image_cache_or_download( Apr 17 15:36:19 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 944, in _create_from_image_cache_or_download Apr 17 15:36:19 user cinder-volume[84696]: ERROR cinder.volume.manager model_update, cloned = self._create_from_image_cache( Apr 17 15:36:19 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 843, in _create_from_image_cache Apr 17 15:36:19 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_from_source_volume( Apr 17 15:36:19 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 659, in _create_from_source_volume Apr 17 15:36:19 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self.driver.create_cloned_volume(volume, srcvol_ref) Apr 17 15:36:19 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 525, in create_cloned_volume Apr 17 15:36:19 user cinder-volume[84696]: ERROR cinder.volume.manager self._clone_object(vname, cvname) Apr 17 15:36:19 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 461, in _clone_object Apr 17 15:36:19 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_snapshot(oname, coname) Apr 17 15:36:19 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 639, in create_snapshot Apr 17 15:36:19 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201, 204): Apr 17 15:36:19 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 15:36:19 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 15:36:19 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (98df3471-98bf-4d2e-ad67-b0292122b87f) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:36:19 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (98df3471-98bf-4d2e-ad67-b0292122b87f) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:36:19 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (41eef73b-9547-438d-8266-ee4f295287e9) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:36:19 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (41eef73b-9547-438d-8266-ee4f295287e9) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:36:19 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (46b5f298-21b2-4289-8c51-0a8dca8c9bbf) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:36:19 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (46b5f298-21b2-4289-8c51-0a8dca8c9bbf) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:36:19 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (3ce8ae22-70a5-4d60-94e6-f9407d0cd7fd) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:36:19 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Updating volume 0fed6b21-0520-4554-bee4-9540a03f9af0 with {'scheduled_at': datetime.datetime(2023, 4, 17, 15, 36, 19, 931438), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 15:36:19 user cinder-volume[84696]: DEBUG cinder.utils [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.584(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:36:19 user cinder-volume[84696]: DEBUG cinder.utils [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_5d9c86cd-3e22-4926-bfc9-e5a37f6ae76c/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:36:19 user cinder-volume[84696]: DEBUG cinder.utils [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.318(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:36:19 user cinder-volume[84696]: DEBUG cinder.utils [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:36:19 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Volume 0fed6b21-0520-4554-bee4-9540a03f9af0: re-scheduling SchedulerAPI.create_volume attempt 2 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 15:36:19 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Volume 0fed6b21-0520-4554-bee4-9540a03f9af0: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 15:36:19 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] deleating volume v_0fed6b21-0520-4554-bee4-9540a03f9af0 {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 15:36:19 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] get snapshots for volume v_0fed6b21-0520-4554-bee4-9540a03f9af0 {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 15:36:19 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_0fed6b21-0520-4554-bee4-9540a03f9af0/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:36:19 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_0fed6b21-0520-4554-bee4-9540a03f9af0/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:36:20 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Volume reschedule parameters: True retry: {'num_attempts': 3, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 15:36:20 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Flow 'volume_create_manager' (cce6b286-e2af-4a14-85bc-d1251e04b47e) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:36:20 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (73e195f3-6579-445a-9aeb-28f7098f6f4d) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:36:20 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (73e195f3-6579-445a-9aeb-28f7098f6f4d) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:31:42Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesCloneTest-Volume-297264340',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=0fed6b21-0520-4554-bee4-9540a03f9af0,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='82fce1724afa40dd914e0791c3ee69a3',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:36:20Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:36:20Z,use_quota=True,user_id='773ce6c2a0f844edaf666e190504e258',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:36:20 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (08b4e6dc-1ce7-49f6-b5aa-72522a91a536) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:36:20 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (08b4e6dc-1ce7-49f6-b5aa-72522a91a536) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:36:20 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (04dcc085-dbfb-4dc0-83a7-4a5794cfbf96) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:36:20 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (04dcc085-dbfb-4dc0-83a7-4a5794cfbf96) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'image', 'volume_id': '0fed6b21-0520-4554-bee4-9540a03f9af0', 'volume_name': 'volume-0fed6b21-0520-4554-bee4-9540a03f9af0', 'volume_size': 1, 'image_id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'image_location': (None, None), 'image_meta': {'name': 'cirros-0.5.2-x86_64-disk', 'disk_format': 'qcow2', 'container_format': 'bare', 'visibility': 'public', 'size': 16300544, 'virtual_size': 117440512, 'status': 'active', 'checksum': 'b874c39491a2377b8490f5f1e89761a4', 'protected': False, 'min_ram': 0, 'min_disk': 0, 'owner': 'a43c211b4ed542538799349489b1d884', 'os_hidden': False, 'os_hash_algo': 'sha512', 'os_hash_value': '6b813aa46bb90b4da216a4d19376593fa3f4fc7e617f03a92b7fe11e9a3981cbe8f0959dbebe36225e5f53dc4492341a4863cac4ed1ee0909f3fc78ef9c3e869', 'id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'created_at': datetime.datetime(2023, 4, 17, 15, 19, 53, tzinfo=datetime.timezone.utc), 'updated_at': datetime.datetime(2023, 4, 17, 15, 19, 55, tzinfo=datetime.timezone.utc), 'tags': [], 'file': '/v2/images/b3f0dec1-7758-44a8-82e0-76ada8837428/file', 'properties': {'hw_rng_model': 'virtio', 'owner_specified.openstack.md5': '', 'owner_specified.openstack.object': 'images/cirros-0.5.2-x86_64-disk', 'owner_specified.openstack.sha256': ''}}, 'image_service': }' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:36:20 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (545df23a-d06f-4506-b87f-847ec7576a67) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:36:20 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (545df23a-d06f-4506-b87f-847ec7576a67) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:36:20 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (d344f7f8-223e-4527-860d-34854bf63518) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:36:20 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Volume 0fed6b21-0520-4554-bee4-9540a03f9af0: being created as image with specification: {'status': 'creating', 'volume_name': 'volume-0fed6b21-0520-4554-bee4-9540a03f9af0', 'volume_size': 1, 'image_id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'image_location': (None, None), 'image_meta': {'name': 'cirros-0.5.2-x86_64-disk', 'disk_format': 'qcow2', 'container_format': 'bare', 'visibility': 'public', 'size': 16300544, 'virtual_size': 117440512, 'status': 'active', 'checksum': 'b874c39491a2377b8490f5f1e89761a4', 'protected': False, 'min_ram': 0, 'min_disk': 0, 'owner': 'a43c211b4ed542538799349489b1d884', 'os_hidden': False, 'os_hash_algo': 'sha512', 'os_hash_value': '6b813aa46bb90b4da216a4d19376593fa3f4fc7e617f03a92b7fe11e9a3981cbe8f0959dbebe36225e5f53dc4492341a4863cac4ed1ee0909f3fc78ef9c3e869', 'id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'created_at': datetime.datetime(2023, 4, 17, 15, 19, 53, tzinfo=datetime.timezone.utc), 'updated_at': datetime.datetime(2023, 4, 17, 15, 19, 55, tzinfo=datetime.timezone.utc), 'tags': [], 'file': '/v2/images/b3f0dec1-7758-44a8-82e0-76ada8837428/file', 'properties': {'hw_rng_model': 'virtio', 'owner_specified.openstack.md5': '', 'owner_specified.openstack.object': 'images/cirros-0.5.2-x86_64-disk', 'owner_specified.openstack.sha256': ''}}, 'image_service': } Apr 17 15:36:20 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Cloning 0fed6b21-0520-4554-bee4-9540a03f9af0 from image b3f0dec1-7758-44a8-82e0-76ada8837428 at location (None, None). {{(pid=84696) _create_from_image /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1071}} Apr 17 15:36:20 user cinder-volume[84696]: DEBUG cinder.coordination [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Acquiring lock "cinder-b3f0dec1-7758-44a8-82e0-76ada8837428" by "_prepare_image_cache_entry" {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:192}} Apr 17 15:36:20 user cinder-volume[84696]: DEBUG cinder.coordination [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Lock "cinder-b3f0dec1-7758-44a8-82e0-76ada8837428" acquired by "_prepare_image_cache_entry" :: waited 0.010s {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:196}} Apr 17 15:36:20 user cinder-volume[84696]: DEBUG cinder.image.cache [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Found image-volume cache entry: {'id': 1, 'image_id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'volume_id': '006f20ae-5306-44fe-afdf-e433f79284e8', 'host': 'user@joviandss-2#joviandss-2', 'size': 1, 'image_updated_at': datetime.datetime(2023, 4, 17, 15, 19, 55), 'last_used': datetime.datetime(2023, 4, 17, 15, 36, 20, 261494)}. {{(pid=84696) get_entry /opt/stack/cinder/cinder/image/cache.py:76}} Apr 17 15:36:20 user cinder-volume[84696]: DEBUG cinder.image.cache [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Image-volume cache entry image_update_at = 2023-04-17 15:19:55+00:00, requested image updated_at = 2023-04-17 15:19:55+00:00. {{(pid=84696) _should_update_entry /opt/stack/cinder/cinder/image/cache.py:259}} Apr 17 15:36:20 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Found cache entry for image = b3f0dec1-7758-44a8-82e0-76ada8837428 on host user@joviandss-2#joviandss-2. {{(pid=84696) _prepare_image_cache_entry /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:886}} Apr 17 15:36:20 user cinder-volume[84696]: DEBUG cinder.coordination [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Lock "cinder-b3f0dec1-7758-44a8-82e0-76ada8837428" released by "_prepare_image_cache_entry" :: held 0.017s {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:207}} Apr 17 15:36:20 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Attempting to retrieve cache entry for image = b3f0dec1-7758-44a8-82e0-76ada8837428 on host user@joviandss-2#joviandss-2. {{(pid=84696) _create_from_image_cache /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:827}} Apr 17 15:36:20 user cinder-volume[84696]: DEBUG cinder.image.cache [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Found image-volume cache entry: {'id': 1, 'image_id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'volume_id': '006f20ae-5306-44fe-afdf-e433f79284e8', 'host': 'user@joviandss-2#joviandss-2', 'size': 1, 'image_updated_at': datetime.datetime(2023, 4, 17, 15, 19, 55), 'last_used': datetime.datetime(2023, 4, 17, 15, 36, 20, 276960)}. {{(pid=84696) get_entry /opt/stack/cinder/cinder/image/cache.py:76}} Apr 17 15:36:20 user cinder-volume[84696]: DEBUG cinder.image.cache [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Image-volume cache entry image_update_at = 2023-04-17 15:19:55+00:00, requested image updated_at = 2023-04-17 15:19:55+00:00. {{(pid=84696) _should_update_entry /opt/stack/cinder/cinder/image/cache.py:259}} Apr 17 15:36:20 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Creating from source image-volume 006f20ae-5306-44fe-afdf-e433f79284e8 {{(pid=84696) _create_from_image_cache /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:841}} Apr 17 15:36:20 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] cloned volume 006f20ae-5306-44fe-afdf-e433f79284e8 to 0fed6b21-0520-4554-bee4-9540a03f9af0 {{(pid=84696) create_cloned_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:521}} Apr 17 15:36:20 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] cloning v_006f20ae-5306-44fe-afdf-e433f79284e8 to v_0fed6b21-0520-4554-bee4-9540a03f9af0 {{(pid=84696) _clone_object /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:456}} Apr 17 15:36:20 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] create snapshot v_0fed6b21-0520-4554-bee4-9540a03f9af0 {{(pid=84696) create_snapshot /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:635}} Apr 17 15:36:20 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:36:20 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:36:22 user cinder-volume[84696]: DEBUG cinder.utils [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.897(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:36:22 user cinder-volume[84696]: DEBUG cinder.utils [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_17f8d4f7-2dcd-4041-8d0b-5c8afebb75bf/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:36:22 user cinder-volume[84696]: DEBUG cinder.utils [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.984(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:36:22 user cinder-volume[84696]: DEBUG cinder.utils [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_0fed6b21-0520-4554-bee4-9540a03f9af0/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:36:22 user cinder-volume[84696]: DEBUG cinder.utils [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.651(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:36:22 user cinder-volume[84696]: DEBUG cinder.utils [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:36:22 user cinder-volume[84696]: DEBUG cinder.utils [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.076(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:36:22 user cinder-volume[84696]: DEBUG cinder.utils [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_44f909cf-7774-41f6-9f86-805456609ff8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:36:22 user cinder-volume[84696]: DEBUG cinder.utils [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.090(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:36:22 user cinder-volume[84696]: DEBUG cinder.utils [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_a4d96ab8-2072-4eff-b803-53b4f27d056f/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:36:22 user cinder-volume[84696]: DEBUG cinder.utils [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 11.852(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:36:22 user cinder-volume[84696]: DEBUG cinder.utils [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:36:24 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:INSERT=6,cinder:SELECT=26,cinder:UPDATE=11 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:36:24 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=31,cinder:UPDATE=9,cinder:INSERT=2,cinder:DELETE=1 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:36:26 user cinder-volume[84696]: DEBUG cinder.utils [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.052(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:36:26 user cinder-volume[84696]: DEBUG cinder.utils [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_0fed6b21-0520-4554-bee4-9540a03f9af0/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:36:26 user cinder-volume[84696]: DEBUG cinder.utils [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.719(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:36:26 user cinder-volume[84696]: DEBUG cinder.utils [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:36:32 user cinder-volume[84696]: DEBUG cinder.utils [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.196(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:36:32 user cinder-volume[84696]: DEBUG cinder.utils [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_0fed6b21-0520-4554-bee4-9540a03f9af0/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:36:32 user cinder-volume[84696]: DEBUG cinder.utils [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.978(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:36:32 user cinder-volume[84696]: DEBUG cinder.utils [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_8cc35ab1-363f-4ad6-9a37-239d2b1ea996/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:36:32 user cinder-volume[84696]: DEBUG cinder.utils [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.062(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:36:32 user cinder-volume[84696]: DEBUG cinder.utils [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:36:32 user cinder-volume[84696]: DEBUG cinder.utils [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.290(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:36:32 user cinder-volume[84696]: DEBUG cinder.utils [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_44f909cf-7774-41f6-9f86-805456609ff8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:36:32 user cinder-volume[84696]: DEBUG cinder.utils [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.303(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:36:32 user cinder-volume[84696]: DEBUG cinder.utils [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_a4d96ab8-2072-4eff-b803-53b4f27d056f/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:36:32 user cinder-volume[84696]: DEBUG cinder.utils [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 11.868(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:36:32 user cinder-volume[84696]: DEBUG cinder.utils [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:36:32 user cinder-volume[84696]: DEBUG cinder.utils [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.814(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:36:32 user cinder-volume[84696]: DEBUG cinder.utils [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:36:33 user cinder-volume[84696]: DEBUG cinder.coordination [None req-dbedb542-4536-4c1d-b79c-b9c0a6c64820 tempest-VolumesAdminNegativeTest-1285212149 None] Acquiring lock "cinder-215a6a5b-25ca-48fa-a8e3-72416d7a39e2-delete_volume" by "delete_volume" {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:192}} Apr 17 15:36:33 user cinder-volume[84696]: DEBUG cinder.coordination [None req-dbedb542-4536-4c1d-b79c-b9c0a6c64820 tempest-VolumesAdminNegativeTest-1285212149 None] Lock "cinder-215a6a5b-25ca-48fa-a8e3-72416d7a39e2-delete_volume" acquired by "delete_volume" :: waited 0.009s {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:196}} Apr 17 15:36:33 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-dbedb542-4536-4c1d-b79c-b9c0a6c64820 tempest-VolumesAdminNegativeTest-1285212149 None] remove_export for volume: 215a6a5b-25ca-48fa-a8e3-72416d7a39e2. {{(pid=84696) remove_export /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:718}} Apr 17 15:36:33 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-dbedb542-4536-4c1d-b79c-b9c0a6c64820 tempest-VolumesAdminNegativeTest-1285212149 None] remove export {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:912}} Apr 17 15:36:33 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-dbedb542-4536-4c1d-b79c-b9c0a6c64820 tempest-VolumesAdminNegativeTest-1285212149 None] detach volume:Volume(_name_id=None,admin_metadata={},attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:26:08Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesAdminNegativeTest-volume-850487146',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=215a6a5b-25ca-48fa-a8e3-72416d7a39e2,launched_at=2023-04-17T15:26:10Z,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='b334c5bc33f143c484518d9bd0e750dd',provider_auth='CHAP wIJnMDkU zdn9a38rfO7goh',provider_geometry=None,provider_id=None,provider_location='172.16.0.220:3260,1 iqn.2010-10.org.openstack:215a6a5b-25ca-48fa-a8e3-72416d7a39e2 0',replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:26:08Z,service_uuid='e628352e-c4cf-40eb-8389-cce40663026a',shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='deleting',terminated_at=2023-04-17T15:36:33Z,updated_at=2023-04-17T15:36:33Z,use_quota=True,user_id='3dfadfd30c394b46abe75ec930007850',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c) from target:iqn.2010-10.org.openstack:215a6a5b-25ca-48fa-a8e3-72416d7a39e2. {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:913}} Apr 17 15:36:33 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-dbedb542-4536-4c1d-b79c-b9c0a6c64820 tempest-VolumesAdminNegativeTest-1285212149 None] detach volume v_215a6a5b-25ca-48fa-a8e3-72416d7a39e2 from target iqn.2010-10.org.openstack:215a6a5b-25ca-48fa-a8e3-72416d7a39e2 {{(pid=84696) detach_target_vol /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:606}} Apr 17 15:36:33 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-dbedb542-4536-4c1d-b79c-b9c0a6c64820 tempest-VolumesAdminNegativeTest-1285212149 None] Sending pool request DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:215a6a5b-25ca-48fa-a8e3-72416d7a39e2/luns/v_215a6a5b-25ca-48fa-a8e3-72416d7a39e2 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:36:33 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-dbedb542-4536-4c1d-b79c-b9c0a6c64820 tempest-VolumesAdminNegativeTest-1285212149 None] Sending DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:215a6a5b-25ca-48fa-a8e3-72416d7a39e2/luns/v_215a6a5b-25ca-48fa-a8e3-72416d7a39e2 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:36:35 user cinder-volume[84696]: DEBUG cinder.utils [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 138.013(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:36:35 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:36:35 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (b1232b89-540b-4b2f-8c7c-9165c17366a0) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 15:36:35 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 15:36:35 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:34:17Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TaggedBootDevicesTest-volume-1099737906',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=e80c78b2-20c9-4f9b-8898-6ab131d1eb06,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='75ab8873f7064d3fbe954746c071cb72',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:34:17Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:34:17Z,use_quota=True,user_id='39328f4ed6da4a85ac312bc5637b1500',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 15:36:35 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:34:17Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TaggedBootDevicesTest-volume-1099737906',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=e80c78b2-20c9-4f9b-8898-6ab131d1eb06,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='75ab8873f7064d3fbe954746c071cb72',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:34:17Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:34:17Z,use_quota=True,user_id='39328f4ed6da4a85ac312bc5637b1500',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=b3f0dec1-7758-44a8-82e0-76ada8837428,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(e80c78b2-20c9-4f9b-8898-6ab131d1eb06),volume_id=e80c78b2-20c9-4f9b-8898-6ab131d1eb06,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'image', 'volume_id': 'e80c78b2-20c9-4f9b-8898-6ab131d1eb06', 'volume_name': 'volume-e80c78b2-20c9-4f9b-8898-6ab131d1eb06', 'volume_size': 1, 'image_id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'image_location': (None, None), 'image_meta': {'name': 'cirros-0.5.2-x86_64-disk', 'disk_format': 'qcow2', 'container_format': 'bare', 'visibility': 'public', 'size': 16300544, 'virtual_size': 117440512, 'status': 'active', 'checksum': 'b874c39491a2377b8490f5f1e89761a4', 'protected': False, 'min_ram': 0, 'min_disk': 0, 'owner': 'a43c211b4ed542538799349489b1d884', 'os_hidden': False, 'os_hash_algo': 'sha512', 'os_hash_value': '6b813aa46bb90b4da216a4d19376593fa3f4fc7e617f03a92b7fe11e9a3981cbe8f0959dbebe36225e5f53dc4492341a4863cac4ed1ee0909f3fc78ef9c3e869', 'id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'created_at': datetime.datetime(2023, 4, 17, 15, 19, 53, tzinfo=datetime.timezone.utc), 'updated_at': datetime.datetime(2023, 4, 17, 15, 19, 55, tzinfo=datetime.timezone.utc), 'tags': [], 'file': '/v2/images/b3f0dec1-7758-44a8-82e0-76ada8837428/file', 'properties': {'hw_rng_model': 'virtio', 'owner_specified.openstack.md5': '', 'owner_specified.openstack.object': 'images/cirros-0.5.2-x86_64-disk', 'owner_specified.openstack.sha256': ''}}, 'image_service': }} Apr 17 15:36:35 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'retry': {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']}, 'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'snapshot_id': None, 'source_volid': None, 'volume_id': 'e80c78b2-20c9-4f9b-8898-6ab131d1eb06', 'volume': {'id': 'e80c78b2-20c9-4f9b-8898-6ab131d1eb06', '_name_id': None, 'ec2_id': None, 'user_id': '39328f4ed6da4a85ac312bc5637b1500', 'project_id': '75ab8873f7064d3fbe954746c071cb72', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-TaggedBootDevicesTest-volume-1099737906', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T15:34:16.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-e80c78b2-20c9-4f9b-8898-6ab131d1eb06', 'name_id': 'e80c78b2-20c9-4f9b-8898-6ab131d1eb06', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-TaggedBootDevicesTest-volume-1099737906', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '75ab8873f7064d3fbe954746c071cb72', 'qos_specs': None, 'reservations': ['d6688115-7e3d-438c-8fc0-69b8ec7a1cde', '8ca9607c-d2b9-47d7-8e24-f752a5ed3e59', 'a110fbe0-e00b-4d63-a1a6-1c96ea375721', '06947137-4319-40d9-b8a3-4db09cd63ce4'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': '39328f4ed6da4a85ac312bc5637b1500', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-TaggedBootDevicesTest-volume-1099737906', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '75ab8873f7064d3fbe954746c071cb72', 'qos_specs': None, 'reservations': ['d6688115-7e3d-438c-8fc0-69b8ec7a1cde', '8ca9607c-d2b9-47d7-8e24-f752a5ed3e59', 'a110fbe0-e00b-4d63-a1a6-1c96ea375721', '06947137-4319-40d9-b8a3-4db09cd63ce4'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': '39328f4ed6da4a85ac312bc5637b1500', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': '39328f4ed6da4a85ac312bc5637b1500', 'metadata': {}, 'qos_specs': None}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=b3f0dec1-7758-44a8-82e0-76ada8837428,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(e80c78b2-20c9-4f9b-8898-6ab131d1eb06),volume_id=e80c78b2-20c9-4f9b-8898-6ab131d1eb06,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:34:17Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TaggedBootDevicesTest-volume-1099737906',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=e80c78b2-20c9-4f9b-8898-6ab131d1eb06,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='75ab8873f7064d3fbe954746c071cb72',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:34:17Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:34:17Z,use_quota=True,user_id='39328f4ed6da4a85ac312bc5637b1500',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 15:36:35 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:34:17Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TaggedBootDevicesTest-volume-1099737906',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=e80c78b2-20c9-4f9b-8898-6ab131d1eb06,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='75ab8873f7064d3fbe954746c071cb72',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:34:17Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:34:17Z,use_quota=True,user_id='39328f4ed6da4a85ac312bc5637b1500',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:34:17Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TaggedBootDevicesTest-volume-1099737906',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=e80c78b2-20c9-4f9b-8898-6ab131d1eb06,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='75ab8873f7064d3fbe954746c071cb72',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:34:17Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:34:17Z,use_quota=True,user_id='39328f4ed6da4a85ac312bc5637b1500',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 15:36:35 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 15:36:35 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 15:36:35 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:36:35 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 15:36:35 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1237, in execute Apr 17 15:36:35 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_from_image(context, Apr 17 15:36:35 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/utils.py", line 670, in _wrapper Apr 17 15:36:35 user cinder-volume[84696]: ERROR cinder.volume.manager return r(f, *args, **kwargs) Apr 17 15:36:35 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/tenacity/__init__.py", line 423, in __call__ Apr 17 15:36:35 user cinder-volume[84696]: ERROR cinder.volume.manager do = self.iter(retry_state=retry_state) Apr 17 15:36:35 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/tenacity/__init__.py", line 360, in iter Apr 17 15:36:35 user cinder-volume[84696]: ERROR cinder.volume.manager return fut.result() Apr 17 15:36:35 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/lib/python3.10/concurrent/futures/_base.py", line 451, in result Apr 17 15:36:35 user cinder-volume[84696]: ERROR cinder.volume.manager return self.__get_result() Apr 17 15:36:35 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result Apr 17 15:36:35 user cinder-volume[84696]: ERROR cinder.volume.manager raise self._exception Apr 17 15:36:35 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/tenacity/__init__.py", line 426, in __call__ Apr 17 15:36:35 user cinder-volume[84696]: ERROR cinder.volume.manager result = fn(*args, **kwargs) Apr 17 15:36:35 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1125, in _create_from_image Apr 17 15:36:35 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_from_image_cache_or_download( Apr 17 15:36:35 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 944, in _create_from_image_cache_or_download Apr 17 15:36:35 user cinder-volume[84696]: ERROR cinder.volume.manager model_update, cloned = self._create_from_image_cache( Apr 17 15:36:35 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 843, in _create_from_image_cache Apr 17 15:36:35 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_from_source_volume( Apr 17 15:36:35 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 659, in _create_from_source_volume Apr 17 15:36:35 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self.driver.create_cloned_volume(volume, srcvol_ref) Apr 17 15:36:35 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 525, in create_cloned_volume Apr 17 15:36:35 user cinder-volume[84696]: ERROR cinder.volume.manager self._clone_object(vname, cvname) Apr 17 15:36:35 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 461, in _clone_object Apr 17 15:36:35 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_snapshot(oname, coname) Apr 17 15:36:35 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 639, in create_snapshot Apr 17 15:36:35 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201, 204): Apr 17 15:36:35 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 15:36:35 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 15:36:35 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (b1232b89-540b-4b2f-8c7c-9165c17366a0) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:36:35 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (b1232b89-540b-4b2f-8c7c-9165c17366a0) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:36:35 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (4bdc302e-f359-4099-99a8-482da233d7cd) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:36:35 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (4bdc302e-f359-4099-99a8-482da233d7cd) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:36:35 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (58293bf3-a01b-4124-8ed5-33388987ea91) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:36:35 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (58293bf3-a01b-4124-8ed5-33388987ea91) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:36:35 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (6298f3cf-bb1b-429d-82fc-ec346f35e33b) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:36:35 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Updating volume e80c78b2-20c9-4f9b-8898-6ab131d1eb06 with {'scheduled_at': datetime.datetime(2023, 4, 17, 15, 36, 35, 276960), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 15:36:35 user cinder-volume[84696]: DEBUG cinder.utils [None req-dbedb542-4536-4c1d-b79c-b9c0a6c64820 tempest-VolumesAdminNegativeTest-1285212149 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 1.935(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:36:35 user cinder-volume[84696]: DEBUG cinder.utils [None req-dbedb542-4536-4c1d-b79c-b9c0a6c64820 tempest-VolumesAdminNegativeTest-1285212149 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:215a6a5b-25ca-48fa-a8e3-72416d7a39e2/luns/v_215a6a5b-25ca-48fa-a8e3-72416d7a39e2 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:36:35 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Volume e80c78b2-20c9-4f9b-8898-6ab131d1eb06: re-scheduling SchedulerAPI.create_volume attempt 1 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 15:36:35 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Volume e80c78b2-20c9-4f9b-8898-6ab131d1eb06: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 15:36:35 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] deleating volume v_e80c78b2-20c9-4f9b-8898-6ab131d1eb06 {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 15:36:35 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] get snapshots for volume v_e80c78b2-20c9-4f9b-8898-6ab131d1eb06 {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 15:36:35 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_e80c78b2-20c9-4f9b-8898-6ab131d1eb06/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:36:35 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_e80c78b2-20c9-4f9b-8898-6ab131d1eb06/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:36:35 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Volume reschedule parameters: True retry: {'num_attempts': 2, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 15:36:35 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Flow 'volume_create_manager' (48ffe1e2-23fc-4875-b092-e00508064396) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:36:35 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (085f0fc6-3c9a-444a-9e38-9707c821a736) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:36:35 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (085f0fc6-3c9a-444a-9e38-9707c821a736) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:34:17Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TaggedBootDevicesTest-volume-1099737906',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=e80c78b2-20c9-4f9b-8898-6ab131d1eb06,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='75ab8873f7064d3fbe954746c071cb72',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:36:35Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:36:35Z,use_quota=True,user_id='39328f4ed6da4a85ac312bc5637b1500',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:36:35 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (fe973769-6b51-4a99-b47e-a6ce4a1b7f6a) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:36:35 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (fe973769-6b51-4a99-b47e-a6ce4a1b7f6a) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:36:35 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (7c5ca36e-3a93-4ece-ab54-7d8d8b42ff32) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:36:35 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (7c5ca36e-3a93-4ece-ab54-7d8d8b42ff32) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'image', 'volume_id': 'e80c78b2-20c9-4f9b-8898-6ab131d1eb06', 'volume_name': 'volume-e80c78b2-20c9-4f9b-8898-6ab131d1eb06', 'volume_size': 1, 'image_id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'image_location': (None, None), 'image_meta': {'name': 'cirros-0.5.2-x86_64-disk', 'disk_format': 'qcow2', 'container_format': 'bare', 'visibility': 'public', 'size': 16300544, 'virtual_size': 117440512, 'status': 'active', 'checksum': 'b874c39491a2377b8490f5f1e89761a4', 'protected': False, 'min_ram': 0, 'min_disk': 0, 'owner': 'a43c211b4ed542538799349489b1d884', 'os_hidden': False, 'os_hash_algo': 'sha512', 'os_hash_value': '6b813aa46bb90b4da216a4d19376593fa3f4fc7e617f03a92b7fe11e9a3981cbe8f0959dbebe36225e5f53dc4492341a4863cac4ed1ee0909f3fc78ef9c3e869', 'id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'created_at': datetime.datetime(2023, 4, 17, 15, 19, 53, tzinfo=datetime.timezone.utc), 'updated_at': datetime.datetime(2023, 4, 17, 15, 19, 55, tzinfo=datetime.timezone.utc), 'tags': [], 'file': '/v2/images/b3f0dec1-7758-44a8-82e0-76ada8837428/file', 'properties': {'hw_rng_model': 'virtio', 'owner_specified.openstack.md5': '', 'owner_specified.openstack.object': 'images/cirros-0.5.2-x86_64-disk', 'owner_specified.openstack.sha256': ''}}, 'image_service': }' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:36:35 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (3fc4288d-ccff-4a94-acbe-e20b6000e098) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:36:35 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (3fc4288d-ccff-4a94-acbe-e20b6000e098) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:36:35 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (f8affc87-9883-4a0c-8037-a89dabe85101) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:36:35 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Volume e80c78b2-20c9-4f9b-8898-6ab131d1eb06: being created as image with specification: {'status': 'creating', 'volume_name': 'volume-e80c78b2-20c9-4f9b-8898-6ab131d1eb06', 'volume_size': 1, 'image_id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'image_location': (None, None), 'image_meta': {'name': 'cirros-0.5.2-x86_64-disk', 'disk_format': 'qcow2', 'container_format': 'bare', 'visibility': 'public', 'size': 16300544, 'virtual_size': 117440512, 'status': 'active', 'checksum': 'b874c39491a2377b8490f5f1e89761a4', 'protected': False, 'min_ram': 0, 'min_disk': 0, 'owner': 'a43c211b4ed542538799349489b1d884', 'os_hidden': False, 'os_hash_algo': 'sha512', 'os_hash_value': '6b813aa46bb90b4da216a4d19376593fa3f4fc7e617f03a92b7fe11e9a3981cbe8f0959dbebe36225e5f53dc4492341a4863cac4ed1ee0909f3fc78ef9c3e869', 'id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'created_at': datetime.datetime(2023, 4, 17, 15, 19, 53, tzinfo=datetime.timezone.utc), 'updated_at': datetime.datetime(2023, 4, 17, 15, 19, 55, tzinfo=datetime.timezone.utc), 'tags': [], 'file': '/v2/images/b3f0dec1-7758-44a8-82e0-76ada8837428/file', 'properties': {'hw_rng_model': 'virtio', 'owner_specified.openstack.md5': '', 'owner_specified.openstack.object': 'images/cirros-0.5.2-x86_64-disk', 'owner_specified.openstack.sha256': ''}}, 'image_service': } Apr 17 15:36:35 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Cloning e80c78b2-20c9-4f9b-8898-6ab131d1eb06 from image b3f0dec1-7758-44a8-82e0-76ada8837428 at location (None, None). {{(pid=84696) _create_from_image /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1071}} Apr 17 15:36:35 user cinder-volume[84696]: DEBUG cinder.coordination [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Acquiring lock "cinder-b3f0dec1-7758-44a8-82e0-76ada8837428" by "_prepare_image_cache_entry" {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:192}} Apr 17 15:36:35 user cinder-volume[84696]: DEBUG cinder.coordination [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Lock "cinder-b3f0dec1-7758-44a8-82e0-76ada8837428" acquired by "_prepare_image_cache_entry" :: waited 0.007s {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:196}} Apr 17 15:36:35 user cinder-volume[84696]: DEBUG cinder.image.cache [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Found image-volume cache entry: {'id': 1, 'image_id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'volume_id': '006f20ae-5306-44fe-afdf-e433f79284e8', 'host': 'user@joviandss-2#joviandss-2', 'size': 1, 'image_updated_at': datetime.datetime(2023, 4, 17, 15, 19, 55), 'last_used': datetime.datetime(2023, 4, 17, 15, 36, 35, 575476)}. {{(pid=84696) get_entry /opt/stack/cinder/cinder/image/cache.py:76}} Apr 17 15:36:35 user cinder-volume[84696]: DEBUG cinder.image.cache [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Image-volume cache entry image_update_at = 2023-04-17 15:19:55+00:00, requested image updated_at = 2023-04-17 15:19:55+00:00. {{(pid=84696) _should_update_entry /opt/stack/cinder/cinder/image/cache.py:259}} Apr 17 15:36:35 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Found cache entry for image = b3f0dec1-7758-44a8-82e0-76ada8837428 on host user@joviandss-2#joviandss-2. {{(pid=84696) _prepare_image_cache_entry /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:886}} Apr 17 15:36:35 user cinder-volume[84696]: DEBUG cinder.coordination [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Lock "cinder-b3f0dec1-7758-44a8-82e0-76ada8837428" released by "_prepare_image_cache_entry" :: held 0.012s {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:207}} Apr 17 15:36:35 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Attempting to retrieve cache entry for image = b3f0dec1-7758-44a8-82e0-76ada8837428 on host user@joviandss-2#joviandss-2. {{(pid=84696) _create_from_image_cache /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:827}} Apr 17 15:36:35 user cinder-volume[84696]: DEBUG cinder.image.cache [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Found image-volume cache entry: {'id': 1, 'image_id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'volume_id': '006f20ae-5306-44fe-afdf-e433f79284e8', 'host': 'user@joviandss-2#joviandss-2', 'size': 1, 'image_updated_at': datetime.datetime(2023, 4, 17, 15, 19, 55), 'last_used': datetime.datetime(2023, 4, 17, 15, 36, 35, 586369)}. {{(pid=84696) get_entry /opt/stack/cinder/cinder/image/cache.py:76}} Apr 17 15:36:35 user cinder-volume[84696]: DEBUG cinder.image.cache [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Image-volume cache entry image_update_at = 2023-04-17 15:19:55+00:00, requested image updated_at = 2023-04-17 15:19:55+00:00. {{(pid=84696) _should_update_entry /opt/stack/cinder/cinder/image/cache.py:259}} Apr 17 15:36:35 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Creating from source image-volume 006f20ae-5306-44fe-afdf-e433f79284e8 {{(pid=84696) _create_from_image_cache /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:841}} Apr 17 15:36:35 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] cloned volume 006f20ae-5306-44fe-afdf-e433f79284e8 to e80c78b2-20c9-4f9b-8898-6ab131d1eb06 {{(pid=84696) create_cloned_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:521}} Apr 17 15:36:35 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] cloning v_006f20ae-5306-44fe-afdf-e433f79284e8 to v_e80c78b2-20c9-4f9b-8898-6ab131d1eb06 {{(pid=84696) _clone_object /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:456}} Apr 17 15:36:35 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] create snapshot v_e80c78b2-20c9-4f9b-8898-6ab131d1eb06 {{(pid=84696) create_snapshot /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:635}} Apr 17 15:36:35 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:36:35 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:36:38 user cinder-volume[84696]: DEBUG cinder.utils [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.994(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:36:38 user cinder-volume[84696]: DEBUG cinder.utils [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_e80c78b2-20c9-4f9b-8898-6ab131d1eb06/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:36:38 user cinder-volume[84696]: DEBUG cinder.utils [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.952(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:36:38 user cinder-volume[84696]: DEBUG cinder.utils [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_5d9c86cd-3e22-4926-bfc9-e5a37f6ae76c/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:36:38 user cinder-volume[84696]: DEBUG cinder.utils [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.703(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:36:38 user cinder-volume[84696]: DEBUG cinder.utils [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:36:38 user cinder-volume[84696]: DEBUG cinder.utils [None req-dbedb542-4536-4c1d-b79c-b9c0a6c64820 tempest-VolumesAdminNegativeTest-1285212149 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 4.960(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:36:38 user cinder-volume[84696]: DEBUG cinder.utils [None req-dbedb542-4536-4c1d-b79c-b9c0a6c64820 tempest-VolumesAdminNegativeTest-1285212149 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:215a6a5b-25ca-48fa-a8e3-72416d7a39e2/luns/v_215a6a5b-25ca-48fa-a8e3-72416d7a39e2 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:36:38 user cinder-volume[84696]: DEBUG cinder.utils [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.689(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:36:38 user cinder-volume[84696]: DEBUG cinder.utils [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:36:41 user cinder-volume[84696]: DEBUG cinder.utils [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.772(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:36:41 user cinder-volume[84696]: DEBUG cinder.utils [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:36:41 user cinder-volume[84696]: DEBUG cinder.utils [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.068(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:36:41 user cinder-volume[84696]: DEBUG cinder.utils [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_e80c78b2-20c9-4f9b-8898-6ab131d1eb06/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:36:41 user cinder-volume[84696]: DEBUG cinder.utils [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.416(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:36:41 user cinder-volume[84696]: DEBUG cinder.utils [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_0fed6b21-0520-4554-bee4-9540a03f9af0/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:36:41 user cinder-volume[84696]: DEBUG cinder.utils [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.083(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:36:41 user cinder-volume[84696]: DEBUG cinder.utils [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:36:44 user cinder-volume[84696]: DEBUG cinder.utils [None req-dbedb542-4536-4c1d-b79c-b9c0a6c64820 tempest-VolumesAdminNegativeTest-1285212149 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 11.100(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:36:44 user cinder-volume[84696]: DEBUG cinder.utils [None req-dbedb542-4536-4c1d-b79c-b9c0a6c64820 tempest-VolumesAdminNegativeTest-1285212149 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:215a6a5b-25ca-48fa-a8e3-72416d7a39e2/luns/v_215a6a5b-25ca-48fa-a8e3-72416d7a39e2 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:36:47 user cinder-volume[84696]: DEBUG cinder.utils [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.210(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:36:47 user cinder-volume[84696]: DEBUG cinder.utils [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_e80c78b2-20c9-4f9b-8898-6ab131d1eb06/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:36:47 user cinder-volume[84696]: DEBUG cinder.utils [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 11.918(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:36:47 user cinder-volume[84696]: DEBUG cinder.utils [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:36:47 user cinder-volume[84696]: DEBUG cinder.utils [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 73.750(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:36:47 user cinder-volume[84696]: DEBUG cinder.utils [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_62709a81-268b-4227-ba4a-070034f36414/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:36:50 user cinder-volume[84696]: DEBUG cinder.utils [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.730(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:36:50 user cinder-volume[84696]: DEBUG cinder.utils [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_a4d96ab8-2072-4eff-b803-53b4f27d056f/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:36:50 user cinder-volume[84696]: DEBUG cinder.utils [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.719(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:36:50 user cinder-volume[84696]: DEBUG cinder.utils [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_44f909cf-7774-41f6-9f86-805456609ff8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:36:50 user cinder-volume[84696]: DEBUG cinder.utils [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.493(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:36:50 user cinder-volume[84696]: DEBUG cinder.utils [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:36:50 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 73.851(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:36:50 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:36:53 user cinder-volume[84696]: DEBUG cinder.utils [None req-dbedb542-4536-4c1d-b79c-b9c0a6c64820 tempest-VolumesAdminNegativeTest-1285212149 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 20.316(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:36:53 user cinder-volume[84696]: DEBUG cinder.utils [None req-dbedb542-4536-4c1d-b79c-b9c0a6c64820 tempest-VolumesAdminNegativeTest-1285212149 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:215a6a5b-25ca-48fa-a8e3-72416d7a39e2/luns/v_215a6a5b-25ca-48fa-a8e3-72416d7a39e2 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:36:56 user cinder-volume[84696]: DEBUG cinder.utils [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.132(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:36:56 user cinder-volume[84696]: DEBUG cinder.utils [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:36:56 user cinder-volume[84696]: DEBUG cinder.utils [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 73.691(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:36:56 user cinder-volume[84696]: DEBUG cinder.utils [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_17f8d4f7-2dcd-4041-8d0b-5c8afebb75bf/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:36:56 user cinder-volume[84696]: DEBUG cinder.utils [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.429(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:36:56 user cinder-volume[84696]: DEBUG cinder.utils [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_e80c78b2-20c9-4f9b-8898-6ab131d1eb06/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:36:56 user cinder-volume[84696]: DEBUG cinder.utils [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 132.050(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:36:56 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:36:56 user cinder-volume[84696]: INFO cinder.message.api [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Creating message record for request_id = req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 Apr 17 15:36:56 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 15:36:56 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=20,cinder:INSERT=1,cinder:UPDATE=1 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:36:56 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=13,cinder:UPDATE=9,cinder:INSERT=1 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:36:56 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (50a8e45b-fa50-4f45-8385-11804bfc7a3e) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 15:36:56 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 15:36:56 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:34:44Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesSnapshotNegativeTestJSON-Volume-379445405',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=29d9e55e-0b8b-4f70-b5c4-830d39f10fcd,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='0b21bda55be444fc9f1a9c8c37b09224',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:34:45Z,service_uuid=None,shared_targets=True,size=2,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:34:45Z,use_quota=True,user_id='df17d7972cc649eb829bde41b8a718e4',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 15:36:56 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:34:44Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesSnapshotNegativeTestJSON-Volume-379445405',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=29d9e55e-0b8b-4f70-b5c4-830d39f10fcd,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='0b21bda55be444fc9f1a9c8c37b09224',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:34:45Z,service_uuid=None,shared_targets=True,size=2,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:34:45Z,use_quota=True,user_id='df17d7972cc649eb829bde41b8a718e4',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(29d9e55e-0b8b-4f70-b5c4-830d39f10fcd),volume_id=29d9e55e-0b8b-4f70-b5c4-830d39f10fcd,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': '29d9e55e-0b8b-4f70-b5c4-830d39f10fcd', 'volume_name': 'volume-29d9e55e-0b8b-4f70-b5c4-830d39f10fcd', 'volume_size': 2}} Apr 17 15:36:56 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'retry': {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']}, 'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': '29d9e55e-0b8b-4f70-b5c4-830d39f10fcd', 'volume': {'id': '29d9e55e-0b8b-4f70-b5c4-830d39f10fcd', '_name_id': None, 'ec2_id': None, 'user_id': 'df17d7972cc649eb829bde41b8a718e4', 'project_id': '0b21bda55be444fc9f1a9c8c37b09224', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 2, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-VolumesSnapshotNegativeTestJSON-Volume-379445405', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T15:34:44.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-29d9e55e-0b8b-4f70-b5c4-830d39f10fcd', 'name_id': '29d9e55e-0b8b-4f70-b5c4-830d39f10fcd', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumesSnapshotNegativeTestJSON-Volume-379445405', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '0b21bda55be444fc9f1a9c8c37b09224', 'qos_specs': None, 'reservations': ['da4aff04-a876-4157-b666-0fcc804a5076', '8e06fcb4-c53a-4537-9e40-36798fd5109c', '6bf081db-b8e9-44ff-ab74-ff0b0db7654a', 'ea5f14dc-ad5d-406a-8281-815b78e4eeb3'], 'size': 2, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'df17d7972cc649eb829bde41b8a718e4', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumesSnapshotNegativeTestJSON-Volume-379445405', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '0b21bda55be444fc9f1a9c8c37b09224', 'qos_specs': None, 'reservations': ['da4aff04-a876-4157-b666-0fcc804a5076', '8e06fcb4-c53a-4537-9e40-36798fd5109c', '6bf081db-b8e9-44ff-ab74-ff0b0db7654a', 'ea5f14dc-ad5d-406a-8281-815b78e4eeb3'], 'size': 2, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'df17d7972cc649eb829bde41b8a718e4', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 2, 'availability_zone': 'nova', 'user_id': 'df17d7972cc649eb829bde41b8a718e4', 'metadata': {}, 'qos_specs': None}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(29d9e55e-0b8b-4f70-b5c4-830d39f10fcd),volume_id=29d9e55e-0b8b-4f70-b5c4-830d39f10fcd,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:34:44Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesSnapshotNegativeTestJSON-Volume-379445405',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=29d9e55e-0b8b-4f70-b5c4-830d39f10fcd,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='0b21bda55be444fc9f1a9c8c37b09224',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:34:45Z,service_uuid=None,shared_targets=True,size=2,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:34:45Z,use_quota=True,user_id='df17d7972cc649eb829bde41b8a718e4',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 15:36:56 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:34:44Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesSnapshotNegativeTestJSON-Volume-379445405',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=29d9e55e-0b8b-4f70-b5c4-830d39f10fcd,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='0b21bda55be444fc9f1a9c8c37b09224',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:34:45Z,service_uuid=None,shared_targets=True,size=2,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:34:45Z,use_quota=True,user_id='df17d7972cc649eb829bde41b8a718e4',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:34:44Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesSnapshotNegativeTestJSON-Volume-379445405',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=29d9e55e-0b8b-4f70-b5c4-830d39f10fcd,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='0b21bda55be444fc9f1a9c8c37b09224',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:34:45Z,service_uuid=None,shared_targets=True,size=2,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:34:45Z,use_quota=True,user_id='df17d7972cc649eb829bde41b8a718e4',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 15:36:56 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 15:36:56 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 15:36:56 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:36:56 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 15:36:56 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 15:36:56 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 15:36:56 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 15:36:56 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 15:36:56 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 15:36:56 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 15:36:56 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 15:36:56 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 15:36:56 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 15:36:56 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 15:36:56 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 15:36:56 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 15:36:56 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 15:36:56 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 15:36:56 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 15:36:56 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 15:36:56 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (50a8e45b-fa50-4f45-8385-11804bfc7a3e) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:36:56 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (50a8e45b-fa50-4f45-8385-11804bfc7a3e) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:36:56 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (a1b2df09-56f1-4ec9-8694-d8d098d9c67e) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:36:56 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (a1b2df09-56f1-4ec9-8694-d8d098d9c67e) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:36:56 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (329f0f77-6eed-4521-be8e-00d969b90d4c) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:36:56 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (329f0f77-6eed-4521-be8e-00d969b90d4c) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:36:56 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (8dd354fa-c2fd-47a3-b436-20c24e43b89c) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:36:56 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Updating volume 29d9e55e-0b8b-4f70-b5c4-830d39f10fcd with {'scheduled_at': datetime.datetime(2023, 4, 17, 15, 36, 56, 800168), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 15:36:56 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Volume 29d9e55e-0b8b-4f70-b5c4-830d39f10fcd: re-scheduling SchedulerAPI.create_volume attempt 1 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 15:36:56 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Volume 29d9e55e-0b8b-4f70-b5c4-830d39f10fcd: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 15:36:56 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] deleating volume v_29d9e55e-0b8b-4f70-b5c4-830d39f10fcd {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 15:36:56 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] get snapshots for volume v_29d9e55e-0b8b-4f70-b5c4-830d39f10fcd {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 15:36:56 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_29d9e55e-0b8b-4f70-b5c4-830d39f10fcd/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:36:56 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_29d9e55e-0b8b-4f70-b5c4-830d39f10fcd/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:36:56 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Volume reschedule parameters: True retry: {'num_attempts': 2, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 15:36:56 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Flow 'volume_create_manager' (8f26591c-ce7d-4cf5-986c-6e45832cded6) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:36:56 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (1207f657-5807-48c5-b093-65262d08fd13) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:36:56 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (1207f657-5807-48c5-b093-65262d08fd13) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:34:44Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesSnapshotNegativeTestJSON-Volume-379445405',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=29d9e55e-0b8b-4f70-b5c4-830d39f10fcd,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='0b21bda55be444fc9f1a9c8c37b09224',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:36:57Z,service_uuid=None,shared_targets=True,size=2,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:36:57Z,use_quota=True,user_id='df17d7972cc649eb829bde41b8a718e4',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:36:56 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (da3db1cb-1b53-4ead-9ae4-b6be31938c54) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:36:56 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (da3db1cb-1b53-4ead-9ae4-b6be31938c54) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:36:56 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (817c79b0-10e6-44e0-a905-f56f897ee2e0) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:36:56 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (817c79b0-10e6-44e0-a905-f56f897ee2e0) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': '29d9e55e-0b8b-4f70-b5c4-830d39f10fcd', 'volume_name': 'volume-29d9e55e-0b8b-4f70-b5c4-830d39f10fcd', 'volume_size': 2}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:36:56 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (88867c92-c4e9-4248-ad4e-f920bbd38421) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:36:56 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (88867c92-c4e9-4248-ad4e-f920bbd38421) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:36:57 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (f0474e26-2b7d-4e8c-b91c-3f0c0fe3fa4a) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:36:57 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Volume 29d9e55e-0b8b-4f70-b5c4-830d39f10fcd: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-29d9e55e-0b8b-4f70-b5c4-830d39f10fcd', 'volume_size': 2} Apr 17 15:36:57 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] creating volume v_29d9e55e-0b8b-4f70-b5c4-830d39f10fcd. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 15:36:57 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] create volume {'name': 'v_29d9e55e-0b8b-4f70-b5c4-830d39f10fcd', 'size': '2147483648', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 15:36:57 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:36:57 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:36:59 user cinder-volume[84696]: DEBUG cinder.utils [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 134.398(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:36:59 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:36:59 user cinder-volume[84696]: INFO cinder.message.api [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Creating message record for request_id = req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 Apr 17 15:36:59 user cinder-volume[84696]: DEBUG cinder.utils [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.982(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:36:59 user cinder-volume[84696]: DEBUG cinder.utils [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_29d9e55e-0b8b-4f70-b5c4-830d39f10fcd/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:36:59 user cinder-volume[84696]: DEBUG cinder.utils [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.853(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:36:59 user cinder-volume[84696]: DEBUG cinder.utils [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_0fed6b21-0520-4554-bee4-9540a03f9af0/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:36:59 user cinder-volume[84696]: DEBUG cinder.utils [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.520(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:36:59 user cinder-volume[84696]: DEBUG cinder.utils [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:36:59 user cinder-volume[84696]: DEBUG cinder.utils [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.810(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:36:59 user cinder-volume[84696]: DEBUG cinder.utils [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:36:59 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 15:36:59 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (1fa3fdbc-1dbf-4f77-9f17-81cf9b9780d1) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 15:36:59 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 15:36:59 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:34:45Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesActionsTest-Volume-737622743',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=44073502-102d-452a-8244-69a8233b9060,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='a8b97ff6ed6047f0a33e74684a1c162b',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:34:45Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:34:45Z,use_quota=True,user_id='c993ceeb4a5c49eea14057e2125518ad',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 15:36:59 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:34:45Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesActionsTest-Volume-737622743',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=44073502-102d-452a-8244-69a8233b9060,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='a8b97ff6ed6047f0a33e74684a1c162b',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:34:45Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:34:45Z,use_quota=True,user_id='c993ceeb4a5c49eea14057e2125518ad',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(44073502-102d-452a-8244-69a8233b9060),volume_id=44073502-102d-452a-8244-69a8233b9060,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': '44073502-102d-452a-8244-69a8233b9060', 'volume_name': 'volume-44073502-102d-452a-8244-69a8233b9060', 'volume_size': 1}} Apr 17 15:36:59 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'retry': {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']}, 'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': '44073502-102d-452a-8244-69a8233b9060', 'volume': {'id': '44073502-102d-452a-8244-69a8233b9060', '_name_id': None, 'ec2_id': None, 'user_id': 'c993ceeb4a5c49eea14057e2125518ad', 'project_id': 'a8b97ff6ed6047f0a33e74684a1c162b', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-VolumesActionsTest-Volume-737622743', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T15:34:45.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-44073502-102d-452a-8244-69a8233b9060', 'name_id': '44073502-102d-452a-8244-69a8233b9060', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumesActionsTest-Volume-737622743', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': 'a8b97ff6ed6047f0a33e74684a1c162b', 'qos_specs': None, 'reservations': ['bb018cba-ad93-4b64-acaa-8608d0a298cb', 'c3b20513-152c-40a3-a11d-aac4f6530172', '567445e0-7c93-4f93-b254-4b34cc804061', '988486ce-3f26-4167-a9e9-e6aa5ffe2065'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'c993ceeb4a5c49eea14057e2125518ad', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumesActionsTest-Volume-737622743', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': 'a8b97ff6ed6047f0a33e74684a1c162b', 'qos_specs': None, 'reservations': ['bb018cba-ad93-4b64-acaa-8608d0a298cb', 'c3b20513-152c-40a3-a11d-aac4f6530172', '567445e0-7c93-4f93-b254-4b34cc804061', '988486ce-3f26-4167-a9e9-e6aa5ffe2065'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'c993ceeb4a5c49eea14057e2125518ad', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': 'c993ceeb4a5c49eea14057e2125518ad', 'metadata': {}, 'qos_specs': None}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(44073502-102d-452a-8244-69a8233b9060),volume_id=44073502-102d-452a-8244-69a8233b9060,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:34:45Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesActionsTest-Volume-737622743',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=44073502-102d-452a-8244-69a8233b9060,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='a8b97ff6ed6047f0a33e74684a1c162b',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:34:45Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:34:45Z,use_quota=True,user_id='c993ceeb4a5c49eea14057e2125518ad',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 15:36:59 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:34:45Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesActionsTest-Volume-737622743',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=44073502-102d-452a-8244-69a8233b9060,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='a8b97ff6ed6047f0a33e74684a1c162b',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:34:45Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:34:45Z,use_quota=True,user_id='c993ceeb4a5c49eea14057e2125518ad',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:34:45Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesActionsTest-Volume-737622743',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=44073502-102d-452a-8244-69a8233b9060,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='a8b97ff6ed6047f0a33e74684a1c162b',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:34:45Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:34:45Z,use_quota=True,user_id='c993ceeb4a5c49eea14057e2125518ad',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 15:36:59 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 15:36:59 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 15:36:59 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:36:59 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 15:36:59 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 15:36:59 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 15:36:59 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 15:36:59 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 15:36:59 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 15:36:59 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 15:36:59 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 15:36:59 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 15:36:59 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 15:36:59 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 15:36:59 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 15:36:59 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 15:36:59 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 15:36:59 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 15:36:59 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 15:36:59 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 15:36:59 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (1fa3fdbc-1dbf-4f77-9f17-81cf9b9780d1) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:36:59 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (1fa3fdbc-1dbf-4f77-9f17-81cf9b9780d1) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:36:59 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (c484ab16-8f47-4695-9188-3ab5b6d4ceaa) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:36:59 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (c484ab16-8f47-4695-9188-3ab5b6d4ceaa) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:36:59 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (56df3c26-a1d6-42f1-a5e2-343041d15337) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:36:59 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (56df3c26-a1d6-42f1-a5e2-343041d15337) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:36:59 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (dcd2578c-e54b-4460-9110-7a625f5c8f06) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:36:59 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Updating volume 44073502-102d-452a-8244-69a8233b9060 with {'scheduled_at': datetime.datetime(2023, 4, 17, 15, 36, 59, 868711), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 15:36:59 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Volume 44073502-102d-452a-8244-69a8233b9060: re-scheduling SchedulerAPI.create_volume attempt 1 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 15:36:59 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Volume 44073502-102d-452a-8244-69a8233b9060: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 15:36:59 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] deleating volume v_44073502-102d-452a-8244-69a8233b9060 {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 15:36:59 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] get snapshots for volume v_44073502-102d-452a-8244-69a8233b9060 {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 15:36:59 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_44073502-102d-452a-8244-69a8233b9060/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:36:59 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_44073502-102d-452a-8244-69a8233b9060/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:37:00 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Volume reschedule parameters: True retry: {'num_attempts': 2, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 15:37:00 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Flow 'volume_create_manager' (1353dfbd-eba4-4318-976f-a758c44f9eb2) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:37:00 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (474a3bd3-757a-410a-b2be-a4c346e7dc7d) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:37:00 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (474a3bd3-757a-410a-b2be-a4c346e7dc7d) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:34:45Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesActionsTest-Volume-737622743',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=44073502-102d-452a-8244-69a8233b9060,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='a8b97ff6ed6047f0a33e74684a1c162b',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:37:00Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:37:00Z,use_quota=True,user_id='c993ceeb4a5c49eea14057e2125518ad',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:37:00 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (d1e1c597-b5d4-4f26-a4cf-5d131084fa80) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:37:00 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (d1e1c597-b5d4-4f26-a4cf-5d131084fa80) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:37:00 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (99e025b0-8c90-4fd6-8973-baba2796f35a) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:37:00 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (99e025b0-8c90-4fd6-8973-baba2796f35a) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': '44073502-102d-452a-8244-69a8233b9060', 'volume_name': 'volume-44073502-102d-452a-8244-69a8233b9060', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:37:00 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (e882b36a-d62e-4e7b-a207-b6b3671d802c) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:37:00 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (e882b36a-d62e-4e7b-a207-b6b3671d802c) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:37:00 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (62bbc6cc-8ee2-47d0-9eeb-4d5fee96582f) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:37:00 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Volume 44073502-102d-452a-8244-69a8233b9060: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-44073502-102d-452a-8244-69a8233b9060', 'volume_size': 1} Apr 17 15:37:00 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] creating volume v_44073502-102d-452a-8244-69a8233b9060. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 15:37:00 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] create volume {'name': 'v_44073502-102d-452a-8244-69a8233b9060', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 15:37:00 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:37:00 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:37:02 user cinder-volume[84696]: DEBUG cinder.utils [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.798(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:37:02 user cinder-volume[84696]: DEBUG cinder.utils [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:37:02 user cinder-volume[84696]: DEBUG cinder.utils [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.986(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:37:02 user cinder-volume[84696]: DEBUG cinder.utils [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_44073502-102d-452a-8244-69a8233b9060/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:37:02 user cinder-volume[84696]: DEBUG cinder.utils [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.066(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:37:02 user cinder-volume[84696]: DEBUG cinder.utils [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_29d9e55e-0b8b-4f70-b5c4-830d39f10fcd/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:37:02 user cinder-volume[84696]: DEBUG cinder.utils [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.890(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:37:02 user cinder-volume[84696]: DEBUG cinder.utils [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:37:05 user cinder-volume[84696]: DEBUG cinder.utils [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 73.767(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:37:05 user cinder-volume[84696]: DEBUG cinder.utils [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_8cc35ab1-363f-4ad6-9a37-239d2b1ea996/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:37:05 user cinder-volume[84696]: DEBUG cinder.utils [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 73.601(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:37:05 user cinder-volume[84696]: DEBUG cinder.utils [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:37:05 user cinder-volume[84696]: DEBUG cinder.utils [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.872(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:37:05 user cinder-volume[84696]: DEBUG cinder.utils [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:37:05 user cinder-volume[84696]: DEBUG cinder.utils [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.059(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:37:05 user cinder-volume[84696]: DEBUG cinder.utils [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_44073502-102d-452a-8244-69a8233b9060/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:37:09 user cinder-volume[84696]: DEBUG cinder.utils [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.192(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:37:09 user cinder-volume[84696]: DEBUG cinder.utils [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_29d9e55e-0b8b-4f70-b5c4-830d39f10fcd/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:37:09 user cinder-volume[84696]: DEBUG cinder.utils [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.016(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:37:09 user cinder-volume[84696]: DEBUG cinder.utils [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:37:12 user cinder-volume[84696]: DEBUG cinder.utils [None req-dbedb542-4536-4c1d-b79c-b9c0a6c64820 tempest-VolumesAdminNegativeTest-1285212149 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 38.748(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:37:12 user cinder-volume[84696]: DEBUG cinder.utils [None req-dbedb542-4536-4c1d-b79c-b9c0a6c64820 tempest-VolumesAdminNegativeTest-1285212149 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:215a6a5b-25ca-48fa-a8e3-72416d7a39e2/luns/v_215a6a5b-25ca-48fa-a8e3-72416d7a39e2 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:37:12 user cinder-volume[84696]: DEBUG cinder.utils [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.200(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:37:12 user cinder-volume[84696]: DEBUG cinder.utils [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_44073502-102d-452a-8244-69a8233b9060/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:37:12 user cinder-volume[84696]: DEBUG cinder.utils [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 73.479(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:37:12 user cinder-volume[84696]: DEBUG cinder.utils [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:37:12 user cinder-volume[84696]: DEBUG cinder.utils [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.018(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:37:12 user cinder-volume[84696]: DEBUG cinder.utils [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:37:12 user cinder-volume[84696]: DEBUG cinder.utils [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 73.749(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:37:12 user cinder-volume[84696]: DEBUG cinder.utils [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_5d9c86cd-3e22-4926-bfc9-e5a37f6ae76c/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:37:15 user cinder-volume[84696]: DEBUG cinder.utils [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.858(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:37:15 user cinder-volume[84696]: DEBUG cinder.utils [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_e80c78b2-20c9-4f9b-8898-6ab131d1eb06/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:37:15 user cinder-volume[84696]: DEBUG cinder.utils [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.566(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:37:15 user cinder-volume[84696]: DEBUG cinder.utils [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:37:15 user cinder-volume[84696]: DEBUG cinder.utils [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 137.506(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:37:15 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:37:15 user cinder-volume[84696]: INFO cinder.message.api [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Creating message record for request_id = req-de22aca9-c3f8-4015-be3e-9d70b59359d2 Apr 17 15:37:15 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 15:37:15 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (02a1a044-9922-44cb-ba7e-18400be516a1) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 15:37:15 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 15:37:15 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:34:57Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesCloneTest-Volume-2030679513',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=153b4df8-1f37-4118-8aa6-1c155ea5700c,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='82fce1724afa40dd914e0791c3ee69a3',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:34:58Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:34:58Z,use_quota=True,user_id='773ce6c2a0f844edaf666e190504e258',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 15:37:15 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:34:57Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesCloneTest-Volume-2030679513',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=153b4df8-1f37-4118-8aa6-1c155ea5700c,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='82fce1724afa40dd914e0791c3ee69a3',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:34:58Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:34:58Z,use_quota=True,user_id='773ce6c2a0f844edaf666e190504e258',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(153b4df8-1f37-4118-8aa6-1c155ea5700c),volume_id=153b4df8-1f37-4118-8aa6-1c155ea5700c,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': '153b4df8-1f37-4118-8aa6-1c155ea5700c', 'volume_name': 'volume-153b4df8-1f37-4118-8aa6-1c155ea5700c', 'volume_size': 1}} Apr 17 15:37:15 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'retry': {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']}, 'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': '153b4df8-1f37-4118-8aa6-1c155ea5700c', 'volume': {'id': '153b4df8-1f37-4118-8aa6-1c155ea5700c', '_name_id': None, 'ec2_id': None, 'user_id': '773ce6c2a0f844edaf666e190504e258', 'project_id': '82fce1724afa40dd914e0791c3ee69a3', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-VolumesCloneTest-Volume-2030679513', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T15:34:57.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-153b4df8-1f37-4118-8aa6-1c155ea5700c', 'name_id': '153b4df8-1f37-4118-8aa6-1c155ea5700c', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumesCloneTest-Volume-2030679513', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '82fce1724afa40dd914e0791c3ee69a3', 'qos_specs': None, 'reservations': ['4e0e5a8a-460c-4231-830c-7500b8b4d874', '4e75ce56-bc87-4771-a3a2-a212a9299682', '1e95db6e-9eea-496d-9f91-b9dc2fd1660c', 'd2d362d2-ccef-4207-b0ce-c2abf0695375'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': '773ce6c2a0f844edaf666e190504e258', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumesCloneTest-Volume-2030679513', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '82fce1724afa40dd914e0791c3ee69a3', 'qos_specs': None, 'reservations': ['4e0e5a8a-460c-4231-830c-7500b8b4d874', '4e75ce56-bc87-4771-a3a2-a212a9299682', '1e95db6e-9eea-496d-9f91-b9dc2fd1660c', 'd2d362d2-ccef-4207-b0ce-c2abf0695375'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': '773ce6c2a0f844edaf666e190504e258', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': '773ce6c2a0f844edaf666e190504e258', 'metadata': {}, 'qos_specs': None}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(153b4df8-1f37-4118-8aa6-1c155ea5700c),volume_id=153b4df8-1f37-4118-8aa6-1c155ea5700c,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:34:57Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesCloneTest-Volume-2030679513',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=153b4df8-1f37-4118-8aa6-1c155ea5700c,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='82fce1724afa40dd914e0791c3ee69a3',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:34:58Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:34:58Z,use_quota=True,user_id='773ce6c2a0f844edaf666e190504e258',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 15:37:15 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:34:57Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesCloneTest-Volume-2030679513',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=153b4df8-1f37-4118-8aa6-1c155ea5700c,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='82fce1724afa40dd914e0791c3ee69a3',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:34:58Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:34:58Z,use_quota=True,user_id='773ce6c2a0f844edaf666e190504e258',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:34:57Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesCloneTest-Volume-2030679513',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=153b4df8-1f37-4118-8aa6-1c155ea5700c,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='82fce1724afa40dd914e0791c3ee69a3',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:34:58Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:34:58Z,use_quota=True,user_id='773ce6c2a0f844edaf666e190504e258',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 15:37:15 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 15:37:15 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 15:37:15 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:37:15 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 15:37:15 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 15:37:15 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 15:37:15 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 15:37:15 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 15:37:15 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 15:37:15 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 15:37:15 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 15:37:15 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 15:37:15 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 15:37:15 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 15:37:15 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 15:37:15 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 15:37:15 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 15:37:15 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 15:37:15 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 15:37:15 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 15:37:15 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (02a1a044-9922-44cb-ba7e-18400be516a1) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:37:15 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (02a1a044-9922-44cb-ba7e-18400be516a1) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:37:15 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (1a455c89-502c-4392-bdb8-2b40a1b1f1eb) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:37:15 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (1a455c89-502c-4392-bdb8-2b40a1b1f1eb) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:37:15 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (0b7ec015-7c9a-4fa0-adc5-8827b35235f9) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:37:15 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (0b7ec015-7c9a-4fa0-adc5-8827b35235f9) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:37:15 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (dc6d0e22-ed05-4c9f-aa05-9065a5afc1a6) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:37:15 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Updating volume 153b4df8-1f37-4118-8aa6-1c155ea5700c with {'scheduled_at': datetime.datetime(2023, 4, 17, 15, 37, 15, 243035), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 15:37:15 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Volume 153b4df8-1f37-4118-8aa6-1c155ea5700c: re-scheduling SchedulerAPI.create_volume attempt 1 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 15:37:15 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Volume 153b4df8-1f37-4118-8aa6-1c155ea5700c: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 15:37:15 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] deleating volume v_153b4df8-1f37-4118-8aa6-1c155ea5700c {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 15:37:15 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] get snapshots for volume v_153b4df8-1f37-4118-8aa6-1c155ea5700c {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 15:37:15 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_153b4df8-1f37-4118-8aa6-1c155ea5700c/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:37:15 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_153b4df8-1f37-4118-8aa6-1c155ea5700c/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:37:15 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Volume reschedule parameters: True retry: {'num_attempts': 2, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 15:37:15 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Flow 'volume_create_manager' (a5e567c6-c297-4177-8116-27fb5330b229) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:37:15 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (85f5f5dc-ad23-4286-b30c-b4b3a55cf7ac) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:37:15 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (85f5f5dc-ad23-4286-b30c-b4b3a55cf7ac) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:34:57Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesCloneTest-Volume-2030679513',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=153b4df8-1f37-4118-8aa6-1c155ea5700c,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='82fce1724afa40dd914e0791c3ee69a3',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:37:15Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:37:15Z,use_quota=True,user_id='773ce6c2a0f844edaf666e190504e258',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:37:15 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (65c9e1e6-5853-421e-acca-4c4add9c4e18) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:37:15 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (65c9e1e6-5853-421e-acca-4c4add9c4e18) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:37:15 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (2660d8eb-518e-462d-9f51-c753ec8870b5) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:37:15 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (2660d8eb-518e-462d-9f51-c753ec8870b5) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': '153b4df8-1f37-4118-8aa6-1c155ea5700c', 'volume_name': 'volume-153b4df8-1f37-4118-8aa6-1c155ea5700c', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:37:15 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (206c7ce3-8060-411f-aa5f-d8852469c119) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:37:15 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (206c7ce3-8060-411f-aa5f-d8852469c119) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:37:15 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (19eb47bb-3590-413b-86ac-1f8b709369d4) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:37:15 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Volume 153b4df8-1f37-4118-8aa6-1c155ea5700c: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-153b4df8-1f37-4118-8aa6-1c155ea5700c', 'volume_size': 1} Apr 17 15:37:15 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] creating volume v_153b4df8-1f37-4118-8aa6-1c155ea5700c. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 15:37:15 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] create volume {'name': 'v_153b4df8-1f37-4118-8aa6-1c155ea5700c', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 15:37:15 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:37:15 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:37:18 user cinder-volume[84696]: DEBUG cinder.utils [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.408(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:37:18 user cinder-volume[84696]: DEBUG cinder.utils [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_29d9e55e-0b8b-4f70-b5c4-830d39f10fcd/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:37:18 user cinder-volume[84696]: DEBUG cinder.utils [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.233(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:37:18 user cinder-volume[84696]: DEBUG cinder.utils [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:37:18 user cinder-volume[84696]: DEBUG cinder.utils [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.963(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:37:18 user cinder-volume[84696]: DEBUG cinder.utils [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_153b4df8-1f37-4118-8aa6-1c155ea5700c/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:37:18 user cinder-volume[84696]: DEBUG cinder.utils [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.698(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:37:18 user cinder-volume[84696]: DEBUG cinder.utils [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:37:21 user cinder-volume[84696]: DEBUG cinder.utils [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.415(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:37:21 user cinder-volume[84696]: DEBUG cinder.utils [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_44073502-102d-452a-8244-69a8233b9060/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:37:21 user cinder-volume[84696]: DEBUG cinder.utils [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.765(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:37:21 user cinder-volume[84696]: DEBUG cinder.utils [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:37:21 user cinder-volume[84696]: DEBUG cinder.utils [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.232(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:37:21 user cinder-volume[84696]: DEBUG cinder.utils [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:37:21 user cinder-volume[84696]: DEBUG cinder.utils [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.034(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:37:21 user cinder-volume[84696]: DEBUG cinder.utils [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_153b4df8-1f37-4118-8aa6-1c155ea5700c/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:37:24 user cinder-volume[84696]: DEBUG cinder.utils [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 73.526(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:37:24 user cinder-volume[84696]: DEBUG cinder.utils [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_a4d96ab8-2072-4eff-b803-53b4f27d056f/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:37:24 user cinder-volume[84696]: DEBUG cinder.utils [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 73.288(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:37:24 user cinder-volume[84696]: DEBUG cinder.utils [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:37:24 user cinder-volume[84696]: DEBUG cinder.utils [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 73.516(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:37:24 user cinder-volume[84696]: DEBUG cinder.utils [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_44f909cf-7774-41f6-9f86-805456609ff8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:37:27 user cinder-volume[84696]: DEBUG cinder.utils [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 11.907(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:37:27 user cinder-volume[84696]: DEBUG cinder.utils [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:37:27 user cinder-volume[84696]: DEBUG cinder.utils [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.175(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:37:27 user cinder-volume[84696]: DEBUG cinder.utils [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_153b4df8-1f37-4118-8aa6-1c155ea5700c/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:37:33 user cinder-volume[84696]: DEBUG cinder.utils [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 73.796(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:37:33 user cinder-volume[84696]: DEBUG cinder.utils [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_0fed6b21-0520-4554-bee4-9540a03f9af0/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:37:33 user cinder-volume[84696]: DEBUG cinder.utils [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 73.463(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:37:33 user cinder-volume[84696]: DEBUG cinder.utils [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:37:34 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=21,cinder:UPDATE=7,cinder:INSERT=1 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:37:34 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=16,cinder:INSERT=4,cinder:UPDATE=5 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:37:36 user cinder-volume[84696]: DEBUG cinder.utils [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.823(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:37:36 user cinder-volume[84696]: DEBUG cinder.utils [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:37:36 user cinder-volume[84696]: DEBUG cinder.utils [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.285(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:37:36 user cinder-volume[84696]: DEBUG cinder.utils [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:37:36 user cinder-volume[84696]: DEBUG cinder.utils [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 40.003(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:37:36 user cinder-volume[84696]: DEBUG cinder.utils [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_29d9e55e-0b8b-4f70-b5c4-830d39f10fcd/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:37:36 user cinder-volume[84696]: DEBUG cinder.utils [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.554(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:37:36 user cinder-volume[84696]: DEBUG cinder.utils [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_153b4df8-1f37-4118-8aa6-1c155ea5700c/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:37:39 user cinder-volume[84696]: DEBUG cinder.utils [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 40.007(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:37:39 user cinder-volume[84696]: DEBUG cinder.utils [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_44073502-102d-452a-8244-69a8233b9060/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:37:39 user cinder-volume[84696]: DEBUG cinder.utils [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.823(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:37:39 user cinder-volume[84696]: DEBUG cinder.utils [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:37:46 user cinder-volume[84696]: DEBUG cinder.utils [None req-dbedb542-4536-4c1d-b79c-b9c0a6c64820 tempest-VolumesAdminNegativeTest-1285212149 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 73.436(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:37:46 user cinder-volume[84696]: DEBUG cinder.utils [None req-dbedb542-4536-4c1d-b79c-b9c0a6c64820 tempest-VolumesAdminNegativeTest-1285212149 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:215a6a5b-25ca-48fa-a8e3-72416d7a39e2/luns/v_215a6a5b-25ca-48fa-a8e3-72416d7a39e2 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:37:49 user cinder-volume[84696]: DEBUG cinder.utils [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 74.252(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:37:49 user cinder-volume[84696]: DEBUG cinder.utils [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:37:49 user cinder-volume[84696]: DEBUG cinder.utils [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 74.547(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:37:49 user cinder-volume[84696]: DEBUG cinder.utils [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_e80c78b2-20c9-4f9b-8898-6ab131d1eb06/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:37:52 user cinder-volume[84696]: DEBUG cinder.utils [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 37.379(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:37:52 user cinder-volume[84696]: DEBUG cinder.utils [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:37:52 user cinder-volume[84696]: DEBUG cinder.utils [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 37.647(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:37:52 user cinder-volume[84696]: DEBUG cinder.utils [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_153b4df8-1f37-4118-8aa6-1c155ea5700c/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:37:52 user cinder-volume[84696]: DEBUG cinder.utils [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 139.156(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:37:52 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_62709a81-268b-4227-ba4a-070034f36414/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:37:52 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (416c7a27-3bbf-4426-ba12-c01dcd02cce5) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 15:37:52 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (c50526d0-27c3-4cfd-9573-b6633f14c23f) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:37:52 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (c50526d0-27c3-4cfd-9573-b6633f14c23f) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:37:52 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Flow 'volume_create_manager' (e73c3c8f-9a3f-40bc-ae86-670b4fc8493c) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 15:37:52 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-e7b5301d-81ea-4ab5-9d69-766347595e11 tempest-VolumesActionsTest-707453245 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 15:37:52 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 15:37:52 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 15:37:52 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 15:37:52 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 15:37:52 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 15:37:52 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 15:37:52 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 15:37:52 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 15:37:52 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 15:37:52 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 15:37:52 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 15:37:52 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 15:37:52 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 15:37:52 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 15:37:52 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 15:37:52 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 15:37:52 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 15:37:52 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 15:37:52 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 15:37:52 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 15:37:52 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 15:37:52 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 15:37:52 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 15:37:52 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:37:52 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 15:37:52 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 15:37:52 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 15:37:52 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 15:37:52 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 15:37:52 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 15:37:52 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 15:37:52 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 15:37:52 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 15:37:52 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 15:37:52 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 15:37:52 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 15:37:52 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 15:37:52 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 15:37:52 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 15:37:52 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 15:37:52 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 15:37:54 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=3,cinder:UPDATE=2 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:37:55 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 139.256(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:37:55 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:37:55 user cinder-volume[84696]: ERROR oslo_service.periodic_task [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Error during VolumeManager.publish_service_capabilities: TypeError: 'NoneType' object is not subscriptable Apr 17 15:37:55 user cinder-volume[84696]: ERROR oslo_service.periodic_task Traceback (most recent call last): Apr 17 15:37:55 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/usr/local/lib/python3.10/dist-packages/oslo_service/periodic_task.py", line 216, in run_periodic_tasks Apr 17 15:37:55 user cinder-volume[84696]: ERROR oslo_service.periodic_task task(self, context) Apr 17 15:37:55 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/manager.py", line 2876, in publish_service_capabilities Apr 17 15:37:55 user cinder-volume[84696]: ERROR oslo_service.periodic_task self._report_driver_status(context) Apr 17 15:37:55 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/manager.py", line 2755, in _report_driver_status Apr 17 15:37:55 user cinder-volume[84696]: ERROR oslo_service.periodic_task volume_stats = get_stats() Apr 17 15:37:55 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/usr/local/lib/python3.10/dist-packages/oslo_utils/timeutils.py", line 319, in wrapper Apr 17 15:37:55 user cinder-volume[84696]: ERROR oslo_service.periodic_task result = func(*args, **kwargs) Apr 17 15:37:55 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/manager.py", line 2753, in get_stats Apr 17 15:37:55 user cinder-volume[84696]: ERROR oslo_service.periodic_task return self.driver.get_volume_stats(refresh=True) Apr 17 15:37:55 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/driver.py", line 666, in get_volume_stats Apr 17 15:37:55 user cinder-volume[84696]: ERROR oslo_service.periodic_task self._update_volume_stats() Apr 17 15:37:55 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 726, in _update_volume_stats Apr 17 15:37:55 user cinder-volume[84696]: ERROR oslo_service.periodic_task pool_stats = self.ra.get_pool_stats() Apr 17 15:37:55 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 897, in get_pool_stats Apr 17 15:37:55 user cinder-volume[84696]: ERROR oslo_service.periodic_task if not resp["error"] and resp["code"] == 200: Apr 17 15:37:55 user cinder-volume[84696]: ERROR oslo_service.periodic_task TypeError: 'NoneType' object is not subscriptable Apr 17 15:37:55 user cinder-volume[84696]: ERROR oslo_service.periodic_task Apr 17 15:37:55 user cinder-volume[84696]: WARNING oslo.service.loopingcall [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Function 'cinder.service.Service.periodic_tasks' run outlasted interval by 79.26 sec Apr 17 15:37:55 user cinder-volume[84696]: DEBUG oslo_service.periodic_task [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Running periodic task VolumeManager.publish_service_capabilities {{(pid=84696) run_periodic_tasks /usr/local/lib/python3.10/dist-packages/oslo_service/periodic_task.py:210}} Apr 17 15:37:55 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Updating volume stats {{(pid=84696) _update_volume_stats /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:724}} Apr 17 15:37:55 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Get pool Pool-0 fsprops {{(pid=84696) get_pool_stats /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:894}} Apr 17 15:37:56 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:37:56 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:37:59 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 3.066(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:37:59 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:38:02 user cinder-volume[84696]: DEBUG cinder.utils [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 139.096(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:38:02 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_17f8d4f7-2dcd-4041-8d0b-5c8afebb75bf/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:38:02 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (68a53ea2-b4bc-426e-bf76-3d7cdf44638d) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 15:38:02 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (24cc5e8e-cc38-409a-b469-bf86389089ce) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:38:02 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (24cc5e8e-cc38-409a-b469-bf86389089ce) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:38:02 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Flow 'volume_create_manager' (cc63b5b8-0ad3-4d49-9a4f-0f06a1267879) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 15:38:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-051b1fdc-54da-4587-a61f-deb7faa24c2c tempest-SnapshotMetadataTestJSON-118043004 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 15:38:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 15:38:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 15:38:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 15:38:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 15:38:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 15:38:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 15:38:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 15:38:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 15:38:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 15:38:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 15:38:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 15:38:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 15:38:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 15:38:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 15:38:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 15:38:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 15:38:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 15:38:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 15:38:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 15:38:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 15:38:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 15:38:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 15:38:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 15:38:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:38:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 15:38:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 15:38:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 15:38:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 15:38:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 15:38:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 15:38:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 15:38:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 15:38:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 15:38:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 15:38:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 15:38:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 15:38:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 15:38:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 15:38:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 15:38:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 15:38:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 15:38:02 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.155(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:38:02 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:38:02 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Volume reschedule parameters: True retry: {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 15:38:02 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Flow 'volume_create_manager' (f343c56c-018a-4dcb-8100-2d310c2bc132) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:38:02 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (b3b3ed96-55cb-4b63-b5cf-dbf33ab64d47) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:38:02 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (b3b3ed96-55cb-4b63-b5cf-dbf33ab64d47) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:38:02Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-CreateVolumesFromSourceVolumeTest-Volume-609446595',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=a50e9659-3e2d-4f04-b0ac-ae78b8f61fad,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='99747be727ae4592ae1b6176b488719a',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:38:02Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:38:02Z,use_quota=True,user_id='0696808ea0b14c0794e8664a37d9e2c3',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:38:02 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (d0b2a951-fe4a-461f-9929-7e538f0f603f) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:38:02 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (d0b2a951-fe4a-461f-9929-7e538f0f603f) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:38:02 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (4fc9a793-5b3b-41ea-ba6a-f192ca9c65b3) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:38:02 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (4fc9a793-5b3b-41ea-ba6a-f192ca9c65b3) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': 'a50e9659-3e2d-4f04-b0ac-ae78b8f61fad', 'volume_name': 'volume-a50e9659-3e2d-4f04-b0ac-ae78b8f61fad', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:38:02 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (d0ff0a55-c6f3-444b-9430-8f5388e67040) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:38:02 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (d0ff0a55-c6f3-444b-9430-8f5388e67040) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:38:02 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (519b4995-f47b-4d89-9fc2-7947903f2f57) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:38:02 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Volume a50e9659-3e2d-4f04-b0ac-ae78b8f61fad: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-a50e9659-3e2d-4f04-b0ac-ae78b8f61fad', 'volume_size': 1} Apr 17 15:38:02 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] creating volume v_a50e9659-3e2d-4f04-b0ac-ae78b8f61fad. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 15:38:02 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] create volume {'name': 'v_a50e9659-3e2d-4f04-b0ac-ae78b8f61fad', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 15:38:02 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:38:02 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:38:02 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Volume reschedule parameters: True retry: {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 15:38:02 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Flow 'volume_create_manager' (41dfb63e-f309-418c-ab2b-9c3f4f759af8) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:38:02 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (c4aa30a2-b694-4dcd-b60a-96ad11091b1c) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:38:02 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (c4aa30a2-b694-4dcd-b60a-96ad11091b1c) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:38:02Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesActionsTest-Volume-1957591186',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=57070f23-6b40-47b9-b9b4-e0d77240f544,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='a8b97ff6ed6047f0a33e74684a1c162b',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:38:02Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:38:02Z,use_quota=True,user_id='c993ceeb4a5c49eea14057e2125518ad',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:38:02 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (a7ae420e-549f-49e0-a4be-a45df8a475f7) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:38:02 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (a7ae420e-549f-49e0-a4be-a45df8a475f7) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:38:02 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (c7d58b83-b97e-4e10-95eb-6e530912e2bb) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:38:02 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (c7d58b83-b97e-4e10-95eb-6e530912e2bb) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': '57070f23-6b40-47b9-b9b4-e0d77240f544', 'volume_name': 'volume-57070f23-6b40-47b9-b9b4-e0d77240f544', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:38:02 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (4f845aa8-09d5-4f53-bc54-94a3254a609b) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:38:02 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (4f845aa8-09d5-4f53-bc54-94a3254a609b) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:38:02 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (d735d96c-06d6-4e96-9c51-afb975869610) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:38:02 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Volume 57070f23-6b40-47b9-b9b4-e0d77240f544: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-57070f23-6b40-47b9-b9b4-e0d77240f544', 'volume_size': 1} Apr 17 15:38:02 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] creating volume v_57070f23-6b40-47b9-b9b4-e0d77240f544. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 15:38:02 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] create volume {'name': 'v_57070f23-6b40-47b9-b9b4-e0d77240f544', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 15:38:02 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:38:02 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:38:05 user cinder-volume[84696]: DEBUG cinder.utils [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.906(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:38:05 user cinder-volume[84696]: DEBUG cinder.utils [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:38:05 user cinder-volume[84696]: DEBUG cinder.utils [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.783(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:38:05 user cinder-volume[84696]: DEBUG cinder.utils [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:38:08 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.282(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:38:08 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:38:08 user cinder-volume[84696]: DEBUG cinder.utils [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.854(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:38:08 user cinder-volume[84696]: DEBUG cinder.utils [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:38:08 user cinder-volume[84696]: DEBUG cinder.utils [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.979(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:38:08 user cinder-volume[84696]: DEBUG cinder.utils [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:38:11 user cinder-volume[84696]: DEBUG cinder.utils [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 139.175(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:38:11 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_8cc35ab1-363f-4ad6-9a37-239d2b1ea996/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:38:11 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (0d5a3677-e116-474e-95e4-84189a7a0841) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 15:38:11 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (f427bcb0-0446-4074-84df-e43cd57b3d83) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:38:11 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (f427bcb0-0446-4074-84df-e43cd57b3d83) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:38:11 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Flow 'volume_create_manager' (16942f14-554a-4c43-8393-9f00cb7324a1) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 15:38:11 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 15:38:11 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 15:38:11 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 15:38:11 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 15:38:11 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 15:38:11 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 15:38:11 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 15:38:11 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 15:38:11 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 15:38:11 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 15:38:11 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 15:38:11 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 15:38:11 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 15:38:11 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 15:38:11 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 15:38:11 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 15:38:11 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 15:38:11 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 15:38:11 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 15:38:11 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 15:38:11 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 15:38:11 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 15:38:11 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 15:38:11 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 15:38:11 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:38:11 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 15:38:11 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 15:38:11 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 15:38:11 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 15:38:11 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 15:38:11 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 15:38:11 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 15:38:11 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 15:38:11 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 15:38:11 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 15:38:11 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 15:38:11 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 15:38:11 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 15:38:11 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 15:38:11 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 15:38:11 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 15:38:11 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 15:38:11 user cinder-volume[84696]: DEBUG cinder.utils [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 74.546(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:38:11 user cinder-volume[84696]: DEBUG cinder.utils [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_29d9e55e-0b8b-4f70-b5c4-830d39f10fcd/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:38:11 user cinder-volume[84696]: DEBUG cinder.utils [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 139.026(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:38:11 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:38:11 user cinder-volume[84696]: INFO cinder.message.api [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Creating message record for request_id = req-822bc81b-4af7-4475-af70-370e1870fe50 Apr 17 15:38:11 user cinder-volume[84696]: DEBUG cinder.utils [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 74.373(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:38:11 user cinder-volume[84696]: DEBUG cinder.utils [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:38:11 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 15:38:11 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (40958e61-dda1-4e8e-858f-047b135f1e2b) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 15:38:11 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 15:38:11 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:33:31Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-ServersNegativeTestJSON-volume-1746588226',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=8cc35ab1-363f-4ad6-9a37-239d2b1ea996,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='d2dd48f279404859bebcc19bcc272915',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:35:52Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:35:52Z,use_quota=True,user_id='b0075b26d6044a50981d29df0d863959',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 15:38:11 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:33:31Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-ServersNegativeTestJSON-volume-1746588226',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=8cc35ab1-363f-4ad6-9a37-239d2b1ea996,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='d2dd48f279404859bebcc19bcc272915',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:35:52Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:35:52Z,use_quota=True,user_id='b0075b26d6044a50981d29df0d863959',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(8cc35ab1-363f-4ad6-9a37-239d2b1ea996),volume_id=8cc35ab1-363f-4ad6-9a37-239d2b1ea996,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': '8cc35ab1-363f-4ad6-9a37-239d2b1ea996', 'volume_name': 'volume-8cc35ab1-363f-4ad6-9a37-239d2b1ea996', 'volume_size': 1}} Apr 17 15:38:11 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': '8cc35ab1-363f-4ad6-9a37-239d2b1ea996', 'volume': {'id': '8cc35ab1-363f-4ad6-9a37-239d2b1ea996', '_name_id': None, 'ec2_id': None, 'user_id': 'b0075b26d6044a50981d29df0d863959', 'project_id': 'd2dd48f279404859bebcc19bcc272915', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-ServersNegativeTestJSON-volume-1746588226', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'admin_metadata': {}, 'glance_metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T15:33:30.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-8cc35ab1-363f-4ad6-9a37-239d2b1ea996', 'name_id': '8cc35ab1-363f-4ad6-9a37-239d2b1ea996', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-ServersNegativeTestJSON-volume-1746588226', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': 'd2dd48f279404859bebcc19bcc272915', 'qos_specs': None, 'reservations': ['1161f3e2-1648-485c-8f2d-e7d723e7cefb', 'b3435402-b7a6-4cc8-ada5-e3f921682979', '131f82e8-6bdb-4b90-85b1-3d76ab20e6b8', '5d42d0e7-f4e0-4d14-83c2-a0df94279daf'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'b0075b26d6044a50981d29df0d863959', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-ServersNegativeTestJSON-volume-1746588226', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': 'd2dd48f279404859bebcc19bcc272915', 'qos_specs': None, 'reservations': ['1161f3e2-1648-485c-8f2d-e7d723e7cefb', 'b3435402-b7a6-4cc8-ada5-e3f921682979', '131f82e8-6bdb-4b90-85b1-3d76ab20e6b8', '5d42d0e7-f4e0-4d14-83c2-a0df94279daf'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'b0075b26d6044a50981d29df0d863959', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': 'b0075b26d6044a50981d29df0d863959', 'metadata': {}, 'qos_specs': None, 'retry': {'num_attempts': 2, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']}}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(8cc35ab1-363f-4ad6-9a37-239d2b1ea996),volume_id=8cc35ab1-363f-4ad6-9a37-239d2b1ea996,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:33:31Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-ServersNegativeTestJSON-volume-1746588226',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=8cc35ab1-363f-4ad6-9a37-239d2b1ea996,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='d2dd48f279404859bebcc19bcc272915',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:35:52Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:35:52Z,use_quota=True,user_id='b0075b26d6044a50981d29df0d863959',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 15:38:11 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:33:31Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-ServersNegativeTestJSON-volume-1746588226',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=8cc35ab1-363f-4ad6-9a37-239d2b1ea996,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='d2dd48f279404859bebcc19bcc272915',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:35:52Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:35:52Z,use_quota=True,user_id='b0075b26d6044a50981d29df0d863959',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:33:31Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-ServersNegativeTestJSON-volume-1746588226',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=8cc35ab1-363f-4ad6-9a37-239d2b1ea996,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='d2dd48f279404859bebcc19bcc272915',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:35:52Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:35:52Z,use_quota=True,user_id='b0075b26d6044a50981d29df0d863959',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 15:38:11 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 15:38:11 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 15:38:11 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:38:11 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 15:38:11 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 15:38:11 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 15:38:11 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 15:38:11 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 15:38:11 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 15:38:11 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 15:38:11 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 15:38:11 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 15:38:11 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 15:38:11 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 15:38:11 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 15:38:11 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 15:38:11 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 15:38:11 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 15:38:11 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 15:38:11 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 15:38:11 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (40958e61-dda1-4e8e-858f-047b135f1e2b) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:38:11 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (40958e61-dda1-4e8e-858f-047b135f1e2b) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:38:11 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (778d0fe4-4c8f-4209-a406-e7fa04393194) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:38:11 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (778d0fe4-4c8f-4209-a406-e7fa04393194) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:38:11 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (60711e10-2763-486c-821b-597051575b6c) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:38:11 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (60711e10-2763-486c-821b-597051575b6c) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:38:11 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (c5b6afb4-5dd5-4a89-816d-25f24aac2e91) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:38:11 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Updating volume 8cc35ab1-363f-4ad6-9a37-239d2b1ea996 with {'scheduled_at': datetime.datetime(2023, 4, 17, 15, 38, 11, 437391), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 15:38:11 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Volume 8cc35ab1-363f-4ad6-9a37-239d2b1ea996: re-scheduling SchedulerAPI.create_volume attempt 2 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 15:38:11 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Volume 8cc35ab1-363f-4ad6-9a37-239d2b1ea996: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 15:38:11 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] deleating volume v_8cc35ab1-363f-4ad6-9a37-239d2b1ea996 {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 15:38:11 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] get snapshots for volume v_8cc35ab1-363f-4ad6-9a37-239d2b1ea996 {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 15:38:11 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_8cc35ab1-363f-4ad6-9a37-239d2b1ea996/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:38:11 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_8cc35ab1-363f-4ad6-9a37-239d2b1ea996/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:38:11 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Volume reschedule parameters: True retry: {'num_attempts': 3, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 15:38:11 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Flow 'volume_create_manager' (649c76c3-4271-423a-afd7-abe4e42d4197) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:38:11 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (bfbef947-a2ca-4dd5-83dc-38072220fdc8) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:38:11 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (bfbef947-a2ca-4dd5-83dc-38072220fdc8) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:33:31Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-ServersNegativeTestJSON-volume-1746588226',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=8cc35ab1-363f-4ad6-9a37-239d2b1ea996,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='d2dd48f279404859bebcc19bcc272915',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:38:12Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:38:12Z,use_quota=True,user_id='b0075b26d6044a50981d29df0d863959',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:38:11 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (5bd01a1e-e653-4328-bb49-1a979b1d0544) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:38:11 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (5bd01a1e-e653-4328-bb49-1a979b1d0544) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:38:11 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (bf5edbae-123c-41ce-ad05-6ba36ffbed39) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:38:11 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (bf5edbae-123c-41ce-ad05-6ba36ffbed39) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': '8cc35ab1-363f-4ad6-9a37-239d2b1ea996', 'volume_name': 'volume-8cc35ab1-363f-4ad6-9a37-239d2b1ea996', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:38:11 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (6f1f51c8-7da6-4308-8e0a-babb120e303d) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:38:11 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (6f1f51c8-7da6-4308-8e0a-babb120e303d) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:38:11 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (60237b8c-d20a-426d-9511-6fa0b66cde02) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:38:11 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Volume 8cc35ab1-363f-4ad6-9a37-239d2b1ea996: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-8cc35ab1-363f-4ad6-9a37-239d2b1ea996', 'volume_size': 1} Apr 17 15:38:11 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] creating volume v_8cc35ab1-363f-4ad6-9a37-239d2b1ea996. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 15:38:11 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] create volume {'name': 'v_8cc35ab1-363f-4ad6-9a37-239d2b1ea996', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 15:38:11 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:38:11 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:38:14 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=17,cinder:UPDATE=4,cinder:INSERT=1 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:38:14 user cinder-volume[84696]: DEBUG cinder.utils [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.752(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:38:14 user cinder-volume[84696]: DEBUG cinder.utils [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:38:14 user cinder-volume[84696]: DEBUG cinder.utils [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.965(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:38:14 user cinder-volume[84696]: DEBUG cinder.utils [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_8cc35ab1-363f-4ad6-9a37-239d2b1ea996/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:38:14 user cinder-volume[84696]: DEBUG cinder.utils [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 74.354(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:38:14 user cinder-volume[84696]: DEBUG cinder.utils [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:38:14 user cinder-volume[84696]: DEBUG cinder.utils [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.127(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:38:14 user cinder-volume[84696]: DEBUG cinder.utils [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:38:14 user cinder-volume[84696]: DEBUG cinder.utils [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.004(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:38:14 user cinder-volume[84696]: DEBUG cinder.utils [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:38:14 user cinder-volume[84696]: DEBUG cinder.utils [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 74.543(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:38:14 user cinder-volume[84696]: DEBUG cinder.utils [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_44073502-102d-452a-8244-69a8233b9060/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:38:16 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Volume reschedule parameters: True retry: {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 15:38:16 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Flow 'volume_create_manager' (aa44070f-2660-491a-8952-dbba9d00738f) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:38:16 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (66d5a1b6-9c8e-411d-a2ef-eafd6a5bb8b1) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:38:16 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (66d5a1b6-9c8e-411d-a2ef-eafd6a5bb8b1) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:38:17Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesSnapshotTestJSON-Volume-1408369126',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=7ce815ce-9603-4fd6-8cc8-9fd798b2f99d,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='aacddeb0a9ec4a0d8d832595382e1189',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:38:17Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:38:17Z,use_quota=True,user_id='fddc0a19102c4ee7920f657f195171c9',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:38:16 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (6b63b69f-daf6-4306-9c39-dcb57b647423) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:38:16 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (6b63b69f-daf6-4306-9c39-dcb57b647423) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:38:16 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (8d0ff060-f3a7-4bb1-ba43-2fd884642faa) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:38:16 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (8d0ff060-f3a7-4bb1-ba43-2fd884642faa) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': '7ce815ce-9603-4fd6-8cc8-9fd798b2f99d', 'volume_name': 'volume-7ce815ce-9603-4fd6-8cc8-9fd798b2f99d', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:38:16 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (f6e9574c-c118-4aaa-a5fb-f10ae4e68fc7) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:38:16 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (f6e9574c-c118-4aaa-a5fb-f10ae4e68fc7) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:38:16 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (83d63a53-b36b-4517-ad48-a188de963f10) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:38:16 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Volume 7ce815ce-9603-4fd6-8cc8-9fd798b2f99d: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-7ce815ce-9603-4fd6-8cc8-9fd798b2f99d', 'volume_size': 1} Apr 17 15:38:16 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] creating volume v_7ce815ce-9603-4fd6-8cc8-9fd798b2f99d. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 15:38:16 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] create volume {'name': 'v_7ce815ce-9603-4fd6-8cc8-9fd798b2f99d', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 15:38:16 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:38:16 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:38:17 user cinder-volume[84696]: DEBUG cinder.utils [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 138.884(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:38:17 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:38:17 user cinder-volume[84696]: INFO cinder.message.api [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Creating message record for request_id = req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 Apr 17 15:38:17 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.504(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:38:17 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:38:17 user cinder-volume[84696]: DEBUG cinder.utils [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.830(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:38:17 user cinder-volume[84696]: DEBUG cinder.utils [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:38:17 user cinder-volume[84696]: DEBUG cinder.utils [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 139.157(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:38:17 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_5d9c86cd-3e22-4926-bfc9-e5a37f6ae76c/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:38:17 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (bab63c23-4055-4aca-85fa-c39397afb88d) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 15:38:17 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (40e3868e-0e66-4d73-aa2b-c28b4c5350b0) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:38:17 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (40e3868e-0e66-4d73-aa2b-c28b4c5350b0) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:38:17 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Flow 'volume_create_manager' (9a244f79-c148-4050-bb04-c14a9784aa3c) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 15:38:17 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 15:38:17 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 15:38:17 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 15:38:17 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 15:38:17 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 15:38:17 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 15:38:17 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 15:38:17 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 15:38:17 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 15:38:17 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 15:38:17 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 15:38:17 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 15:38:17 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 15:38:17 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 15:38:17 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 15:38:17 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 15:38:17 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 15:38:17 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 15:38:17 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 15:38:17 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 15:38:17 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 15:38:17 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 15:38:17 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 15:38:17 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 15:38:17 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:38:17 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 15:38:17 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 15:38:17 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 15:38:17 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 15:38:17 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 15:38:17 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 15:38:17 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 15:38:17 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 15:38:17 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 15:38:17 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 15:38:17 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 15:38:17 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 15:38:17 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 15:38:17 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 15:38:17 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 15:38:17 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 15:38:17 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 15:38:17 user cinder-volume[84696]: DEBUG cinder.utils [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 0.581(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:38:17 user cinder-volume[84696]: DEBUG cinder.utils [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:38:17 user cinder-volume[84696]: DEBUG cinder.utils [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.061(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:38:17 user cinder-volume[84696]: DEBUG cinder.utils [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_8cc35ab1-363f-4ad6-9a37-239d2b1ea996/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:38:17 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 15:38:17 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (4e33be93-f26b-41f2-aa05-c7fee5b48dfd) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 15:38:17 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 15:38:17 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:31:16Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesServicesTestJSON-Volume-968297362',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=5d9c86cd-3e22-4926-bfc9-e5a37f6ae76c,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='6cf93c7193f842d58d323e62e508a5eb',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:35:59Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:35:59Z,use_quota=True,user_id='225768ea457e484f8790d273f178b879',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 15:38:17 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:31:16Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesServicesTestJSON-Volume-968297362',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=5d9c86cd-3e22-4926-bfc9-e5a37f6ae76c,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='6cf93c7193f842d58d323e62e508a5eb',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:35:59Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:35:59Z,use_quota=True,user_id='225768ea457e484f8790d273f178b879',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(5d9c86cd-3e22-4926-bfc9-e5a37f6ae76c),volume_id=5d9c86cd-3e22-4926-bfc9-e5a37f6ae76c,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': '5d9c86cd-3e22-4926-bfc9-e5a37f6ae76c', 'volume_name': 'volume-5d9c86cd-3e22-4926-bfc9-e5a37f6ae76c', 'volume_size': 1}} Apr 17 15:38:17 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': '5d9c86cd-3e22-4926-bfc9-e5a37f6ae76c', 'volume': {'id': '5d9c86cd-3e22-4926-bfc9-e5a37f6ae76c', '_name_id': None, 'ec2_id': None, 'user_id': '225768ea457e484f8790d273f178b879', 'project_id': '6cf93c7193f842d58d323e62e508a5eb', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-VolumesServicesTestJSON-Volume-968297362', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'admin_metadata': {}, 'glance_metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T15:31:15.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-5d9c86cd-3e22-4926-bfc9-e5a37f6ae76c', 'name_id': '5d9c86cd-3e22-4926-bfc9-e5a37f6ae76c', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumesServicesTestJSON-Volume-968297362', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '6cf93c7193f842d58d323e62e508a5eb', 'qos_specs': None, 'reservations': ['ad0fd489-3554-44ca-a47d-2b78caa30877', 'ee3d5634-17e5-4946-81db-b7539ccc1b48', '991d4c78-00b3-4a69-98e2-174c64fa2675', '86cc929c-76c2-4c7f-a486-8495ac6da9a2'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': '225768ea457e484f8790d273f178b879', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumesServicesTestJSON-Volume-968297362', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '6cf93c7193f842d58d323e62e508a5eb', 'qos_specs': None, 'reservations': ['ad0fd489-3554-44ca-a47d-2b78caa30877', 'ee3d5634-17e5-4946-81db-b7539ccc1b48', '991d4c78-00b3-4a69-98e2-174c64fa2675', '86cc929c-76c2-4c7f-a486-8495ac6da9a2'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': '225768ea457e484f8790d273f178b879', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': '225768ea457e484f8790d273f178b879', 'metadata': {}, 'qos_specs': None, 'retry': {'num_attempts': 3, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']}}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(5d9c86cd-3e22-4926-bfc9-e5a37f6ae76c),volume_id=5d9c86cd-3e22-4926-bfc9-e5a37f6ae76c,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:31:16Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesServicesTestJSON-Volume-968297362',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=5d9c86cd-3e22-4926-bfc9-e5a37f6ae76c,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='6cf93c7193f842d58d323e62e508a5eb',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:35:59Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:35:59Z,use_quota=True,user_id='225768ea457e484f8790d273f178b879',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 15:38:17 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:31:16Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesServicesTestJSON-Volume-968297362',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=5d9c86cd-3e22-4926-bfc9-e5a37f6ae76c,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='6cf93c7193f842d58d323e62e508a5eb',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:35:59Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:35:59Z,use_quota=True,user_id='225768ea457e484f8790d273f178b879',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:31:16Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesServicesTestJSON-Volume-968297362',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=5d9c86cd-3e22-4926-bfc9-e5a37f6ae76c,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='6cf93c7193f842d58d323e62e508a5eb',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:35:59Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:35:59Z,use_quota=True,user_id='225768ea457e484f8790d273f178b879',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 15:38:17 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 15:38:17 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 15:38:17 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:38:17 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 15:38:17 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 15:38:17 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 15:38:17 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 15:38:17 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 15:38:17 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 15:38:17 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 15:38:17 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 15:38:17 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 15:38:17 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 15:38:17 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 15:38:17 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 15:38:17 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 15:38:17 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 15:38:17 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 15:38:17 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 15:38:17 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 15:38:17 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (4e33be93-f26b-41f2-aa05-c7fee5b48dfd) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:38:17 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (4e33be93-f26b-41f2-aa05-c7fee5b48dfd) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:38:17 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (81b3d3e2-9250-4276-b1d9-4a599db0b361) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:38:17 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (81b3d3e2-9250-4276-b1d9-4a599db0b361) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:38:17 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (e8f61d79-7eed-4da9-b850-506b5b1f71c0) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:38:17 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (e8f61d79-7eed-4da9-b850-506b5b1f71c0) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:38:17 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (b04285b7-aa2c-4e25-91ee-7eaa61ab5ad3) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:38:17 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Updating volume 5d9c86cd-3e22-4926-bfc9-e5a37f6ae76c with {'scheduled_at': datetime.datetime(2023, 4, 17, 15, 38, 17, 581430), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 15:38:17 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Volume 5d9c86cd-3e22-4926-bfc9-e5a37f6ae76c: re-scheduling SchedulerAPI.create_volume attempt 3 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 15:38:17 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Volume 5d9c86cd-3e22-4926-bfc9-e5a37f6ae76c: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 15:38:17 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] deleating volume v_5d9c86cd-3e22-4926-bfc9-e5a37f6ae76c {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 15:38:17 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] get snapshots for volume v_5d9c86cd-3e22-4926-bfc9-e5a37f6ae76c {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 15:38:17 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_5d9c86cd-3e22-4926-bfc9-e5a37f6ae76c/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:38:17 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_5d9c86cd-3e22-4926-bfc9-e5a37f6ae76c/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:38:20 user cinder-volume[84696]: DEBUG cinder.utils [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.949(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:38:20 user cinder-volume[84696]: DEBUG cinder.utils [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_5d9c86cd-3e22-4926-bfc9-e5a37f6ae76c/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:38:20 user cinder-volume[84696]: DEBUG cinder.utils [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 3.629(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:38:20 user cinder-volume[84696]: DEBUG cinder.utils [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:38:23 user cinder-volume[84696]: DEBUG cinder.utils [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 11.968(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:38:23 user cinder-volume[84696]: DEBUG cinder.utils [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:38:23 user cinder-volume[84696]: DEBUG cinder.utils [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.022(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:38:23 user cinder-volume[84696]: DEBUG cinder.utils [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_5d9c86cd-3e22-4926-bfc9-e5a37f6ae76c/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:38:23 user cinder-volume[84696]: DEBUG cinder.utils [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.216(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:38:23 user cinder-volume[84696]: DEBUG cinder.utils [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:38:23 user cinder-volume[84696]: DEBUG cinder.utils [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.341(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:38:23 user cinder-volume[84696]: DEBUG cinder.utils [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:38:23 user cinder-volume[84696]: DEBUG cinder.utils [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.182(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:38:23 user cinder-volume[84696]: DEBUG cinder.utils [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_8cc35ab1-363f-4ad6-9a37-239d2b1ea996/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:38:26 user cinder-volume[84696]: DEBUG cinder.utils [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 71.172(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:38:26 user cinder-volume[84696]: DEBUG cinder.utils [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:38:26 user cinder-volume[84696]: DEBUG cinder.utils [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 9.772(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:38:26 user cinder-volume[84696]: DEBUG cinder.utils [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:38:26 user cinder-volume[84696]: DEBUG cinder.utils [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 71.442(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:38:26 user cinder-volume[84696]: DEBUG cinder.utils [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_153b4df8-1f37-4118-8aa6-1c155ea5700c/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:38:29 user cinder-volume[84696]: DEBUG cinder.utils [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.173(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:38:29 user cinder-volume[84696]: DEBUG cinder.utils [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_5d9c86cd-3e22-4926-bfc9-e5a37f6ae76c/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:38:29 user cinder-volume[84696]: DEBUG cinder.utils [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 138.701(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:38:29 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:38:29 user cinder-volume[84696]: INFO cinder.message.api [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Creating message record for request_id = req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df Apr 17 15:38:29 user cinder-volume[84696]: DEBUG cinder.utils [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 138.932(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:38:29 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_44f909cf-7774-41f6-9f86-805456609ff8/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:38:29 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (e4df08b7-3778-4055-9368-a4b9d604e358) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 15:38:29 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (9c5b4cb2-aa1c-4d03-af48-60f8ec135096) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:38:29 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (9c5b4cb2-aa1c-4d03-af48-60f8ec135096) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:38:29 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Flow 'volume_create_manager' (04029fc4-38ce-4344-87e4-a6a557fe365d) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 15:38:29 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 15:38:29 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 15:38:29 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 15:38:29 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 15:38:29 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 15:38:29 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 15:38:29 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 15:38:29 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 15:38:29 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 15:38:29 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 15:38:29 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 15:38:29 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 15:38:29 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 15:38:29 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 15:38:29 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 15:38:29 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 15:38:29 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 15:38:29 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 15:38:29 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 15:38:29 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 15:38:29 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 15:38:29 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 15:38:29 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 15:38:29 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 15:38:29 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:38:29 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 15:38:29 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 15:38:29 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 15:38:29 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 15:38:29 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 15:38:29 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 15:38:29 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 15:38:29 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 15:38:29 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 15:38:29 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 15:38:29 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 15:38:29 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 15:38:29 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 15:38:29 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 15:38:29 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 15:38:29 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 15:38:29 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 15:38:29 user cinder-volume[84696]: DEBUG cinder.utils [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 138.966(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:38:29 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_a4d96ab8-2072-4eff-b803-53b4f27d056f/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:38:29 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (3998f8e0-3ba1-4802-8f74-76b867b943f7) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 15:38:29 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (a3ffb6ea-004c-40fb-81a9-76ef1088252c) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:38:29 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (a3ffb6ea-004c-40fb-81a9-76ef1088252c) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:38:29 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Flow 'volume_create_manager' (120a1031-6402-4316-aa77-94af172d19ce) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 15:38:29 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-6e1f5cff-98a0-49d3-b32d-29a667939689 tempest-VolumesAdminNegativeTest-1285212149 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 15:38:29 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 15:38:29 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 15:38:29 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 15:38:29 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 15:38:29 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 15:38:29 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 15:38:29 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 15:38:29 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 15:38:29 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 15:38:29 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 15:38:29 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 15:38:29 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 15:38:29 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 15:38:29 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 15:38:29 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 15:38:29 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 15:38:29 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 15:38:29 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 15:38:29 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 15:38:29 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 15:38:29 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 15:38:29 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 15:38:29 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 15:38:29 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:38:29 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 15:38:29 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 15:38:29 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 15:38:29 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 15:38:29 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 15:38:29 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 15:38:29 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 15:38:29 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 15:38:29 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 15:38:29 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 15:38:29 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 15:38:29 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 15:38:29 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 15:38:29 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 15:38:29 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 15:38:29 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 15:38:29 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 15:38:29 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 15:38:29 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=22,cinder:UPDATE=5,cinder:INSERT=2 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:38:29 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (b391e404-ecbf-4536-85c7-738e3109e043) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 15:38:29 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 15:38:29 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:31:28Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesActionsTest-Volume-2081546358',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=44f909cf-7774-41f6-9f86-805456609ff8,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='a8b97ff6ed6047f0a33e74684a1c162b',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:36:11Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:36:11Z,use_quota=True,user_id='c993ceeb4a5c49eea14057e2125518ad',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 15:38:29 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:31:28Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesActionsTest-Volume-2081546358',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=44f909cf-7774-41f6-9f86-805456609ff8,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='a8b97ff6ed6047f0a33e74684a1c162b',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:36:11Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:36:11Z,use_quota=True,user_id='c993ceeb4a5c49eea14057e2125518ad',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(44f909cf-7774-41f6-9f86-805456609ff8),volume_id=44f909cf-7774-41f6-9f86-805456609ff8,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': '44f909cf-7774-41f6-9f86-805456609ff8', 'volume_name': 'volume-44f909cf-7774-41f6-9f86-805456609ff8', 'volume_size': 1}} Apr 17 15:38:29 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': '44f909cf-7774-41f6-9f86-805456609ff8', 'volume': {'id': '44f909cf-7774-41f6-9f86-805456609ff8', '_name_id': None, 'ec2_id': None, 'user_id': 'c993ceeb4a5c49eea14057e2125518ad', 'project_id': 'a8b97ff6ed6047f0a33e74684a1c162b', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-VolumesActionsTest-Volume-2081546358', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'admin_metadata': {}, 'glance_metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T15:31:28.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-44f909cf-7774-41f6-9f86-805456609ff8', 'name_id': '44f909cf-7774-41f6-9f86-805456609ff8', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumesActionsTest-Volume-2081546358', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': 'a8b97ff6ed6047f0a33e74684a1c162b', 'qos_specs': None, 'reservations': ['85f3837d-d106-4fab-b4df-d6deba5e2524', 'a6a77e64-4343-4b2d-a087-e1d5e603fbe7', '46947bf4-8977-46e0-91f8-7f7221eb5f15', '82ecd437-b8ec-41ab-ad06-ff748c2a9c71'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'c993ceeb4a5c49eea14057e2125518ad', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumesActionsTest-Volume-2081546358', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': 'a8b97ff6ed6047f0a33e74684a1c162b', 'qos_specs': None, 'reservations': ['85f3837d-d106-4fab-b4df-d6deba5e2524', 'a6a77e64-4343-4b2d-a087-e1d5e603fbe7', '46947bf4-8977-46e0-91f8-7f7221eb5f15', '82ecd437-b8ec-41ab-ad06-ff748c2a9c71'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'c993ceeb4a5c49eea14057e2125518ad', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': 'c993ceeb4a5c49eea14057e2125518ad', 'metadata': {}, 'qos_specs': None, 'retry': {'num_attempts': 3, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']}}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(44f909cf-7774-41f6-9f86-805456609ff8),volume_id=44f909cf-7774-41f6-9f86-805456609ff8,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:31:28Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesActionsTest-Volume-2081546358',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=44f909cf-7774-41f6-9f86-805456609ff8,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='a8b97ff6ed6047f0a33e74684a1c162b',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:36:11Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:36:11Z,use_quota=True,user_id='c993ceeb4a5c49eea14057e2125518ad',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 15:38:29 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:31:28Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesActionsTest-Volume-2081546358',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=44f909cf-7774-41f6-9f86-805456609ff8,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='a8b97ff6ed6047f0a33e74684a1c162b',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:36:11Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:36:11Z,use_quota=True,user_id='c993ceeb4a5c49eea14057e2125518ad',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:31:28Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesActionsTest-Volume-2081546358',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=44f909cf-7774-41f6-9f86-805456609ff8,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='a8b97ff6ed6047f0a33e74684a1c162b',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:36:11Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:36:11Z,use_quota=True,user_id='c993ceeb4a5c49eea14057e2125518ad',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 15:38:29 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 15:38:29 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 15:38:29 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:38:29 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 15:38:29 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 15:38:29 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 15:38:29 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 15:38:29 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 15:38:29 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 15:38:29 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 15:38:29 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 15:38:29 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 15:38:29 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 15:38:29 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 15:38:29 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 15:38:29 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 15:38:29 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 15:38:29 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 15:38:29 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 15:38:29 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 15:38:29 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (b391e404-ecbf-4536-85c7-738e3109e043) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:38:29 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (b391e404-ecbf-4536-85c7-738e3109e043) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:38:29 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (6b4d6194-64a0-455e-8780-79eb13cfd925) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:38:29 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (6b4d6194-64a0-455e-8780-79eb13cfd925) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:38:29 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (9ffe98ef-6fc5-4654-9da2-bf6ae717aa79) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:38:29 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (9ffe98ef-6fc5-4654-9da2-bf6ae717aa79) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:38:29 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (ad21cf54-42e6-4d2d-bbc8-2aa4dc0604d7) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:38:29 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Updating volume 44f909cf-7774-41f6-9f86-805456609ff8 with {'scheduled_at': datetime.datetime(2023, 4, 17, 15, 38, 29, 906136), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 15:38:29 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Volume 44f909cf-7774-41f6-9f86-805456609ff8: re-scheduling SchedulerAPI.create_volume attempt 3 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 15:38:29 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Volume 44f909cf-7774-41f6-9f86-805456609ff8: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 15:38:29 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] deleating volume v_44f909cf-7774-41f6-9f86-805456609ff8 {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 15:38:29 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] get snapshots for volume v_44f909cf-7774-41f6-9f86-805456609ff8 {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 15:38:29 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_44f909cf-7774-41f6-9f86-805456609ff8/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:38:29 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_44f909cf-7774-41f6-9f86-805456609ff8/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:38:32 user cinder-volume[84696]: DEBUG cinder.utils [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.925(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:38:32 user cinder-volume[84696]: DEBUG cinder.utils [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_44f909cf-7774-41f6-9f86-805456609ff8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:38:32 user cinder-volume[84696]: DEBUG cinder.utils [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.396(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:38:32 user cinder-volume[84696]: DEBUG cinder.utils [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_8cc35ab1-363f-4ad6-9a37-239d2b1ea996/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:38:32 user cinder-volume[84696]: DEBUG cinder.utils [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.188(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:38:32 user cinder-volume[84696]: DEBUG cinder.utils [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:38:35 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.930(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:38:35 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:38:35 user cinder-volume[84696]: DEBUG cinder.utils [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 18.987(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:38:35 user cinder-volume[84696]: DEBUG cinder.utils [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:38:35 user cinder-volume[84696]: DEBUG cinder.utils [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.000(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:38:35 user cinder-volume[84696]: DEBUG cinder.utils [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_44f909cf-7774-41f6-9f86-805456609ff8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:38:39 user cinder-volume[84696]: DEBUG cinder.utils [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 139.044(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:38:39 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_0fed6b21-0520-4554-bee4-9540a03f9af0/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:38:39 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (3ce8ae22-70a5-4d60-94e6-f9407d0cd7fd) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 15:38:39 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (2c43f562-4ae1-47c3-bf56-e2d22afd7ee1) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:38:39 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (2c43f562-4ae1-47c3-bf56-e2d22afd7ee1) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:38:39 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Flow 'volume_create_manager' (455eb968-f753-40eb-a170-f452ecb3538e) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 15:38:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 15:38:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 15:38:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 15:38:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 15:38:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 15:38:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 15:38:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 15:38:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 15:38:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 15:38:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 15:38:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 15:38:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 15:38:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 15:38:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 15:38:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 15:38:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 15:38:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 15:38:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 15:38:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 15:38:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 15:38:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 15:38:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 15:38:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 15:38:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 15:38:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:38:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 15:38:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1237, in execute Apr 17 15:38:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_from_image(context, Apr 17 15:38:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/utils.py", line 670, in _wrapper Apr 17 15:38:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return r(f, *args, **kwargs) Apr 17 15:38:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/tenacity/__init__.py", line 423, in __call__ Apr 17 15:38:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server do = self.iter(retry_state=retry_state) Apr 17 15:38:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/tenacity/__init__.py", line 360, in iter Apr 17 15:38:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return fut.result() Apr 17 15:38:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/lib/python3.10/concurrent/futures/_base.py", line 451, in result Apr 17 15:38:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self.__get_result() Apr 17 15:38:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result Apr 17 15:38:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self._exception Apr 17 15:38:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/tenacity/__init__.py", line 426, in __call__ Apr 17 15:38:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = fn(*args, **kwargs) Apr 17 15:38:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1125, in _create_from_image Apr 17 15:38:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_from_image_cache_or_download( Apr 17 15:38:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 944, in _create_from_image_cache_or_download Apr 17 15:38:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update, cloned = self._create_from_image_cache( Apr 17 15:38:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 843, in _create_from_image_cache Apr 17 15:38:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_from_source_volume( Apr 17 15:38:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 659, in _create_from_source_volume Apr 17 15:38:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self.driver.create_cloned_volume(volume, srcvol_ref) Apr 17 15:38:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 525, in create_cloned_volume Apr 17 15:38:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self._clone_object(vname, cvname) Apr 17 15:38:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 461, in _clone_object Apr 17 15:38:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_snapshot(oname, coname) Apr 17 15:38:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 639, in create_snapshot Apr 17 15:38:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201, 204): Apr 17 15:38:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 15:38:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 15:38:39 user cinder-volume[84696]: DEBUG cinder.utils [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 138.728(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:38:39 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:38:39 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (d344f7f8-223e-4527-860d-34854bf63518) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 15:38:39 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 15:38:39 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:31:42Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesCloneTest-Volume-297264340',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=0fed6b21-0520-4554-bee4-9540a03f9af0,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='82fce1724afa40dd914e0791c3ee69a3',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:36:20Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:36:20Z,use_quota=True,user_id='773ce6c2a0f844edaf666e190504e258',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 15:38:39 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:31:42Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesCloneTest-Volume-297264340',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=0fed6b21-0520-4554-bee4-9540a03f9af0,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='82fce1724afa40dd914e0791c3ee69a3',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:36:20Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:36:20Z,use_quota=True,user_id='773ce6c2a0f844edaf666e190504e258',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=b3f0dec1-7758-44a8-82e0-76ada8837428,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(0fed6b21-0520-4554-bee4-9540a03f9af0),volume_id=0fed6b21-0520-4554-bee4-9540a03f9af0,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'image', 'volume_id': '0fed6b21-0520-4554-bee4-9540a03f9af0', 'volume_name': 'volume-0fed6b21-0520-4554-bee4-9540a03f9af0', 'volume_size': 1, 'image_id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'image_location': (None, None), 'image_meta': {'name': 'cirros-0.5.2-x86_64-disk', 'disk_format': 'qcow2', 'container_format': 'bare', 'visibility': 'public', 'size': 16300544, 'virtual_size': 117440512, 'status': 'active', 'checksum': 'b874c39491a2377b8490f5f1e89761a4', 'protected': False, 'min_ram': 0, 'min_disk': 0, 'owner': 'a43c211b4ed542538799349489b1d884', 'os_hidden': False, 'os_hash_algo': 'sha512', 'os_hash_value': '6b813aa46bb90b4da216a4d19376593fa3f4fc7e617f03a92b7fe11e9a3981cbe8f0959dbebe36225e5f53dc4492341a4863cac4ed1ee0909f3fc78ef9c3e869', 'id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'created_at': datetime.datetime(2023, 4, 17, 15, 19, 53, tzinfo=datetime.timezone.utc), 'updated_at': datetime.datetime(2023, 4, 17, 15, 19, 55, tzinfo=datetime.timezone.utc), 'tags': [], 'file': '/v2/images/b3f0dec1-7758-44a8-82e0-76ada8837428/file', 'properties': {'hw_rng_model': 'virtio', 'owner_specified.openstack.md5': '', 'owner_specified.openstack.object': 'images/cirros-0.5.2-x86_64-disk', 'owner_specified.openstack.sha256': ''}}, 'image_service': }} Apr 17 15:38:39 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'snapshot_id': None, 'source_volid': None, 'volume_id': '0fed6b21-0520-4554-bee4-9540a03f9af0', 'volume': {'id': '0fed6b21-0520-4554-bee4-9540a03f9af0', '_name_id': None, 'ec2_id': None, 'user_id': '773ce6c2a0f844edaf666e190504e258', 'project_id': '82fce1724afa40dd914e0791c3ee69a3', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-VolumesCloneTest-Volume-297264340', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'admin_metadata': {}, 'glance_metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T15:31:41.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-0fed6b21-0520-4554-bee4-9540a03f9af0', 'name_id': '0fed6b21-0520-4554-bee4-9540a03f9af0', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumesCloneTest-Volume-297264340', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '82fce1724afa40dd914e0791c3ee69a3', 'qos_specs': None, 'reservations': ['1379a6d8-9976-4d94-a18a-ef0001e5de6b', 'ebe18ce3-cd83-4706-9c9a-b20a8ec73b8f', '3321e2fc-3bcb-4c90-bfc9-3a5a122ad7a3', '0805cd4d-24d4-47d8-b230-5b3c0a1d66aa'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': '773ce6c2a0f844edaf666e190504e258', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumesCloneTest-Volume-297264340', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '82fce1724afa40dd914e0791c3ee69a3', 'qos_specs': None, 'reservations': ['1379a6d8-9976-4d94-a18a-ef0001e5de6b', 'ebe18ce3-cd83-4706-9c9a-b20a8ec73b8f', '3321e2fc-3bcb-4c90-bfc9-3a5a122ad7a3', '0805cd4d-24d4-47d8-b230-5b3c0a1d66aa'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': '773ce6c2a0f844edaf666e190504e258', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': '773ce6c2a0f844edaf666e190504e258', 'metadata': {}, 'qos_specs': None, 'retry': {'num_attempts': 3, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']}}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=b3f0dec1-7758-44a8-82e0-76ada8837428,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(0fed6b21-0520-4554-bee4-9540a03f9af0),volume_id=0fed6b21-0520-4554-bee4-9540a03f9af0,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:31:42Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesCloneTest-Volume-297264340',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=0fed6b21-0520-4554-bee4-9540a03f9af0,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='82fce1724afa40dd914e0791c3ee69a3',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:36:20Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:36:20Z,use_quota=True,user_id='773ce6c2a0f844edaf666e190504e258',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 15:38:39 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:31:42Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesCloneTest-Volume-297264340',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=0fed6b21-0520-4554-bee4-9540a03f9af0,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='82fce1724afa40dd914e0791c3ee69a3',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:36:20Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:36:20Z,use_quota=True,user_id='773ce6c2a0f844edaf666e190504e258',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:31:42Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesCloneTest-Volume-297264340',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=0fed6b21-0520-4554-bee4-9540a03f9af0,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='82fce1724afa40dd914e0791c3ee69a3',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:36:20Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:36:20Z,use_quota=True,user_id='773ce6c2a0f844edaf666e190504e258',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 15:38:39 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 15:38:39 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 15:38:39 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:38:39 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 15:38:39 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1237, in execute Apr 17 15:38:39 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_from_image(context, Apr 17 15:38:39 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/utils.py", line 670, in _wrapper Apr 17 15:38:39 user cinder-volume[84696]: ERROR cinder.volume.manager return r(f, *args, **kwargs) Apr 17 15:38:39 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/tenacity/__init__.py", line 423, in __call__ Apr 17 15:38:39 user cinder-volume[84696]: ERROR cinder.volume.manager do = self.iter(retry_state=retry_state) Apr 17 15:38:39 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/tenacity/__init__.py", line 360, in iter Apr 17 15:38:39 user cinder-volume[84696]: ERROR cinder.volume.manager return fut.result() Apr 17 15:38:39 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/lib/python3.10/concurrent/futures/_base.py", line 451, in result Apr 17 15:38:39 user cinder-volume[84696]: ERROR cinder.volume.manager return self.__get_result() Apr 17 15:38:39 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result Apr 17 15:38:39 user cinder-volume[84696]: ERROR cinder.volume.manager raise self._exception Apr 17 15:38:39 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/tenacity/__init__.py", line 426, in __call__ Apr 17 15:38:39 user cinder-volume[84696]: ERROR cinder.volume.manager result = fn(*args, **kwargs) Apr 17 15:38:39 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1125, in _create_from_image Apr 17 15:38:39 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_from_image_cache_or_download( Apr 17 15:38:39 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 944, in _create_from_image_cache_or_download Apr 17 15:38:39 user cinder-volume[84696]: ERROR cinder.volume.manager model_update, cloned = self._create_from_image_cache( Apr 17 15:38:39 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 843, in _create_from_image_cache Apr 17 15:38:39 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_from_source_volume( Apr 17 15:38:39 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 659, in _create_from_source_volume Apr 17 15:38:39 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self.driver.create_cloned_volume(volume, srcvol_ref) Apr 17 15:38:39 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 525, in create_cloned_volume Apr 17 15:38:39 user cinder-volume[84696]: ERROR cinder.volume.manager self._clone_object(vname, cvname) Apr 17 15:38:39 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 461, in _clone_object Apr 17 15:38:39 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_snapshot(oname, coname) Apr 17 15:38:39 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 639, in create_snapshot Apr 17 15:38:39 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201, 204): Apr 17 15:38:39 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 15:38:39 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 15:38:39 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (d344f7f8-223e-4527-860d-34854bf63518) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:38:39 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (d344f7f8-223e-4527-860d-34854bf63518) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:38:39 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (545df23a-d06f-4506-b87f-847ec7576a67) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:38:39 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (545df23a-d06f-4506-b87f-847ec7576a67) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:38:39 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (04dcc085-dbfb-4dc0-83a7-4a5794cfbf96) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:38:39 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (04dcc085-dbfb-4dc0-83a7-4a5794cfbf96) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:38:39 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (08b4e6dc-1ce7-49f6-b5aa-72522a91a536) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:38:39 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Updating volume 0fed6b21-0520-4554-bee4-9540a03f9af0 with {'scheduled_at': datetime.datetime(2023, 4, 17, 15, 38, 39, 65895), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 15:38:39 user cinder-volume[84696]: DEBUG cinder.utils [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.445(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:38:39 user cinder-volume[84696]: DEBUG cinder.utils [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_5d9c86cd-3e22-4926-bfc9-e5a37f6ae76c/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:38:39 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Volume 0fed6b21-0520-4554-bee4-9540a03f9af0: re-scheduling SchedulerAPI.create_volume attempt 3 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 15:38:39 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Volume 0fed6b21-0520-4554-bee4-9540a03f9af0: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 15:38:39 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] deleating volume v_0fed6b21-0520-4554-bee4-9540a03f9af0 {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 15:38:39 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] get snapshots for volume v_0fed6b21-0520-4554-bee4-9540a03f9af0 {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 15:38:39 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_0fed6b21-0520-4554-bee4-9540a03f9af0/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:38:39 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_0fed6b21-0520-4554-bee4-9540a03f9af0/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:38:42 user cinder-volume[84696]: DEBUG cinder.utils [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.770(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:38:42 user cinder-volume[84696]: DEBUG cinder.utils [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:38:42 user cinder-volume[84696]: DEBUG cinder.utils [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.987(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:38:42 user cinder-volume[84696]: DEBUG cinder.utils [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_0fed6b21-0520-4554-bee4-9540a03f9af0/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:38:42 user cinder-volume[84696]: DEBUG cinder.utils [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.147(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:38:42 user cinder-volume[84696]: DEBUG cinder.utils [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_44f909cf-7774-41f6-9f86-805456609ff8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:38:42 user cinder-volume[84696]: DEBUG cinder.utils [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.652(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:38:42 user cinder-volume[84696]: DEBUG cinder.utils [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:38:45 user cinder-volume[84696]: DEBUG cinder.utils [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.056(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:38:45 user cinder-volume[84696]: DEBUG cinder.utils [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_0fed6b21-0520-4554-bee4-9540a03f9af0/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:38:47 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:UPDATE=5,cinder:SELECT=11,cinder:INSERT=4 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:38:47 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Volume reschedule parameters: True retry: {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 15:38:47 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Flow 'volume_create_manager' (ead7cc50-5ceb-43d4-9614-1ed0bfbb7450) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:38:47 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (1bd8d5f5-00a3-49cb-804d-ac3208bc81f5) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:38:47 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (1bd8d5f5-00a3-49cb-804d-ac3208bc81f5) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:38:47Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-ServersNegativeTestJSON-volume-2082564469',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=a27d0ff3-5de9-4feb-82f7-d4571af88bb9,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='d2dd48f279404859bebcc19bcc272915',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:38:47Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:38:47Z,use_quota=True,user_id='b0075b26d6044a50981d29df0d863959',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:38:47 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (314a2907-5948-4ac0-85b7-c7ec1690c5b5) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:38:47 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (314a2907-5948-4ac0-85b7-c7ec1690c5b5) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:38:47 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (f8e2c763-d9fc-47d6-a6f5-0ebd48e8f47b) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:38:47 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (f8e2c763-d9fc-47d6-a6f5-0ebd48e8f47b) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': 'a27d0ff3-5de9-4feb-82f7-d4571af88bb9', 'volume_name': 'volume-a27d0ff3-5de9-4feb-82f7-d4571af88bb9', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:38:47 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (1de34f58-5d68-47ef-93aa-81234c771231) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:38:47 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (1de34f58-5d68-47ef-93aa-81234c771231) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:38:47 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (10330e8b-f1b7-43a7-9551-0888eeb88387) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:38:47 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Volume a27d0ff3-5de9-4feb-82f7-d4571af88bb9: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-a27d0ff3-5de9-4feb-82f7-d4571af88bb9', 'volume_size': 1} Apr 17 15:38:47 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] creating volume v_a27d0ff3-5de9-4feb-82f7-d4571af88bb9. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 15:38:47 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] create volume {'name': 'v_a27d0ff3-5de9-4feb-82f7-d4571af88bb9', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 15:38:47 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:38:47 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:38:48 user cinder-volume[84696]: DEBUG cinder.utils [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 0.878(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:38:48 user cinder-volume[84696]: DEBUG cinder.utils [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:38:51 user cinder-volume[84696]: DEBUG cinder.utils [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.200(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:38:51 user cinder-volume[84696]: DEBUG cinder.utils [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_0fed6b21-0520-4554-bee4-9540a03f9af0/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:38:51 user cinder-volume[84696]: DEBUG cinder.utils [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.618(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:38:51 user cinder-volume[84696]: DEBUG cinder.utils [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:38:51 user cinder-volume[84696]: DEBUG cinder.utils [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.828(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:38:51 user cinder-volume[84696]: DEBUG cinder.utils [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_8cc35ab1-363f-4ad6-9a37-239d2b1ea996/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:38:51 user cinder-volume[84696]: DEBUG cinder.utils [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.361(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:38:51 user cinder-volume[84696]: DEBUG cinder.utils [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_44f909cf-7774-41f6-9f86-805456609ff8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:38:51 user cinder-volume[84696]: DEBUG cinder.utils [None req-dbedb542-4536-4c1d-b79c-b9c0a6c64820 tempest-VolumesAdminNegativeTest-1285212149 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 137.953(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:38:51 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-dbedb542-4536-4c1d-b79c-b9c0a6c64820 tempest-VolumesAdminNegativeTest-1285212149 None] Geting None from DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:215a6a5b-25ca-48fa-a8e3-72416d7a39e2/luns/v_215a6a5b-25ca-48fa-a8e3-72416d7a39e2 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:38:51 user cinder-volume[84696]: DEBUG cinder.utils [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 3.958(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:38:51 user cinder-volume[84696]: DEBUG cinder.utils [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:38:51 user cinder-volume[84696]: DEBUG cinder.coordination [None req-dbedb542-4536-4c1d-b79c-b9c0a6c64820 tempest-VolumesAdminNegativeTest-1285212149 None] Lock "cinder-215a6a5b-25ca-48fa-a8e3-72416d7a39e2-delete_volume" released by "delete_volume" :: held 138.234s {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:207}} Apr 17 15:38:51 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-dbedb542-4536-4c1d-b79c-b9c0a6c64820 tempest-VolumesAdminNegativeTest-1285212149 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 15:38:51 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 15:38:51 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 15:38:51 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 15:38:51 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 15:38:51 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 15:38:51 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 15:38:51 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 15:38:51 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 203, in wrapper Apr 17 15:38:51 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server skip_clean = func(self, context, volume, *args, **kwargs) Apr 17 15:38:51 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 15:38:51 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 15:38:51 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/coordination.py", line 200, in _synchronized Apr 17 15:38:51 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return f(*a, **k) Apr 17 15:38:51 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 15:38:51 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 15:38:51 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 15:38:51 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 15:38:51 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 1035, in delete_volume Apr 17 15:38:51 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 15:38:51 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 15:38:51 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 15:38:51 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 15:38:51 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 15:38:51 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 1004, in delete_volume Apr 17 15:38:51 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.driver.remove_export(context, volume) Apr 17 15:38:51 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 720, in remove_export Apr 17 15:38:51 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self._remove_target_volume(volume) Apr 17 15:38:51 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 918, in _remove_target_volume Apr 17 15:38:51 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.detach_target_vol(target_name, jcom.vname(volume.id)) Apr 17 15:38:51 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 612, in detach_target_vol Apr 17 15:38:51 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if resp["code"] in (200, 201, 204): Apr 17 15:38:51 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 15:38:51 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 15:38:54 user cinder-volume[84696]: DEBUG cinder.utils [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 139.058(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:38:54 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_e80c78b2-20c9-4f9b-8898-6ab131d1eb06/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:38:54 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (6298f3cf-bb1b-429d-82fc-ec346f35e33b) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 15:38:54 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (9df5963a-0c49-4b10-8145-f7009247aea2) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:38:54 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (9df5963a-0c49-4b10-8145-f7009247aea2) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:38:54 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Flow 'volume_create_manager' (a8614e59-5bc3-4dab-90a0-4251e85b09b8) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 15:38:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 15:38:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 15:38:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 15:38:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 15:38:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 15:38:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 15:38:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 15:38:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 15:38:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 15:38:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 15:38:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 15:38:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 15:38:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 15:38:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 15:38:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 15:38:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 15:38:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 15:38:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 15:38:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 15:38:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 15:38:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 15:38:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 15:38:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 15:38:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 15:38:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:38:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 15:38:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1237, in execute Apr 17 15:38:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_from_image(context, Apr 17 15:38:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/utils.py", line 670, in _wrapper Apr 17 15:38:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return r(f, *args, **kwargs) Apr 17 15:38:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/tenacity/__init__.py", line 423, in __call__ Apr 17 15:38:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server do = self.iter(retry_state=retry_state) Apr 17 15:38:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/tenacity/__init__.py", line 360, in iter Apr 17 15:38:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return fut.result() Apr 17 15:38:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/lib/python3.10/concurrent/futures/_base.py", line 451, in result Apr 17 15:38:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self.__get_result() Apr 17 15:38:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result Apr 17 15:38:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self._exception Apr 17 15:38:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/tenacity/__init__.py", line 426, in __call__ Apr 17 15:38:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = fn(*args, **kwargs) Apr 17 15:38:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1125, in _create_from_image Apr 17 15:38:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_from_image_cache_or_download( Apr 17 15:38:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 944, in _create_from_image_cache_or_download Apr 17 15:38:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update, cloned = self._create_from_image_cache( Apr 17 15:38:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 843, in _create_from_image_cache Apr 17 15:38:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_from_source_volume( Apr 17 15:38:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 659, in _create_from_source_volume Apr 17 15:38:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self.driver.create_cloned_volume(volume, srcvol_ref) Apr 17 15:38:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 525, in create_cloned_volume Apr 17 15:38:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self._clone_object(vname, cvname) Apr 17 15:38:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 461, in _clone_object Apr 17 15:38:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_snapshot(oname, coname) Apr 17 15:38:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 639, in create_snapshot Apr 17 15:38:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201, 204): Apr 17 15:38:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 15:38:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 15:38:54 user cinder-volume[84696]: DEBUG cinder.utils [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 37.441(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:38:54 user cinder-volume[84696]: DEBUG cinder.utils [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:38:54 user cinder-volume[84696]: DEBUG cinder.utils [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 138.790(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:38:54 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:38:54 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (f8affc87-9883-4a0c-8037-a89dabe85101) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 15:38:54 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 15:38:54 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:34:17Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TaggedBootDevicesTest-volume-1099737906',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=e80c78b2-20c9-4f9b-8898-6ab131d1eb06,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='75ab8873f7064d3fbe954746c071cb72',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:36:35Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:36:35Z,use_quota=True,user_id='39328f4ed6da4a85ac312bc5637b1500',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 15:38:54 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:34:17Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TaggedBootDevicesTest-volume-1099737906',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=e80c78b2-20c9-4f9b-8898-6ab131d1eb06,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='75ab8873f7064d3fbe954746c071cb72',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:36:35Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:36:35Z,use_quota=True,user_id='39328f4ed6da4a85ac312bc5637b1500',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=b3f0dec1-7758-44a8-82e0-76ada8837428,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(e80c78b2-20c9-4f9b-8898-6ab131d1eb06),volume_id=e80c78b2-20c9-4f9b-8898-6ab131d1eb06,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'image', 'volume_id': 'e80c78b2-20c9-4f9b-8898-6ab131d1eb06', 'volume_name': 'volume-e80c78b2-20c9-4f9b-8898-6ab131d1eb06', 'volume_size': 1, 'image_id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'image_location': (None, None), 'image_meta': {'name': 'cirros-0.5.2-x86_64-disk', 'disk_format': 'qcow2', 'container_format': 'bare', 'visibility': 'public', 'size': 16300544, 'virtual_size': 117440512, 'status': 'active', 'checksum': 'b874c39491a2377b8490f5f1e89761a4', 'protected': False, 'min_ram': 0, 'min_disk': 0, 'owner': 'a43c211b4ed542538799349489b1d884', 'os_hidden': False, 'os_hash_algo': 'sha512', 'os_hash_value': '6b813aa46bb90b4da216a4d19376593fa3f4fc7e617f03a92b7fe11e9a3981cbe8f0959dbebe36225e5f53dc4492341a4863cac4ed1ee0909f3fc78ef9c3e869', 'id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'created_at': datetime.datetime(2023, 4, 17, 15, 19, 53, tzinfo=datetime.timezone.utc), 'updated_at': datetime.datetime(2023, 4, 17, 15, 19, 55, tzinfo=datetime.timezone.utc), 'tags': [], 'file': '/v2/images/b3f0dec1-7758-44a8-82e0-76ada8837428/file', 'properties': {'hw_rng_model': 'virtio', 'owner_specified.openstack.md5': '', 'owner_specified.openstack.object': 'images/cirros-0.5.2-x86_64-disk', 'owner_specified.openstack.sha256': ''}}, 'image_service': }} Apr 17 15:38:54 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'snapshot_id': None, 'source_volid': None, 'volume_id': 'e80c78b2-20c9-4f9b-8898-6ab131d1eb06', 'volume': {'id': 'e80c78b2-20c9-4f9b-8898-6ab131d1eb06', '_name_id': None, 'ec2_id': None, 'user_id': '39328f4ed6da4a85ac312bc5637b1500', 'project_id': '75ab8873f7064d3fbe954746c071cb72', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-TaggedBootDevicesTest-volume-1099737906', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'admin_metadata': {}, 'glance_metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T15:34:16.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-e80c78b2-20c9-4f9b-8898-6ab131d1eb06', 'name_id': 'e80c78b2-20c9-4f9b-8898-6ab131d1eb06', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-TaggedBootDevicesTest-volume-1099737906', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '75ab8873f7064d3fbe954746c071cb72', 'qos_specs': None, 'reservations': ['d6688115-7e3d-438c-8fc0-69b8ec7a1cde', '8ca9607c-d2b9-47d7-8e24-f752a5ed3e59', 'a110fbe0-e00b-4d63-a1a6-1c96ea375721', '06947137-4319-40d9-b8a3-4db09cd63ce4'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': '39328f4ed6da4a85ac312bc5637b1500', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-TaggedBootDevicesTest-volume-1099737906', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '75ab8873f7064d3fbe954746c071cb72', 'qos_specs': None, 'reservations': ['d6688115-7e3d-438c-8fc0-69b8ec7a1cde', '8ca9607c-d2b9-47d7-8e24-f752a5ed3e59', 'a110fbe0-e00b-4d63-a1a6-1c96ea375721', '06947137-4319-40d9-b8a3-4db09cd63ce4'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': '39328f4ed6da4a85ac312bc5637b1500', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': '39328f4ed6da4a85ac312bc5637b1500', 'metadata': {}, 'qos_specs': None, 'retry': {'num_attempts': 2, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']}}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=b3f0dec1-7758-44a8-82e0-76ada8837428,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(e80c78b2-20c9-4f9b-8898-6ab131d1eb06),volume_id=e80c78b2-20c9-4f9b-8898-6ab131d1eb06,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:34:17Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TaggedBootDevicesTest-volume-1099737906',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=e80c78b2-20c9-4f9b-8898-6ab131d1eb06,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='75ab8873f7064d3fbe954746c071cb72',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:36:35Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:36:35Z,use_quota=True,user_id='39328f4ed6da4a85ac312bc5637b1500',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 15:38:54 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:34:17Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TaggedBootDevicesTest-volume-1099737906',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=e80c78b2-20c9-4f9b-8898-6ab131d1eb06,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='75ab8873f7064d3fbe954746c071cb72',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:36:35Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:36:35Z,use_quota=True,user_id='39328f4ed6da4a85ac312bc5637b1500',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:34:17Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TaggedBootDevicesTest-volume-1099737906',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=e80c78b2-20c9-4f9b-8898-6ab131d1eb06,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='75ab8873f7064d3fbe954746c071cb72',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:36:35Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:36:35Z,use_quota=True,user_id='39328f4ed6da4a85ac312bc5637b1500',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 15:38:54 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 15:38:54 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 15:38:54 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:38:54 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 15:38:54 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1237, in execute Apr 17 15:38:54 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_from_image(context, Apr 17 15:38:54 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/utils.py", line 670, in _wrapper Apr 17 15:38:54 user cinder-volume[84696]: ERROR cinder.volume.manager return r(f, *args, **kwargs) Apr 17 15:38:54 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/tenacity/__init__.py", line 423, in __call__ Apr 17 15:38:54 user cinder-volume[84696]: ERROR cinder.volume.manager do = self.iter(retry_state=retry_state) Apr 17 15:38:54 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/tenacity/__init__.py", line 360, in iter Apr 17 15:38:54 user cinder-volume[84696]: ERROR cinder.volume.manager return fut.result() Apr 17 15:38:54 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/lib/python3.10/concurrent/futures/_base.py", line 451, in result Apr 17 15:38:54 user cinder-volume[84696]: ERROR cinder.volume.manager return self.__get_result() Apr 17 15:38:54 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result Apr 17 15:38:54 user cinder-volume[84696]: ERROR cinder.volume.manager raise self._exception Apr 17 15:38:54 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/tenacity/__init__.py", line 426, in __call__ Apr 17 15:38:54 user cinder-volume[84696]: ERROR cinder.volume.manager result = fn(*args, **kwargs) Apr 17 15:38:54 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1125, in _create_from_image Apr 17 15:38:54 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_from_image_cache_or_download( Apr 17 15:38:54 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 944, in _create_from_image_cache_or_download Apr 17 15:38:54 user cinder-volume[84696]: ERROR cinder.volume.manager model_update, cloned = self._create_from_image_cache( Apr 17 15:38:54 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 843, in _create_from_image_cache Apr 17 15:38:54 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_from_source_volume( Apr 17 15:38:54 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 659, in _create_from_source_volume Apr 17 15:38:54 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self.driver.create_cloned_volume(volume, srcvol_ref) Apr 17 15:38:54 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 525, in create_cloned_volume Apr 17 15:38:54 user cinder-volume[84696]: ERROR cinder.volume.manager self._clone_object(vname, cvname) Apr 17 15:38:54 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 461, in _clone_object Apr 17 15:38:54 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_snapshot(oname, coname) Apr 17 15:38:54 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 639, in create_snapshot Apr 17 15:38:54 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201, 204): Apr 17 15:38:54 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 15:38:54 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 15:38:54 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (f8affc87-9883-4a0c-8037-a89dabe85101) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:38:54 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (f8affc87-9883-4a0c-8037-a89dabe85101) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:38:54 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (3fc4288d-ccff-4a94-acbe-e20b6000e098) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:38:54 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (3fc4288d-ccff-4a94-acbe-e20b6000e098) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:38:54 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (7c5ca36e-3a93-4ece-ab54-7d8d8b42ff32) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:38:54 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (7c5ca36e-3a93-4ece-ab54-7d8d8b42ff32) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:38:54 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (fe973769-6b51-4a99-b47e-a6ce4a1b7f6a) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:38:54 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Updating volume e80c78b2-20c9-4f9b-8898-6ab131d1eb06 with {'scheduled_at': datetime.datetime(2023, 4, 17, 15, 38, 54, 440413), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 15:38:54 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Volume e80c78b2-20c9-4f9b-8898-6ab131d1eb06: re-scheduling SchedulerAPI.create_volume attempt 2 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 15:38:54 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Volume e80c78b2-20c9-4f9b-8898-6ab131d1eb06: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 15:38:54 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] deleating volume v_e80c78b2-20c9-4f9b-8898-6ab131d1eb06 {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 15:38:54 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] get snapshots for volume v_e80c78b2-20c9-4f9b-8898-6ab131d1eb06 {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 15:38:54 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_e80c78b2-20c9-4f9b-8898-6ab131d1eb06/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:38:54 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_e80c78b2-20c9-4f9b-8898-6ab131d1eb06/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:38:54 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Volume reschedule parameters: True retry: {'num_attempts': 3, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 15:38:54 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Flow 'volume_create_manager' (5a27dc12-5c2b-4af0-b886-997e5825d26e) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:38:54 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (3e667095-326a-44c2-bf11-c7596ca0b673) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:38:54 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (3e667095-326a-44c2-bf11-c7596ca0b673) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:34:17Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TaggedBootDevicesTest-volume-1099737906',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=e80c78b2-20c9-4f9b-8898-6ab131d1eb06,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='75ab8873f7064d3fbe954746c071cb72',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:38:55Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:38:55Z,use_quota=True,user_id='39328f4ed6da4a85ac312bc5637b1500',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:38:54 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (a9adefa1-d40c-47d2-91d7-eca949dad911) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:38:54 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (a9adefa1-d40c-47d2-91d7-eca949dad911) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:38:54 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (3824494b-4da4-4a57-b785-2f6d29ed7b48) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:38:54 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (3824494b-4da4-4a57-b785-2f6d29ed7b48) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'image', 'volume_id': 'e80c78b2-20c9-4f9b-8898-6ab131d1eb06', 'volume_name': 'volume-e80c78b2-20c9-4f9b-8898-6ab131d1eb06', 'volume_size': 1, 'image_id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'image_location': (None, None), 'image_meta': {'name': 'cirros-0.5.2-x86_64-disk', 'disk_format': 'qcow2', 'container_format': 'bare', 'visibility': 'public', 'size': 16300544, 'virtual_size': 117440512, 'status': 'active', 'checksum': 'b874c39491a2377b8490f5f1e89761a4', 'protected': False, 'min_ram': 0, 'min_disk': 0, 'owner': 'a43c211b4ed542538799349489b1d884', 'os_hidden': False, 'os_hash_algo': 'sha512', 'os_hash_value': '6b813aa46bb90b4da216a4d19376593fa3f4fc7e617f03a92b7fe11e9a3981cbe8f0959dbebe36225e5f53dc4492341a4863cac4ed1ee0909f3fc78ef9c3e869', 'id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'created_at': datetime.datetime(2023, 4, 17, 15, 19, 53, tzinfo=datetime.timezone.utc), 'updated_at': datetime.datetime(2023, 4, 17, 15, 19, 55, tzinfo=datetime.timezone.utc), 'tags': [], 'file': '/v2/images/b3f0dec1-7758-44a8-82e0-76ada8837428/file', 'properties': {'hw_rng_model': 'virtio', 'owner_specified.openstack.md5': '', 'owner_specified.openstack.object': 'images/cirros-0.5.2-x86_64-disk', 'owner_specified.openstack.sha256': ''}}, 'image_service': }' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:38:54 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (37a3045e-bf49-40c1-bd38-69ade0379faa) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:38:54 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (37a3045e-bf49-40c1-bd38-69ade0379faa) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:38:54 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (36490734-f2a7-4d3f-81d9-412a304fb3bb) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:38:54 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Volume e80c78b2-20c9-4f9b-8898-6ab131d1eb06: being created as image with specification: {'status': 'creating', 'volume_name': 'volume-e80c78b2-20c9-4f9b-8898-6ab131d1eb06', 'volume_size': 1, 'image_id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'image_location': (None, None), 'image_meta': {'name': 'cirros-0.5.2-x86_64-disk', 'disk_format': 'qcow2', 'container_format': 'bare', 'visibility': 'public', 'size': 16300544, 'virtual_size': 117440512, 'status': 'active', 'checksum': 'b874c39491a2377b8490f5f1e89761a4', 'protected': False, 'min_ram': 0, 'min_disk': 0, 'owner': 'a43c211b4ed542538799349489b1d884', 'os_hidden': False, 'os_hash_algo': 'sha512', 'os_hash_value': '6b813aa46bb90b4da216a4d19376593fa3f4fc7e617f03a92b7fe11e9a3981cbe8f0959dbebe36225e5f53dc4492341a4863cac4ed1ee0909f3fc78ef9c3e869', 'id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'created_at': datetime.datetime(2023, 4, 17, 15, 19, 53, tzinfo=datetime.timezone.utc), 'updated_at': datetime.datetime(2023, 4, 17, 15, 19, 55, tzinfo=datetime.timezone.utc), 'tags': [], 'file': '/v2/images/b3f0dec1-7758-44a8-82e0-76ada8837428/file', 'properties': {'hw_rng_model': 'virtio', 'owner_specified.openstack.md5': '', 'owner_specified.openstack.object': 'images/cirros-0.5.2-x86_64-disk', 'owner_specified.openstack.sha256': ''}}, 'image_service': } Apr 17 15:38:54 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Cloning e80c78b2-20c9-4f9b-8898-6ab131d1eb06 from image b3f0dec1-7758-44a8-82e0-76ada8837428 at location (None, None). {{(pid=84696) _create_from_image /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1071}} Apr 17 15:38:54 user cinder-volume[84696]: DEBUG cinder.coordination [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Acquiring lock "cinder-b3f0dec1-7758-44a8-82e0-76ada8837428" by "_prepare_image_cache_entry" {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:192}} Apr 17 15:38:54 user cinder-volume[84696]: DEBUG cinder.coordination [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Lock "cinder-b3f0dec1-7758-44a8-82e0-76ada8837428" acquired by "_prepare_image_cache_entry" :: waited 0.012s {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:196}} Apr 17 15:38:54 user cinder-volume[84696]: DEBUG cinder.image.cache [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Found image-volume cache entry: {'id': 1, 'image_id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'volume_id': '006f20ae-5306-44fe-afdf-e433f79284e8', 'host': 'user@joviandss-2#joviandss-2', 'size': 1, 'image_updated_at': datetime.datetime(2023, 4, 17, 15, 19, 55), 'last_used': datetime.datetime(2023, 4, 17, 15, 38, 54, 775455)}. {{(pid=84696) get_entry /opt/stack/cinder/cinder/image/cache.py:76}} Apr 17 15:38:54 user cinder-volume[84696]: DEBUG cinder.image.cache [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Image-volume cache entry image_update_at = 2023-04-17 15:19:55+00:00, requested image updated_at = 2023-04-17 15:19:55+00:00. {{(pid=84696) _should_update_entry /opt/stack/cinder/cinder/image/cache.py:259}} Apr 17 15:38:54 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Found cache entry for image = b3f0dec1-7758-44a8-82e0-76ada8837428 on host user@joviandss-2#joviandss-2. {{(pid=84696) _prepare_image_cache_entry /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:886}} Apr 17 15:38:54 user cinder-volume[84696]: DEBUG cinder.coordination [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Lock "cinder-b3f0dec1-7758-44a8-82e0-76ada8837428" released by "_prepare_image_cache_entry" :: held 0.023s {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:207}} Apr 17 15:38:54 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Attempting to retrieve cache entry for image = b3f0dec1-7758-44a8-82e0-76ada8837428 on host user@joviandss-2#joviandss-2. {{(pid=84696) _create_from_image_cache /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:827}} Apr 17 15:38:54 user cinder-volume[84696]: DEBUG cinder.image.cache [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Found image-volume cache entry: {'id': 1, 'image_id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'volume_id': '006f20ae-5306-44fe-afdf-e433f79284e8', 'host': 'user@joviandss-2#joviandss-2', 'size': 1, 'image_updated_at': datetime.datetime(2023, 4, 17, 15, 19, 55), 'last_used': datetime.datetime(2023, 4, 17, 15, 38, 54, 790243)}. {{(pid=84696) get_entry /opt/stack/cinder/cinder/image/cache.py:76}} Apr 17 15:38:54 user cinder-volume[84696]: DEBUG cinder.image.cache [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Image-volume cache entry image_update_at = 2023-04-17 15:19:55+00:00, requested image updated_at = 2023-04-17 15:19:55+00:00. {{(pid=84696) _should_update_entry /opt/stack/cinder/cinder/image/cache.py:259}} Apr 17 15:38:54 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Creating from source image-volume 006f20ae-5306-44fe-afdf-e433f79284e8 {{(pid=84696) _create_from_image_cache /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:841}} Apr 17 15:38:54 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] cloned volume 006f20ae-5306-44fe-afdf-e433f79284e8 to e80c78b2-20c9-4f9b-8898-6ab131d1eb06 {{(pid=84696) create_cloned_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:521}} Apr 17 15:38:54 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] cloning v_006f20ae-5306-44fe-afdf-e433f79284e8 to v_e80c78b2-20c9-4f9b-8898-6ab131d1eb06 {{(pid=84696) _clone_object /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:456}} Apr 17 15:38:54 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] create snapshot v_e80c78b2-20c9-4f9b-8898-6ab131d1eb06 {{(pid=84696) create_snapshot /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:635}} Apr 17 15:38:54 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:38:54 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:38:57 user cinder-volume[84696]: DEBUG cinder.utils [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.812(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:38:57 user cinder-volume[84696]: DEBUG cinder.utils [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_5d9c86cd-3e22-4926-bfc9-e5a37f6ae76c/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:38:57 user cinder-volume[84696]: DEBUG cinder.utils [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.629(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:38:57 user cinder-volume[84696]: DEBUG cinder.utils [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:38:57 user cinder-volume[84696]: DEBUG cinder.utils [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 10.097(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:38:57 user cinder-volume[84696]: DEBUG cinder.utils [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:38:57 user cinder-volume[84696]: DEBUG cinder.utils [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.972(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:38:57 user cinder-volume[84696]: DEBUG cinder.utils [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_e80c78b2-20c9-4f9b-8898-6ab131d1eb06/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:39:00 user cinder-volume[84696]: DEBUG cinder.utils [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.420(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:39:00 user cinder-volume[84696]: DEBUG cinder.utils [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_0fed6b21-0520-4554-bee4-9540a03f9af0/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:39:00 user cinder-volume[84696]: DEBUG cinder.utils [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.705(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:39:00 user cinder-volume[84696]: DEBUG cinder.utils [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:39:00 user cinder-volume[84696]: DEBUG cinder.utils [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.048(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:39:00 user cinder-volume[84696]: DEBUG cinder.utils [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_e80c78b2-20c9-4f9b-8898-6ab131d1eb06/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:39:04 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=18,cinder:UPDATE=10,cinder:DELETE=1,cinder:INSERT=1 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:39:06 user cinder-volume[84696]: DEBUG cinder.utils [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 19.470(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:39:06 user cinder-volume[84696]: DEBUG cinder.utils [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:39:06 user cinder-volume[84696]: DEBUG cinder.utils [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.004(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:39:06 user cinder-volume[84696]: DEBUG cinder.utils [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:39:06 user cinder-volume[84696]: DEBUG cinder.utils [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.346(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:39:06 user cinder-volume[84696]: DEBUG cinder.utils [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_e80c78b2-20c9-4f9b-8898-6ab131d1eb06/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:39:09 user cinder-volume[84696]: DEBUG cinder.utils [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.953(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:39:09 user cinder-volume[84696]: DEBUG cinder.utils [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_44f909cf-7774-41f6-9f86-805456609ff8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:39:09 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 73.888(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:39:09 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:39:14 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=13,cinder:UPDATE=3 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:39:16 user cinder-volume[84696]: DEBUG cinder.utils [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.219(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:39:16 user cinder-volume[84696]: DEBUG cinder.utils [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:39:16 user cinder-volume[84696]: DEBUG cinder.utils [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.562(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:39:16 user cinder-volume[84696]: DEBUG cinder.utils [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_e80c78b2-20c9-4f9b-8898-6ab131d1eb06/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:39:16 user cinder-volume[84696]: DEBUG cinder.utils [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 139.203(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:39:16 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_29d9e55e-0b8b-4f70-b5c4-830d39f10fcd/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:39:16 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (8dd354fa-c2fd-47a3-b436-20c24e43b89c) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 15:39:16 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (b724dd98-84e2-44ad-809d-9f3d9f9cffb0) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:39:16 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (b724dd98-84e2-44ad-809d-9f3d9f9cffb0) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:39:16 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Flow 'volume_create_manager' (d55eec9d-6854-427d-a3dd-46f6883c264e) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 15:39:16 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 15:39:16 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 15:39:16 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 15:39:16 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 15:39:16 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 15:39:16 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 15:39:16 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 15:39:16 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 15:39:16 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 15:39:16 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 15:39:16 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 15:39:16 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 15:39:16 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 15:39:16 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 15:39:16 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 15:39:16 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 15:39:16 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 15:39:16 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 15:39:16 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 15:39:16 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 15:39:16 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 15:39:16 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 15:39:16 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 15:39:16 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 15:39:16 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:39:16 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 15:39:16 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 15:39:16 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 15:39:16 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 15:39:16 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 15:39:16 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 15:39:16 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 15:39:16 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 15:39:16 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 15:39:16 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 15:39:16 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 15:39:16 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 15:39:16 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 15:39:16 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 15:39:16 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 15:39:16 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 15:39:16 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 15:39:16 user cinder-volume[84696]: DEBUG cinder.utils [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 139.048(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:39:16 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:39:16 user cinder-volume[84696]: INFO cinder.message.api [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Creating message record for request_id = req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 Apr 17 15:39:16 user cinder-volume[84696]: DEBUG cinder.utils [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 73.626(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:39:16 user cinder-volume[84696]: DEBUG cinder.utils [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:39:16 user cinder-volume[84696]: DEBUG cinder.utils [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 73.751(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:39:16 user cinder-volume[84696]: DEBUG cinder.utils [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:39:16 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 15:39:16 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (f0474e26-2b7d-4e8c-b91c-3f0c0fe3fa4a) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 15:39:16 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 15:39:16 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:34:44Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesSnapshotNegativeTestJSON-Volume-379445405',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=29d9e55e-0b8b-4f70-b5c4-830d39f10fcd,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='0b21bda55be444fc9f1a9c8c37b09224',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:36:57Z,service_uuid=None,shared_targets=True,size=2,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:36:57Z,use_quota=True,user_id='df17d7972cc649eb829bde41b8a718e4',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 15:39:16 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:34:44Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesSnapshotNegativeTestJSON-Volume-379445405',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=29d9e55e-0b8b-4f70-b5c4-830d39f10fcd,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='0b21bda55be444fc9f1a9c8c37b09224',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:36:57Z,service_uuid=None,shared_targets=True,size=2,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:36:57Z,use_quota=True,user_id='df17d7972cc649eb829bde41b8a718e4',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(29d9e55e-0b8b-4f70-b5c4-830d39f10fcd),volume_id=29d9e55e-0b8b-4f70-b5c4-830d39f10fcd,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': '29d9e55e-0b8b-4f70-b5c4-830d39f10fcd', 'volume_name': 'volume-29d9e55e-0b8b-4f70-b5c4-830d39f10fcd', 'volume_size': 2}} Apr 17 15:39:16 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': '29d9e55e-0b8b-4f70-b5c4-830d39f10fcd', 'volume': {'id': '29d9e55e-0b8b-4f70-b5c4-830d39f10fcd', '_name_id': None, 'ec2_id': None, 'user_id': 'df17d7972cc649eb829bde41b8a718e4', 'project_id': '0b21bda55be444fc9f1a9c8c37b09224', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 2, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-VolumesSnapshotNegativeTestJSON-Volume-379445405', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'admin_metadata': {}, 'glance_metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T15:34:44.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-29d9e55e-0b8b-4f70-b5c4-830d39f10fcd', 'name_id': '29d9e55e-0b8b-4f70-b5c4-830d39f10fcd', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumesSnapshotNegativeTestJSON-Volume-379445405', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '0b21bda55be444fc9f1a9c8c37b09224', 'qos_specs': None, 'reservations': ['da4aff04-a876-4157-b666-0fcc804a5076', '8e06fcb4-c53a-4537-9e40-36798fd5109c', '6bf081db-b8e9-44ff-ab74-ff0b0db7654a', 'ea5f14dc-ad5d-406a-8281-815b78e4eeb3'], 'size': 2, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'df17d7972cc649eb829bde41b8a718e4', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumesSnapshotNegativeTestJSON-Volume-379445405', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '0b21bda55be444fc9f1a9c8c37b09224', 'qos_specs': None, 'reservations': ['da4aff04-a876-4157-b666-0fcc804a5076', '8e06fcb4-c53a-4537-9e40-36798fd5109c', '6bf081db-b8e9-44ff-ab74-ff0b0db7654a', 'ea5f14dc-ad5d-406a-8281-815b78e4eeb3'], 'size': 2, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'df17d7972cc649eb829bde41b8a718e4', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 2, 'availability_zone': 'nova', 'user_id': 'df17d7972cc649eb829bde41b8a718e4', 'metadata': {}, 'qos_specs': None, 'retry': {'num_attempts': 2, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']}}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(29d9e55e-0b8b-4f70-b5c4-830d39f10fcd),volume_id=29d9e55e-0b8b-4f70-b5c4-830d39f10fcd,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:34:44Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesSnapshotNegativeTestJSON-Volume-379445405',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=29d9e55e-0b8b-4f70-b5c4-830d39f10fcd,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='0b21bda55be444fc9f1a9c8c37b09224',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:36:57Z,service_uuid=None,shared_targets=True,size=2,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:36:57Z,use_quota=True,user_id='df17d7972cc649eb829bde41b8a718e4',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 15:39:16 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:34:44Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesSnapshotNegativeTestJSON-Volume-379445405',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=29d9e55e-0b8b-4f70-b5c4-830d39f10fcd,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='0b21bda55be444fc9f1a9c8c37b09224',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:36:57Z,service_uuid=None,shared_targets=True,size=2,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:36:57Z,use_quota=True,user_id='df17d7972cc649eb829bde41b8a718e4',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:34:44Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesSnapshotNegativeTestJSON-Volume-379445405',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=29d9e55e-0b8b-4f70-b5c4-830d39f10fcd,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='0b21bda55be444fc9f1a9c8c37b09224',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:36:57Z,service_uuid=None,shared_targets=True,size=2,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:36:57Z,use_quota=True,user_id='df17d7972cc649eb829bde41b8a718e4',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 15:39:16 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 15:39:16 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 15:39:16 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:39:16 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 15:39:16 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 15:39:16 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 15:39:16 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 15:39:16 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 15:39:16 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 15:39:16 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 15:39:16 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 15:39:16 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 15:39:16 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 15:39:16 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 15:39:16 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 15:39:16 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 15:39:16 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 15:39:16 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 15:39:16 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 15:39:16 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 15:39:16 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (f0474e26-2b7d-4e8c-b91c-3f0c0fe3fa4a) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:39:16 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (f0474e26-2b7d-4e8c-b91c-3f0c0fe3fa4a) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:39:16 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (88867c92-c4e9-4248-ad4e-f920bbd38421) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:39:16 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (88867c92-c4e9-4248-ad4e-f920bbd38421) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:39:16 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (817c79b0-10e6-44e0-a905-f56f897ee2e0) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:39:16 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (817c79b0-10e6-44e0-a905-f56f897ee2e0) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:39:16 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (da3db1cb-1b53-4ead-9ae4-b6be31938c54) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:39:16 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Updating volume 29d9e55e-0b8b-4f70-b5c4-830d39f10fcd with {'scheduled_at': datetime.datetime(2023, 4, 17, 15, 39, 16, 113962), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 15:39:16 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Volume 29d9e55e-0b8b-4f70-b5c4-830d39f10fcd: re-scheduling SchedulerAPI.create_volume attempt 2 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 15:39:16 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Volume 29d9e55e-0b8b-4f70-b5c4-830d39f10fcd: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 15:39:16 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] deleating volume v_29d9e55e-0b8b-4f70-b5c4-830d39f10fcd {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 15:39:16 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] get snapshots for volume v_29d9e55e-0b8b-4f70-b5c4-830d39f10fcd {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 15:39:16 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_29d9e55e-0b8b-4f70-b5c4-830d39f10fcd/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:39:16 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_29d9e55e-0b8b-4f70-b5c4-830d39f10fcd/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:39:16 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Volume reschedule parameters: True retry: {'num_attempts': 3, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 15:39:16 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Flow 'volume_create_manager' (849d3848-80f5-4a4d-8bd9-2bbb47a5c5c3) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:39:16 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (36fbe028-1cf1-477f-a456-9ddceaa032ac) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:39:16 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (36fbe028-1cf1-477f-a456-9ddceaa032ac) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:34:44Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesSnapshotNegativeTestJSON-Volume-379445405',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=29d9e55e-0b8b-4f70-b5c4-830d39f10fcd,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='0b21bda55be444fc9f1a9c8c37b09224',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:39:16Z,service_uuid=None,shared_targets=True,size=2,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:39:16Z,use_quota=True,user_id='df17d7972cc649eb829bde41b8a718e4',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:39:16 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (993cffe6-ba6b-4465-b578-d70fa9f61aa4) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:39:16 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (993cffe6-ba6b-4465-b578-d70fa9f61aa4) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:39:16 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (805eeac4-44a5-47ea-a5f9-0075b41b4116) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:39:16 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (805eeac4-44a5-47ea-a5f9-0075b41b4116) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': '29d9e55e-0b8b-4f70-b5c4-830d39f10fcd', 'volume_name': 'volume-29d9e55e-0b8b-4f70-b5c4-830d39f10fcd', 'volume_size': 2}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:39:16 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (5fe25169-c5dd-4130-aee6-d40987ae3394) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:39:16 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (5fe25169-c5dd-4130-aee6-d40987ae3394) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:39:16 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (cdcc8d8b-051c-4af3-91ac-bd03fc2b1945) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:39:16 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Volume 29d9e55e-0b8b-4f70-b5c4-830d39f10fcd: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-29d9e55e-0b8b-4f70-b5c4-830d39f10fcd', 'volume_size': 2} Apr 17 15:39:16 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] creating volume v_29d9e55e-0b8b-4f70-b5c4-830d39f10fcd. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 15:39:16 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] create volume {'name': 'v_29d9e55e-0b8b-4f70-b5c4-830d39f10fcd', 'size': '2147483648', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 15:39:16 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:39:16 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:39:19 user cinder-volume[84696]: DEBUG cinder.utils [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.789(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:39:19 user cinder-volume[84696]: DEBUG cinder.utils [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:39:19 user cinder-volume[84696]: DEBUG cinder.utils [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.964(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:39:19 user cinder-volume[84696]: DEBUG cinder.utils [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_29d9e55e-0b8b-4f70-b5c4-830d39f10fcd/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:39:19 user cinder-volume[84696]: DEBUG cinder.utils [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 40.014(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:39:19 user cinder-volume[84696]: DEBUG cinder.utils [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_0fed6b21-0520-4554-bee4-9540a03f9af0/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:39:19 user cinder-volume[84696]: DEBUG cinder.utils [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 139.029(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:39:19 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:39:19 user cinder-volume[84696]: INFO cinder.message.api [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Creating message record for request_id = req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 Apr 17 15:39:19 user cinder-volume[84696]: DEBUG cinder.utils [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 139.218(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:39:19 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_44073502-102d-452a-8244-69a8233b9060/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:39:19 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (dcd2578c-e54b-4460-9110-7a625f5c8f06) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 15:39:19 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (a08f7252-02d4-4246-8712-cb78ca4dbe35) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:39:19 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (a08f7252-02d4-4246-8712-cb78ca4dbe35) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:39:19 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Flow 'volume_create_manager' (70c615e4-6344-4f7d-9e21-81127ba4e18f) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 15:39:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 15:39:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 15:39:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 15:39:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 15:39:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 15:39:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 15:39:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 15:39:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 15:39:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 15:39:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 15:39:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 15:39:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 15:39:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 15:39:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 15:39:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 15:39:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 15:39:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 15:39:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 15:39:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 15:39:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 15:39:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 15:39:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 15:39:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 15:39:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 15:39:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:39:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 15:39:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 15:39:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 15:39:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 15:39:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 15:39:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 15:39:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 15:39:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 15:39:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 15:39:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 15:39:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 15:39:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 15:39:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 15:39:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 15:39:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 15:39:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 15:39:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 15:39:19 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 15:39:19 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (62bbc6cc-8ee2-47d0-9eeb-4d5fee96582f) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 15:39:19 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 15:39:19 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:34:45Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesActionsTest-Volume-737622743',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=44073502-102d-452a-8244-69a8233b9060,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='a8b97ff6ed6047f0a33e74684a1c162b',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:37:00Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:37:00Z,use_quota=True,user_id='c993ceeb4a5c49eea14057e2125518ad',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 15:39:19 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:34:45Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesActionsTest-Volume-737622743',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=44073502-102d-452a-8244-69a8233b9060,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='a8b97ff6ed6047f0a33e74684a1c162b',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:37:00Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:37:00Z,use_quota=True,user_id='c993ceeb4a5c49eea14057e2125518ad',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(44073502-102d-452a-8244-69a8233b9060),volume_id=44073502-102d-452a-8244-69a8233b9060,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': '44073502-102d-452a-8244-69a8233b9060', 'volume_name': 'volume-44073502-102d-452a-8244-69a8233b9060', 'volume_size': 1}} Apr 17 15:39:19 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': '44073502-102d-452a-8244-69a8233b9060', 'volume': {'id': '44073502-102d-452a-8244-69a8233b9060', '_name_id': None, 'ec2_id': None, 'user_id': 'c993ceeb4a5c49eea14057e2125518ad', 'project_id': 'a8b97ff6ed6047f0a33e74684a1c162b', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-VolumesActionsTest-Volume-737622743', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'admin_metadata': {}, 'glance_metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T15:34:45.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-44073502-102d-452a-8244-69a8233b9060', 'name_id': '44073502-102d-452a-8244-69a8233b9060', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumesActionsTest-Volume-737622743', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': 'a8b97ff6ed6047f0a33e74684a1c162b', 'qos_specs': None, 'reservations': ['bb018cba-ad93-4b64-acaa-8608d0a298cb', 'c3b20513-152c-40a3-a11d-aac4f6530172', '567445e0-7c93-4f93-b254-4b34cc804061', '988486ce-3f26-4167-a9e9-e6aa5ffe2065'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'c993ceeb4a5c49eea14057e2125518ad', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumesActionsTest-Volume-737622743', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': 'a8b97ff6ed6047f0a33e74684a1c162b', 'qos_specs': None, 'reservations': ['bb018cba-ad93-4b64-acaa-8608d0a298cb', 'c3b20513-152c-40a3-a11d-aac4f6530172', '567445e0-7c93-4f93-b254-4b34cc804061', '988486ce-3f26-4167-a9e9-e6aa5ffe2065'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'c993ceeb4a5c49eea14057e2125518ad', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': 'c993ceeb4a5c49eea14057e2125518ad', 'metadata': {}, 'qos_specs': None, 'retry': {'num_attempts': 2, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']}}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(44073502-102d-452a-8244-69a8233b9060),volume_id=44073502-102d-452a-8244-69a8233b9060,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:34:45Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesActionsTest-Volume-737622743',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=44073502-102d-452a-8244-69a8233b9060,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='a8b97ff6ed6047f0a33e74684a1c162b',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:37:00Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:37:00Z,use_quota=True,user_id='c993ceeb4a5c49eea14057e2125518ad',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 15:39:19 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:34:45Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesActionsTest-Volume-737622743',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=44073502-102d-452a-8244-69a8233b9060,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='a8b97ff6ed6047f0a33e74684a1c162b',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:37:00Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:37:00Z,use_quota=True,user_id='c993ceeb4a5c49eea14057e2125518ad',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:34:45Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesActionsTest-Volume-737622743',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=44073502-102d-452a-8244-69a8233b9060,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='a8b97ff6ed6047f0a33e74684a1c162b',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:37:00Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:37:00Z,use_quota=True,user_id='c993ceeb4a5c49eea14057e2125518ad',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 15:39:19 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 15:39:19 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 15:39:19 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:39:19 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 15:39:19 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 15:39:19 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 15:39:19 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 15:39:19 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 15:39:19 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 15:39:19 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 15:39:19 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 15:39:19 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 15:39:19 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 15:39:19 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 15:39:19 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 15:39:19 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 15:39:19 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 15:39:19 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 15:39:19 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 15:39:19 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 15:39:19 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (62bbc6cc-8ee2-47d0-9eeb-4d5fee96582f) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:39:19 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (62bbc6cc-8ee2-47d0-9eeb-4d5fee96582f) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:39:19 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (e882b36a-d62e-4e7b-a207-b6b3671d802c) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:39:19 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (e882b36a-d62e-4e7b-a207-b6b3671d802c) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:39:19 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (99e025b0-8c90-4fd6-8973-baba2796f35a) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:39:19 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (99e025b0-8c90-4fd6-8973-baba2796f35a) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:39:19 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (d1e1c597-b5d4-4f26-a4cf-5d131084fa80) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:39:19 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Updating volume 44073502-102d-452a-8244-69a8233b9060 with {'scheduled_at': datetime.datetime(2023, 4, 17, 15, 39, 19, 186600), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 15:39:19 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Volume 44073502-102d-452a-8244-69a8233b9060: re-scheduling SchedulerAPI.create_volume attempt 2 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 15:39:19 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Volume 44073502-102d-452a-8244-69a8233b9060: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 15:39:19 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] deleating volume v_44073502-102d-452a-8244-69a8233b9060 {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 15:39:19 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] get snapshots for volume v_44073502-102d-452a-8244-69a8233b9060 {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 15:39:19 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_44073502-102d-452a-8244-69a8233b9060/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:39:19 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_44073502-102d-452a-8244-69a8233b9060/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:39:19 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Volume reschedule parameters: True retry: {'num_attempts': 3, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 15:39:19 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Flow 'volume_create_manager' (e8170c0b-3484-48ff-9fce-b5176a90fd27) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:39:19 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (b8ed0dc1-929c-461f-b9e4-2dad2e96b158) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:39:19 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (b8ed0dc1-929c-461f-b9e4-2dad2e96b158) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:34:45Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesActionsTest-Volume-737622743',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=44073502-102d-452a-8244-69a8233b9060,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='a8b97ff6ed6047f0a33e74684a1c162b',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:39:19Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:39:19Z,use_quota=True,user_id='c993ceeb4a5c49eea14057e2125518ad',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:39:19 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (3ce1e426-0c16-497b-adfe-a54970dc7680) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:39:19 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (3ce1e426-0c16-497b-adfe-a54970dc7680) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:39:19 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (06ede640-5cb3-4469-a8a1-bb96c7244b71) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:39:19 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (06ede640-5cb3-4469-a8a1-bb96c7244b71) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': '44073502-102d-452a-8244-69a8233b9060', 'volume_name': 'volume-44073502-102d-452a-8244-69a8233b9060', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:39:19 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (15d255df-caf9-495f-a84c-2da06d90d80c) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:39:19 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (15d255df-caf9-495f-a84c-2da06d90d80c) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:39:19 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (f06dc9e1-67eb-4807-90dd-151e38da0ccf) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:39:19 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Volume 44073502-102d-452a-8244-69a8233b9060: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-44073502-102d-452a-8244-69a8233b9060', 'volume_size': 1} Apr 17 15:39:19 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] creating volume v_44073502-102d-452a-8244-69a8233b9060. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 15:39:19 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] create volume {'name': 'v_44073502-102d-452a-8244-69a8233b9060', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 15:39:19 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:39:19 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:39:22 user cinder-volume[84696]: DEBUG cinder.utils [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.793(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:39:22 user cinder-volume[84696]: DEBUG cinder.utils [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:39:22 user cinder-volume[84696]: DEBUG cinder.utils [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.859(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:39:22 user cinder-volume[84696]: DEBUG cinder.utils [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:39:22 user cinder-volume[84696]: DEBUG cinder.utils [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.962(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:39:22 user cinder-volume[84696]: DEBUG cinder.utils [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_44073502-102d-452a-8244-69a8233b9060/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:39:22 user cinder-volume[84696]: DEBUG cinder.utils [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.034(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:39:22 user cinder-volume[84696]: DEBUG cinder.utils [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_29d9e55e-0b8b-4f70-b5c4-830d39f10fcd/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:39:25 user cinder-volume[84696]: DEBUG cinder.utils [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.031(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:39:25 user cinder-volume[84696]: DEBUG cinder.utils [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_44073502-102d-452a-8244-69a8233b9060/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:39:25 user cinder-volume[84696]: DEBUG cinder.utils [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 73.569(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:39:25 user cinder-volume[84696]: DEBUG cinder.utils [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:39:25 user cinder-volume[84696]: DEBUG cinder.utils [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 37.904(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:39:25 user cinder-volume[84696]: DEBUG cinder.utils [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:39:25 user cinder-volume[84696]: DEBUG cinder.utils [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.867(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:39:25 user cinder-volume[84696]: DEBUG cinder.utils [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:39:25 user cinder-volume[84696]: DEBUG cinder.utils [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 73.781(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:39:25 user cinder-volume[84696]: DEBUG cinder.utils [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_8cc35ab1-363f-4ad6-9a37-239d2b1ea996/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:39:28 user cinder-volume[84696]: DEBUG cinder.utils [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.175(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:39:28 user cinder-volume[84696]: DEBUG cinder.utils [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_29d9e55e-0b8b-4f70-b5c4-830d39f10fcd/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:39:28 user cinder-volume[84696]: DEBUG cinder.utils [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.003(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:39:28 user cinder-volume[84696]: DEBUG cinder.utils [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:39:28 user cinder-volume[84696]: DEBUG cinder.utils [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 71.373(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:39:28 user cinder-volume[84696]: DEBUG cinder.utils [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:39:31 user cinder-volume[84696]: DEBUG cinder.utils [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.013(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:39:31 user cinder-volume[84696]: DEBUG cinder.utils [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:39:31 user cinder-volume[84696]: DEBUG cinder.utils [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.181(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:39:31 user cinder-volume[84696]: DEBUG cinder.utils [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_44073502-102d-452a-8244-69a8233b9060/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:39:31 user cinder-volume[84696]: DEBUG cinder.utils [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 73.771(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:39:31 user cinder-volume[84696]: DEBUG cinder.utils [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_5d9c86cd-3e22-4926-bfc9-e5a37f6ae76c/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:39:31 user cinder-volume[84696]: DEBUG cinder.utils [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 135.851(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:39:31 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:39:31 user cinder-volume[84696]: INFO cinder.message.api [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Creating message record for request_id = req-de22aca9-c3f8-4015-be3e-9d70b59359d2 Apr 17 15:39:31 user cinder-volume[84696]: DEBUG cinder.utils [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 136.121(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:39:31 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_153b4df8-1f37-4118-8aa6-1c155ea5700c/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:39:31 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (dc6d0e22-ed05-4c9f-aa05-9065a5afc1a6) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 15:39:31 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (c781a08d-c532-4244-8543-10941f333b67) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:39:31 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (c781a08d-c532-4244-8543-10941f333b67) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:39:31 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Flow 'volume_create_manager' (fe34318b-61e5-4779-b002-9463edfd5e99) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 15:39:31 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 15:39:31 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 15:39:31 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 15:39:31 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 15:39:31 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 15:39:31 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 15:39:31 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 15:39:31 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 15:39:31 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 15:39:31 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 15:39:31 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 15:39:31 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 15:39:31 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 15:39:31 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 15:39:31 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 15:39:31 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 15:39:31 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 15:39:31 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 15:39:31 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 15:39:31 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 15:39:31 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 15:39:31 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 15:39:31 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 15:39:31 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 15:39:31 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:39:31 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 15:39:31 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 15:39:31 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 15:39:31 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 15:39:31 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 15:39:31 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 15:39:31 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 15:39:31 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 15:39:31 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 15:39:31 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 15:39:31 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 15:39:31 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 15:39:31 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 15:39:31 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 15:39:31 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 15:39:31 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 15:39:31 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 15:39:31 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 15:39:31 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (19eb47bb-3590-413b-86ac-1f8b709369d4) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 15:39:31 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 15:39:31 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:34:57Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesCloneTest-Volume-2030679513',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=153b4df8-1f37-4118-8aa6-1c155ea5700c,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='82fce1724afa40dd914e0791c3ee69a3',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:37:15Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:37:15Z,use_quota=True,user_id='773ce6c2a0f844edaf666e190504e258',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 15:39:31 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:34:57Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesCloneTest-Volume-2030679513',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=153b4df8-1f37-4118-8aa6-1c155ea5700c,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='82fce1724afa40dd914e0791c3ee69a3',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:37:15Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:37:15Z,use_quota=True,user_id='773ce6c2a0f844edaf666e190504e258',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(153b4df8-1f37-4118-8aa6-1c155ea5700c),volume_id=153b4df8-1f37-4118-8aa6-1c155ea5700c,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': '153b4df8-1f37-4118-8aa6-1c155ea5700c', 'volume_name': 'volume-153b4df8-1f37-4118-8aa6-1c155ea5700c', 'volume_size': 1}} Apr 17 15:39:31 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': '153b4df8-1f37-4118-8aa6-1c155ea5700c', 'volume': {'id': '153b4df8-1f37-4118-8aa6-1c155ea5700c', '_name_id': None, 'ec2_id': None, 'user_id': '773ce6c2a0f844edaf666e190504e258', 'project_id': '82fce1724afa40dd914e0791c3ee69a3', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-VolumesCloneTest-Volume-2030679513', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'admin_metadata': {}, 'glance_metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T15:34:57.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-153b4df8-1f37-4118-8aa6-1c155ea5700c', 'name_id': '153b4df8-1f37-4118-8aa6-1c155ea5700c', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumesCloneTest-Volume-2030679513', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '82fce1724afa40dd914e0791c3ee69a3', 'qos_specs': None, 'reservations': ['4e0e5a8a-460c-4231-830c-7500b8b4d874', '4e75ce56-bc87-4771-a3a2-a212a9299682', '1e95db6e-9eea-496d-9f91-b9dc2fd1660c', 'd2d362d2-ccef-4207-b0ce-c2abf0695375'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': '773ce6c2a0f844edaf666e190504e258', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumesCloneTest-Volume-2030679513', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '82fce1724afa40dd914e0791c3ee69a3', 'qos_specs': None, 'reservations': ['4e0e5a8a-460c-4231-830c-7500b8b4d874', '4e75ce56-bc87-4771-a3a2-a212a9299682', '1e95db6e-9eea-496d-9f91-b9dc2fd1660c', 'd2d362d2-ccef-4207-b0ce-c2abf0695375'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': '773ce6c2a0f844edaf666e190504e258', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': '773ce6c2a0f844edaf666e190504e258', 'metadata': {}, 'qos_specs': None, 'retry': {'num_attempts': 2, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']}}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(153b4df8-1f37-4118-8aa6-1c155ea5700c),volume_id=153b4df8-1f37-4118-8aa6-1c155ea5700c,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:34:57Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesCloneTest-Volume-2030679513',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=153b4df8-1f37-4118-8aa6-1c155ea5700c,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='82fce1724afa40dd914e0791c3ee69a3',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:37:15Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:37:15Z,use_quota=True,user_id='773ce6c2a0f844edaf666e190504e258',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 15:39:31 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:34:57Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesCloneTest-Volume-2030679513',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=153b4df8-1f37-4118-8aa6-1c155ea5700c,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='82fce1724afa40dd914e0791c3ee69a3',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:37:15Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:37:15Z,use_quota=True,user_id='773ce6c2a0f844edaf666e190504e258',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:34:57Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesCloneTest-Volume-2030679513',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=153b4df8-1f37-4118-8aa6-1c155ea5700c,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='82fce1724afa40dd914e0791c3ee69a3',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:37:15Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:37:15Z,use_quota=True,user_id='773ce6c2a0f844edaf666e190504e258',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 15:39:31 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 15:39:31 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 15:39:31 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:39:31 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 15:39:31 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 15:39:31 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 15:39:31 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 15:39:31 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 15:39:31 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 15:39:31 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 15:39:31 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 15:39:31 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 15:39:31 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 15:39:31 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 15:39:31 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 15:39:31 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 15:39:31 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 15:39:31 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 15:39:31 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 15:39:31 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 15:39:31 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (19eb47bb-3590-413b-86ac-1f8b709369d4) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:39:31 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (19eb47bb-3590-413b-86ac-1f8b709369d4) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:39:31 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (206c7ce3-8060-411f-aa5f-d8852469c119) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:39:31 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (206c7ce3-8060-411f-aa5f-d8852469c119) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:39:31 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (2660d8eb-518e-462d-9f51-c753ec8870b5) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:39:31 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (2660d8eb-518e-462d-9f51-c753ec8870b5) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:39:31 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (65c9e1e6-5853-421e-acca-4c4add9c4e18) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:39:31 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Updating volume 153b4df8-1f37-4118-8aa6-1c155ea5700c with {'scheduled_at': datetime.datetime(2023, 4, 17, 15, 39, 31, 478612), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 15:39:31 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Volume 153b4df8-1f37-4118-8aa6-1c155ea5700c: re-scheduling SchedulerAPI.create_volume attempt 2 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 15:39:31 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Volume 153b4df8-1f37-4118-8aa6-1c155ea5700c: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 15:39:31 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] deleating volume v_153b4df8-1f37-4118-8aa6-1c155ea5700c {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 15:39:31 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] get snapshots for volume v_153b4df8-1f37-4118-8aa6-1c155ea5700c {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 15:39:31 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_153b4df8-1f37-4118-8aa6-1c155ea5700c/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:39:31 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_153b4df8-1f37-4118-8aa6-1c155ea5700c/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:39:31 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Volume reschedule parameters: True retry: {'num_attempts': 3, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 15:39:31 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Flow 'volume_create_manager' (e5ab2144-842b-4aa0-821b-f3352dc9c371) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:39:31 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (d479dfe8-d2ee-4a0b-9f0e-2b183635c771) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:39:31 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (d479dfe8-d2ee-4a0b-9f0e-2b183635c771) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:34:57Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesCloneTest-Volume-2030679513',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=153b4df8-1f37-4118-8aa6-1c155ea5700c,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='82fce1724afa40dd914e0791c3ee69a3',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:39:32Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:39:32Z,use_quota=True,user_id='773ce6c2a0f844edaf666e190504e258',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:39:31 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (7c086dcf-f653-40fe-aa4b-b1e6bb7df085) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:39:31 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (7c086dcf-f653-40fe-aa4b-b1e6bb7df085) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:39:31 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (d8c97eff-b5fc-498b-b8d0-f31782630c29) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:39:31 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (d8c97eff-b5fc-498b-b8d0-f31782630c29) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': '153b4df8-1f37-4118-8aa6-1c155ea5700c', 'volume_name': 'volume-153b4df8-1f37-4118-8aa6-1c155ea5700c', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:39:31 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (8094f8b2-c537-494d-a389-b4d5a24ce4bf) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:39:31 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (8094f8b2-c537-494d-a389-b4d5a24ce4bf) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:39:31 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (d0d80f8c-1637-4101-a116-fabe4281bba4) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:39:31 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Volume 153b4df8-1f37-4118-8aa6-1c155ea5700c: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-153b4df8-1f37-4118-8aa6-1c155ea5700c', 'volume_size': 1} Apr 17 15:39:31 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] creating volume v_153b4df8-1f37-4118-8aa6-1c155ea5700c. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 15:39:31 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] create volume {'name': 'v_153b4df8-1f37-4118-8aa6-1c155ea5700c', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 15:39:31 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:39:31 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:39:34 user cinder-volume[84696]: DEBUG cinder.utils [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.941(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:39:34 user cinder-volume[84696]: DEBUG cinder.utils [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_153b4df8-1f37-4118-8aa6-1c155ea5700c/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:39:34 user cinder-volume[84696]: DEBUG cinder.utils [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.734(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:39:34 user cinder-volume[84696]: DEBUG cinder.utils [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:39:34 user cinder-volume[84696]: DEBUG cinder.utils [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.658(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:39:34 user cinder-volume[84696]: DEBUG cinder.utils [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:39:34 user cinder-volume[84696]: DEBUG cinder.utils [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 40.000(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:39:34 user cinder-volume[84696]: DEBUG cinder.utils [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_e80c78b2-20c9-4f9b-8898-6ab131d1eb06/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:39:37 user cinder-volume[84696]: DEBUG cinder.utils [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.390(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:39:37 user cinder-volume[84696]: DEBUG cinder.utils [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_29d9e55e-0b8b-4f70-b5c4-830d39f10fcd/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:39:37 user cinder-volume[84696]: DEBUG cinder.utils [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.010(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:39:37 user cinder-volume[84696]: DEBUG cinder.utils [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_153b4df8-1f37-4118-8aa6-1c155ea5700c/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:39:37 user cinder-volume[84696]: DEBUG cinder.utils [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.220(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:39:37 user cinder-volume[84696]: DEBUG cinder.utils [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:39:37 user cinder-volume[84696]: DEBUG cinder.utils [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.804(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:39:37 user cinder-volume[84696]: DEBUG cinder.utils [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:39:40 user cinder-volume[84696]: DEBUG cinder.utils [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.391(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:39:40 user cinder-volume[84696]: DEBUG cinder.utils [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_44073502-102d-452a-8244-69a8233b9060/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:39:40 user cinder-volume[84696]: DEBUG cinder.utils [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.226(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:39:40 user cinder-volume[84696]: DEBUG cinder.utils [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:39:43 user cinder-volume[84696]: DEBUG cinder.utils [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 11.944(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:39:43 user cinder-volume[84696]: DEBUG cinder.utils [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:39:43 user cinder-volume[84696]: DEBUG cinder.utils [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 73.742(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:39:43 user cinder-volume[84696]: DEBUG cinder.utils [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_44f909cf-7774-41f6-9f86-805456609ff8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:39:43 user cinder-volume[84696]: DEBUG cinder.utils [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.155(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:39:43 user cinder-volume[84696]: DEBUG cinder.utils [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_153b4df8-1f37-4118-8aa6-1c155ea5700c/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:39:44 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=14,cinder:UPDATE=7,cinder:INSERT=1 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:39:44 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=23,cinder:UPDATE=5,cinder:INSERT=5 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:39:52 user cinder-volume[84696]: DEBUG cinder.utils [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.400(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:39:52 user cinder-volume[84696]: DEBUG cinder.utils [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_153b4df8-1f37-4118-8aa6-1c155ea5700c/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:39:52 user cinder-volume[84696]: DEBUG cinder.utils [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.193(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:39:52 user cinder-volume[84696]: DEBUG cinder.utils [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:39:52 user cinder-volume[84696]: DEBUG cinder.utils [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 73.834(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:39:52 user cinder-volume[84696]: DEBUG cinder.utils [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_0fed6b21-0520-4554-bee4-9540a03f9af0/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:39:55 user cinder-volume[84696]: DEBUG cinder.utils [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.854(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:39:55 user cinder-volume[84696]: DEBUG cinder.utils [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_29d9e55e-0b8b-4f70-b5c4-830d39f10fcd/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:39:55 user cinder-volume[84696]: DEBUG cinder.utils [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.683(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:39:55 user cinder-volume[84696]: DEBUG cinder.utils [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:39:59 user cinder-volume[84696]: DEBUG cinder.utils [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 71.726(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:39:59 user cinder-volume[84696]: DEBUG cinder.utils [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:39:59 user cinder-volume[84696]: DEBUG cinder.utils [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.690(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:39:59 user cinder-volume[84696]: DEBUG cinder.utils [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:39:59 user cinder-volume[84696]: DEBUG cinder.utils [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.857(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:39:59 user cinder-volume[84696]: DEBUG cinder.utils [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_44073502-102d-452a-8244-69a8233b9060/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:40:04 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=5,cinder:UPDATE=1 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:40:04 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:UPDATE=1 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:40:08 user cinder-volume[84696]: DEBUG cinder.utils [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 74.051(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:40:08 user cinder-volume[84696]: DEBUG cinder.utils [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:40:08 user cinder-volume[84696]: DEBUG cinder.utils [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 74.395(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:40:08 user cinder-volume[84696]: DEBUG cinder.utils [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_e80c78b2-20c9-4f9b-8898-6ab131d1eb06/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:40:11 user cinder-volume[84696]: DEBUG cinder.utils [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 40.205(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:40:11 user cinder-volume[84696]: DEBUG cinder.utils [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:40:11 user cinder-volume[84696]: DEBUG cinder.utils [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 40.415(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:40:11 user cinder-volume[84696]: DEBUG cinder.utils [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_153b4df8-1f37-4118-8aa6-1c155ea5700c/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:40:15 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 139.002(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:40:15 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:40:15 user cinder-volume[84696]: ERROR oslo_service.periodic_task [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Error during VolumeManager.publish_service_capabilities: TypeError: 'NoneType' object is not subscriptable Apr 17 15:40:15 user cinder-volume[84696]: ERROR oslo_service.periodic_task Traceback (most recent call last): Apr 17 15:40:15 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/usr/local/lib/python3.10/dist-packages/oslo_service/periodic_task.py", line 216, in run_periodic_tasks Apr 17 15:40:15 user cinder-volume[84696]: ERROR oslo_service.periodic_task task(self, context) Apr 17 15:40:15 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/manager.py", line 2876, in publish_service_capabilities Apr 17 15:40:15 user cinder-volume[84696]: ERROR oslo_service.periodic_task self._report_driver_status(context) Apr 17 15:40:15 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/manager.py", line 2755, in _report_driver_status Apr 17 15:40:15 user cinder-volume[84696]: ERROR oslo_service.periodic_task volume_stats = get_stats() Apr 17 15:40:15 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/usr/local/lib/python3.10/dist-packages/oslo_utils/timeutils.py", line 319, in wrapper Apr 17 15:40:15 user cinder-volume[84696]: ERROR oslo_service.periodic_task result = func(*args, **kwargs) Apr 17 15:40:15 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/manager.py", line 2753, in get_stats Apr 17 15:40:15 user cinder-volume[84696]: ERROR oslo_service.periodic_task return self.driver.get_volume_stats(refresh=True) Apr 17 15:40:15 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/driver.py", line 666, in get_volume_stats Apr 17 15:40:15 user cinder-volume[84696]: ERROR oslo_service.periodic_task self._update_volume_stats() Apr 17 15:40:15 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 726, in _update_volume_stats Apr 17 15:40:15 user cinder-volume[84696]: ERROR oslo_service.periodic_task pool_stats = self.ra.get_pool_stats() Apr 17 15:40:15 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 897, in get_pool_stats Apr 17 15:40:15 user cinder-volume[84696]: ERROR oslo_service.periodic_task if not resp["error"] and resp["code"] == 200: Apr 17 15:40:15 user cinder-volume[84696]: ERROR oslo_service.periodic_task TypeError: 'NoneType' object is not subscriptable Apr 17 15:40:15 user cinder-volume[84696]: ERROR oslo_service.periodic_task Apr 17 15:40:15 user cinder-volume[84696]: WARNING oslo.service.loopingcall [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Function 'cinder.service.Service.periodic_tasks' run outlasted interval by 79.01 sec Apr 17 15:40:15 user cinder-volume[84696]: DEBUG oslo_service.periodic_task [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Running periodic task VolumeManager.publish_service_capabilities {{(pid=84696) run_periodic_tasks /usr/local/lib/python3.10/dist-packages/oslo_service/periodic_task.py:210}} Apr 17 15:40:15 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Updating volume stats {{(pid=84696) _update_volume_stats /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:724}} Apr 17 15:40:15 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Get pool Pool-0 fsprops {{(pid=84696) get_pool_stats /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:894}} Apr 17 15:40:15 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:40:15 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:40:18 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 3.064(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:40:18 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:40:21 user cinder-volume[84696]: DEBUG cinder.utils [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 138.717(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:40:21 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:40:21 user cinder-volume[84696]: INFO cinder.message.api [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Creating message record for request_id = req-e55035f7-bed9-4dad-9cbc-b8b208b454bf Apr 17 15:40:21 user cinder-volume[84696]: DEBUG cinder.utils [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 138.852(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:40:21 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:40:21 user cinder-volume[84696]: INFO cinder.message.api [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Creating message record for request_id = req-e0039512-b662-485b-95d9-48de5642a75d Apr 17 15:40:21 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.153(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:40:21 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:40:21 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 15:40:21 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 15:40:21 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (d735d96c-06d6-4e96-9c51-afb975869610) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 15:40:21 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 15:40:21 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:38:02Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesActionsTest-Volume-1957591186',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=57070f23-6b40-47b9-b9b4-e0d77240f544,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='a8b97ff6ed6047f0a33e74684a1c162b',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:38:02Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:38:02Z,use_quota=True,user_id='c993ceeb4a5c49eea14057e2125518ad',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 15:40:21 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:38:02Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesActionsTest-Volume-1957591186',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=57070f23-6b40-47b9-b9b4-e0d77240f544,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='a8b97ff6ed6047f0a33e74684a1c162b',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:38:02Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:38:02Z,use_quota=True,user_id='c993ceeb4a5c49eea14057e2125518ad',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(57070f23-6b40-47b9-b9b4-e0d77240f544),volume_id=57070f23-6b40-47b9-b9b4-e0d77240f544,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': '57070f23-6b40-47b9-b9b4-e0d77240f544', 'volume_name': 'volume-57070f23-6b40-47b9-b9b4-e0d77240f544', 'volume_size': 1}} Apr 17 15:40:21 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'retry': {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']}, 'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': '57070f23-6b40-47b9-b9b4-e0d77240f544', 'volume': {'id': '57070f23-6b40-47b9-b9b4-e0d77240f544', '_name_id': None, 'ec2_id': None, 'user_id': 'c993ceeb4a5c49eea14057e2125518ad', 'project_id': 'a8b97ff6ed6047f0a33e74684a1c162b', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-VolumesActionsTest-Volume-1957591186', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T15:38:02.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-57070f23-6b40-47b9-b9b4-e0d77240f544', 'name_id': '57070f23-6b40-47b9-b9b4-e0d77240f544', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumesActionsTest-Volume-1957591186', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': 'a8b97ff6ed6047f0a33e74684a1c162b', 'qos_specs': None, 'reservations': ['24ba47c4-9401-42c6-8d85-7f78767497d2', '1d6429be-3a45-4be3-a3f2-22a585792470', 'd0b8f941-f03c-4f32-b89a-dd9df5910de8', '4cdcf710-02d8-4b32-903f-875263491ca7'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'c993ceeb4a5c49eea14057e2125518ad', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumesActionsTest-Volume-1957591186', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': 'a8b97ff6ed6047f0a33e74684a1c162b', 'qos_specs': None, 'reservations': ['24ba47c4-9401-42c6-8d85-7f78767497d2', '1d6429be-3a45-4be3-a3f2-22a585792470', 'd0b8f941-f03c-4f32-b89a-dd9df5910de8', '4cdcf710-02d8-4b32-903f-875263491ca7'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'c993ceeb4a5c49eea14057e2125518ad', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': 'c993ceeb4a5c49eea14057e2125518ad', 'metadata': {}, 'qos_specs': None}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(57070f23-6b40-47b9-b9b4-e0d77240f544),volume_id=57070f23-6b40-47b9-b9b4-e0d77240f544,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:38:02Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesActionsTest-Volume-1957591186',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=57070f23-6b40-47b9-b9b4-e0d77240f544,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='a8b97ff6ed6047f0a33e74684a1c162b',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:38:02Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:38:02Z,use_quota=True,user_id='c993ceeb4a5c49eea14057e2125518ad',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 15:40:21 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:38:02Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesActionsTest-Volume-1957591186',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=57070f23-6b40-47b9-b9b4-e0d77240f544,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='a8b97ff6ed6047f0a33e74684a1c162b',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:38:02Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:38:02Z,use_quota=True,user_id='c993ceeb4a5c49eea14057e2125518ad',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:38:02Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesActionsTest-Volume-1957591186',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=57070f23-6b40-47b9-b9b4-e0d77240f544,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='a8b97ff6ed6047f0a33e74684a1c162b',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:38:02Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:38:02Z,use_quota=True,user_id='c993ceeb4a5c49eea14057e2125518ad',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 15:40:21 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 15:40:21 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 15:40:21 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:40:21 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 15:40:21 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 15:40:21 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 15:40:21 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 15:40:21 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 15:40:21 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 15:40:21 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 15:40:21 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 15:40:21 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 15:40:21 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 15:40:21 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 15:40:21 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 15:40:21 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 15:40:21 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 15:40:21 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 15:40:21 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 15:40:21 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 15:40:21 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (d735d96c-06d6-4e96-9c51-afb975869610) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:40:21 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (d735d96c-06d6-4e96-9c51-afb975869610) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:40:21 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (4f845aa8-09d5-4f53-bc54-94a3254a609b) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:40:21 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (4f845aa8-09d5-4f53-bc54-94a3254a609b) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:40:21 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (c7d58b83-b97e-4e10-95eb-6e530912e2bb) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:40:21 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (c7d58b83-b97e-4e10-95eb-6e530912e2bb) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:40:21 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (a7ae420e-549f-49e0-a4be-a45df8a475f7) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:40:21 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Updating volume 57070f23-6b40-47b9-b9b4-e0d77240f544 with {'scheduled_at': datetime.datetime(2023, 4, 17, 15, 40, 21, 236324), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 15:40:21 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (519b4995-f47b-4d89-9fc2-7947903f2f57) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 15:40:21 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 15:40:21 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:38:02Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-CreateVolumesFromSourceVolumeTest-Volume-609446595',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=a50e9659-3e2d-4f04-b0ac-ae78b8f61fad,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='99747be727ae4592ae1b6176b488719a',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:38:02Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:38:02Z,use_quota=True,user_id='0696808ea0b14c0794e8664a37d9e2c3',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 15:40:21 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:38:02Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-CreateVolumesFromSourceVolumeTest-Volume-609446595',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=a50e9659-3e2d-4f04-b0ac-ae78b8f61fad,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='99747be727ae4592ae1b6176b488719a',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:38:02Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:38:02Z,use_quota=True,user_id='0696808ea0b14c0794e8664a37d9e2c3',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(a50e9659-3e2d-4f04-b0ac-ae78b8f61fad),volume_id=a50e9659-3e2d-4f04-b0ac-ae78b8f61fad,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': 'a50e9659-3e2d-4f04-b0ac-ae78b8f61fad', 'volume_name': 'volume-a50e9659-3e2d-4f04-b0ac-ae78b8f61fad', 'volume_size': 1}} Apr 17 15:40:21 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'retry': {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']}, 'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': 'a50e9659-3e2d-4f04-b0ac-ae78b8f61fad', 'volume': {'id': 'a50e9659-3e2d-4f04-b0ac-ae78b8f61fad', '_name_id': None, 'ec2_id': None, 'user_id': '0696808ea0b14c0794e8664a37d9e2c3', 'project_id': '99747be727ae4592ae1b6176b488719a', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-CreateVolumesFromSourceVolumeTest-Volume-609446595', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T15:38:02.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-a50e9659-3e2d-4f04-b0ac-ae78b8f61fad', 'name_id': 'a50e9659-3e2d-4f04-b0ac-ae78b8f61fad', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-CreateVolumesFromSourceVolumeTest-Volume-609446595', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '99747be727ae4592ae1b6176b488719a', 'qos_specs': None, 'reservations': ['e9df84e9-0002-4c5e-9db7-eae2489ad0e9', 'd438171a-8bd2-47cf-81e3-e4af8b2d355c', 'cf24dd26-1a0b-48b2-95ca-4eceaffb6e37', 'e9528d9a-1681-4b42-a96b-ae681a21177b'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': '0696808ea0b14c0794e8664a37d9e2c3', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-CreateVolumesFromSourceVolumeTest-Volume-609446595', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '99747be727ae4592ae1b6176b488719a', 'qos_specs': None, 'reservations': ['e9df84e9-0002-4c5e-9db7-eae2489ad0e9', 'd438171a-8bd2-47cf-81e3-e4af8b2d355c', 'cf24dd26-1a0b-48b2-95ca-4eceaffb6e37', 'e9528d9a-1681-4b42-a96b-ae681a21177b'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': '0696808ea0b14c0794e8664a37d9e2c3', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': '0696808ea0b14c0794e8664a37d9e2c3', 'metadata': {}, 'qos_specs': None}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(a50e9659-3e2d-4f04-b0ac-ae78b8f61fad),volume_id=a50e9659-3e2d-4f04-b0ac-ae78b8f61fad,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:38:02Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-CreateVolumesFromSourceVolumeTest-Volume-609446595',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=a50e9659-3e2d-4f04-b0ac-ae78b8f61fad,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='99747be727ae4592ae1b6176b488719a',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:38:02Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:38:02Z,use_quota=True,user_id='0696808ea0b14c0794e8664a37d9e2c3',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 15:40:21 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:38:02Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-CreateVolumesFromSourceVolumeTest-Volume-609446595',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=a50e9659-3e2d-4f04-b0ac-ae78b8f61fad,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='99747be727ae4592ae1b6176b488719a',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:38:02Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:38:02Z,use_quota=True,user_id='0696808ea0b14c0794e8664a37d9e2c3',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:38:02Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-CreateVolumesFromSourceVolumeTest-Volume-609446595',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=a50e9659-3e2d-4f04-b0ac-ae78b8f61fad,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='99747be727ae4592ae1b6176b488719a',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:38:02Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:38:02Z,use_quota=True,user_id='0696808ea0b14c0794e8664a37d9e2c3',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 15:40:21 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 15:40:21 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 15:40:21 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:40:21 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 15:40:21 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 15:40:21 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 15:40:21 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 15:40:21 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 15:40:21 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 15:40:21 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 15:40:21 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 15:40:21 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 15:40:21 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 15:40:21 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 15:40:21 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 15:40:21 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 15:40:21 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 15:40:21 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 15:40:21 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 15:40:21 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 15:40:21 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (519b4995-f47b-4d89-9fc2-7947903f2f57) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:40:21 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (519b4995-f47b-4d89-9fc2-7947903f2f57) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:40:21 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (d0ff0a55-c6f3-444b-9430-8f5388e67040) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:40:21 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (d0ff0a55-c6f3-444b-9430-8f5388e67040) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:40:21 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (4fc9a793-5b3b-41ea-ba6a-f192ca9c65b3) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:40:21 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (4fc9a793-5b3b-41ea-ba6a-f192ca9c65b3) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:40:21 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (d0b2a951-fe4a-461f-9929-7e538f0f603f) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:40:21 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Updating volume a50e9659-3e2d-4f04-b0ac-ae78b8f61fad with {'scheduled_at': datetime.datetime(2023, 4, 17, 15, 40, 21, 288758), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 15:40:21 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Volume a50e9659-3e2d-4f04-b0ac-ae78b8f61fad: re-scheduling SchedulerAPI.create_volume attempt 1 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 15:40:21 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Volume 57070f23-6b40-47b9-b9b4-e0d77240f544: re-scheduling SchedulerAPI.create_volume attempt 1 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 15:40:21 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Volume a50e9659-3e2d-4f04-b0ac-ae78b8f61fad: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 15:40:21 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] deleating volume v_a50e9659-3e2d-4f04-b0ac-ae78b8f61fad {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 15:40:21 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] get snapshots for volume v_a50e9659-3e2d-4f04-b0ac-ae78b8f61fad {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 15:40:21 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_a50e9659-3e2d-4f04-b0ac-ae78b8f61fad/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:40:21 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_a50e9659-3e2d-4f04-b0ac-ae78b8f61fad/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:40:21 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Volume 57070f23-6b40-47b9-b9b4-e0d77240f544: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 15:40:21 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] deleating volume v_57070f23-6b40-47b9-b9b4-e0d77240f544 {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 15:40:21 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] get snapshots for volume v_57070f23-6b40-47b9-b9b4-e0d77240f544 {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 15:40:21 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_57070f23-6b40-47b9-b9b4-e0d77240f544/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:40:21 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_57070f23-6b40-47b9-b9b4-e0d77240f544/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:40:21 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Volume reschedule parameters: True retry: {'num_attempts': 2, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 15:40:21 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Flow 'volume_create_manager' (6ce3f926-71f3-4cc6-86ce-7b3b8d601821) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:40:21 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (a76e2bc9-d5d6-4c71-9ace-6a787a70e9e2) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:40:21 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (a76e2bc9-d5d6-4c71-9ace-6a787a70e9e2) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:38:02Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-CreateVolumesFromSourceVolumeTest-Volume-609446595',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=a50e9659-3e2d-4f04-b0ac-ae78b8f61fad,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='99747be727ae4592ae1b6176b488719a',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:40:21Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:40:21Z,use_quota=True,user_id='0696808ea0b14c0794e8664a37d9e2c3',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:40:21 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (a0c6b43b-6bc3-4e94-973e-6e446ec021f2) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:40:21 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (a0c6b43b-6bc3-4e94-973e-6e446ec021f2) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:40:21 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (98989c1a-5627-4523-b1dc-df8a73e72b21) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:40:21 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (98989c1a-5627-4523-b1dc-df8a73e72b21) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': 'a50e9659-3e2d-4f04-b0ac-ae78b8f61fad', 'volume_name': 'volume-a50e9659-3e2d-4f04-b0ac-ae78b8f61fad', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:40:21 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (40dd90f2-4132-4175-ba5a-9be4760756f3) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:40:21 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (40dd90f2-4132-4175-ba5a-9be4760756f3) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:40:21 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (c66f5f28-b7f0-46ea-8fa5-69e971fbfe72) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:40:21 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Volume a50e9659-3e2d-4f04-b0ac-ae78b8f61fad: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-a50e9659-3e2d-4f04-b0ac-ae78b8f61fad', 'volume_size': 1} Apr 17 15:40:21 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] creating volume v_a50e9659-3e2d-4f04-b0ac-ae78b8f61fad. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 15:40:21 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] create volume {'name': 'v_a50e9659-3e2d-4f04-b0ac-ae78b8f61fad', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 15:40:21 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:40:21 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:40:21 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Volume reschedule parameters: True retry: {'num_attempts': 2, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 15:40:21 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Flow 'volume_create_manager' (9b1b0d98-6674-45cb-9465-f6cbc1d7939e) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:40:21 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (c2ded1ae-f9cb-46ae-8906-a1b14db7dbf4) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:40:21 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (c2ded1ae-f9cb-46ae-8906-a1b14db7dbf4) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:38:02Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesActionsTest-Volume-1957591186',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=57070f23-6b40-47b9-b9b4-e0d77240f544,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='a8b97ff6ed6047f0a33e74684a1c162b',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:40:21Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:40:21Z,use_quota=True,user_id='c993ceeb4a5c49eea14057e2125518ad',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:40:21 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (f52a7c8f-8888-422c-a059-bcd722bb11a9) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:40:21 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (f52a7c8f-8888-422c-a059-bcd722bb11a9) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:40:21 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (99c4cf9e-b630-4d9c-a4ce-df04867da0f5) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:40:21 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (99c4cf9e-b630-4d9c-a4ce-df04867da0f5) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': '57070f23-6b40-47b9-b9b4-e0d77240f544', 'volume_name': 'volume-57070f23-6b40-47b9-b9b4-e0d77240f544', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:40:21 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (e0548b12-e511-42f0-8807-129eb0fc0344) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:40:21 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (e0548b12-e511-42f0-8807-129eb0fc0344) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:40:21 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (1a21b330-e2f1-48db-a8b5-679e9cc0476b) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:40:21 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Volume 57070f23-6b40-47b9-b9b4-e0d77240f544: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-57070f23-6b40-47b9-b9b4-e0d77240f544', 'volume_size': 1} Apr 17 15:40:21 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] creating volume v_57070f23-6b40-47b9-b9b4-e0d77240f544. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 15:40:21 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] create volume {'name': 'v_57070f23-6b40-47b9-b9b4-e0d77240f544', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 15:40:21 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:40:21 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:40:24 user cinder-volume[84696]: DEBUG cinder.utils [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.900(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:40:24 user cinder-volume[84696]: DEBUG cinder.utils [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_a50e9659-3e2d-4f04-b0ac-ae78b8f61fad/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:40:24 user cinder-volume[84696]: DEBUG cinder.utils [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.885(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:40:24 user cinder-volume[84696]: DEBUG cinder.utils [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_57070f23-6b40-47b9-b9b4-e0d77240f544/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:40:24 user cinder-volume[84696]: DEBUG cinder.utils [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.604(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:40:24 user cinder-volume[84696]: DEBUG cinder.utils [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:40:24 user cinder-volume[84696]: DEBUG cinder.utils [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.687(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:40:24 user cinder-volume[84696]: DEBUG cinder.utils [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:40:27 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.281(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:40:27 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:40:27 user cinder-volume[84696]: DEBUG cinder.utils [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.973(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:40:27 user cinder-volume[84696]: DEBUG cinder.utils [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_a50e9659-3e2d-4f04-b0ac-ae78b8f61fad/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:40:27 user cinder-volume[84696]: DEBUG cinder.utils [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.757(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:40:27 user cinder-volume[84696]: DEBUG cinder.utils [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:40:27 user cinder-volume[84696]: DEBUG cinder.utils [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.676(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:40:27 user cinder-volume[84696]: DEBUG cinder.utils [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:40:27 user cinder-volume[84696]: DEBUG cinder.utils [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.959(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:40:27 user cinder-volume[84696]: DEBUG cinder.utils [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_57070f23-6b40-47b9-b9b4-e0d77240f544/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:40:30 user cinder-volume[84696]: DEBUG cinder.utils [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 74.226(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:40:30 user cinder-volume[84696]: DEBUG cinder.utils [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_29d9e55e-0b8b-4f70-b5c4-830d39f10fcd/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:40:30 user cinder-volume[84696]: DEBUG cinder.utils [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 74.055(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:40:30 user cinder-volume[84696]: DEBUG cinder.utils [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:40:30 user cinder-volume[84696]: DEBUG cinder.utils [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 138.696(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:40:30 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:40:30 user cinder-volume[84696]: INFO cinder.message.api [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Creating message record for request_id = req-822bc81b-4af7-4475-af70-370e1870fe50 Apr 17 15:40:30 user cinder-volume[84696]: DEBUG cinder.utils [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 138.910(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:40:30 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_8cc35ab1-363f-4ad6-9a37-239d2b1ea996/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:40:30 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (c5b6afb4-5dd5-4a89-816d-25f24aac2e91) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 15:40:30 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (b4343f33-09c3-438a-97a1-2c34cd07593b) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:40:30 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (b4343f33-09c3-438a-97a1-2c34cd07593b) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:40:30 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Flow 'volume_create_manager' (63832f38-dba4-49a1-beef-e4709ad0af78) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 15:40:30 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 15:40:30 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 15:40:30 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 15:40:30 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 15:40:30 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 15:40:30 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 15:40:30 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 15:40:30 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 15:40:30 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 15:40:30 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 15:40:30 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 15:40:30 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 15:40:30 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 15:40:30 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 15:40:30 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 15:40:30 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 15:40:30 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 15:40:30 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 15:40:30 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 15:40:30 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 15:40:30 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 15:40:30 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 15:40:30 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 15:40:30 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 15:40:30 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:40:30 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 15:40:30 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 15:40:30 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 15:40:30 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 15:40:30 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 15:40:30 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 15:40:30 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 15:40:30 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 15:40:30 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 15:40:30 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 15:40:30 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 15:40:30 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 15:40:30 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 15:40:30 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 15:40:30 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 15:40:30 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 15:40:30 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 15:40:30 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 15:40:30 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (60237b8c-d20a-426d-9511-6fa0b66cde02) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 15:40:30 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 15:40:30 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:33:31Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-ServersNegativeTestJSON-volume-1746588226',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=8cc35ab1-363f-4ad6-9a37-239d2b1ea996,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='d2dd48f279404859bebcc19bcc272915',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:38:12Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:38:12Z,use_quota=True,user_id='b0075b26d6044a50981d29df0d863959',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 15:40:30 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:33:31Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-ServersNegativeTestJSON-volume-1746588226',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=8cc35ab1-363f-4ad6-9a37-239d2b1ea996,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='d2dd48f279404859bebcc19bcc272915',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:38:12Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:38:12Z,use_quota=True,user_id='b0075b26d6044a50981d29df0d863959',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(8cc35ab1-363f-4ad6-9a37-239d2b1ea996),volume_id=8cc35ab1-363f-4ad6-9a37-239d2b1ea996,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': '8cc35ab1-363f-4ad6-9a37-239d2b1ea996', 'volume_name': 'volume-8cc35ab1-363f-4ad6-9a37-239d2b1ea996', 'volume_size': 1}} Apr 17 15:40:30 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': '8cc35ab1-363f-4ad6-9a37-239d2b1ea996', 'volume': {'id': '8cc35ab1-363f-4ad6-9a37-239d2b1ea996', '_name_id': None, 'ec2_id': None, 'user_id': 'b0075b26d6044a50981d29df0d863959', 'project_id': 'd2dd48f279404859bebcc19bcc272915', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-ServersNegativeTestJSON-volume-1746588226', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'admin_metadata': {}, 'glance_metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T15:33:30.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-8cc35ab1-363f-4ad6-9a37-239d2b1ea996', 'name_id': '8cc35ab1-363f-4ad6-9a37-239d2b1ea996', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-ServersNegativeTestJSON-volume-1746588226', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': 'd2dd48f279404859bebcc19bcc272915', 'qos_specs': None, 'reservations': ['1161f3e2-1648-485c-8f2d-e7d723e7cefb', 'b3435402-b7a6-4cc8-ada5-e3f921682979', '131f82e8-6bdb-4b90-85b1-3d76ab20e6b8', '5d42d0e7-f4e0-4d14-83c2-a0df94279daf'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'b0075b26d6044a50981d29df0d863959', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-ServersNegativeTestJSON-volume-1746588226', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': 'd2dd48f279404859bebcc19bcc272915', 'qos_specs': None, 'reservations': ['1161f3e2-1648-485c-8f2d-e7d723e7cefb', 'b3435402-b7a6-4cc8-ada5-e3f921682979', '131f82e8-6bdb-4b90-85b1-3d76ab20e6b8', '5d42d0e7-f4e0-4d14-83c2-a0df94279daf'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'b0075b26d6044a50981d29df0d863959', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': 'b0075b26d6044a50981d29df0d863959', 'metadata': {}, 'qos_specs': None, 'retry': {'num_attempts': 3, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']}}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(8cc35ab1-363f-4ad6-9a37-239d2b1ea996),volume_id=8cc35ab1-363f-4ad6-9a37-239d2b1ea996,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:33:31Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-ServersNegativeTestJSON-volume-1746588226',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=8cc35ab1-363f-4ad6-9a37-239d2b1ea996,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='d2dd48f279404859bebcc19bcc272915',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:38:12Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:38:12Z,use_quota=True,user_id='b0075b26d6044a50981d29df0d863959',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 15:40:30 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:33:31Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-ServersNegativeTestJSON-volume-1746588226',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=8cc35ab1-363f-4ad6-9a37-239d2b1ea996,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='d2dd48f279404859bebcc19bcc272915',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:38:12Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:38:12Z,use_quota=True,user_id='b0075b26d6044a50981d29df0d863959',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:33:31Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-ServersNegativeTestJSON-volume-1746588226',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=8cc35ab1-363f-4ad6-9a37-239d2b1ea996,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='d2dd48f279404859bebcc19bcc272915',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:38:12Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:38:12Z,use_quota=True,user_id='b0075b26d6044a50981d29df0d863959',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 15:40:30 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 15:40:30 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 15:40:30 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:40:30 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 15:40:30 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 15:40:30 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 15:40:30 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 15:40:30 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 15:40:30 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 15:40:30 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 15:40:30 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 15:40:30 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 15:40:30 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 15:40:30 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 15:40:30 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 15:40:30 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 15:40:30 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 15:40:30 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 15:40:30 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 15:40:30 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 15:40:30 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (60237b8c-d20a-426d-9511-6fa0b66cde02) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:40:30 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (60237b8c-d20a-426d-9511-6fa0b66cde02) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:40:30 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (6f1f51c8-7da6-4308-8e0a-babb120e303d) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:40:30 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (6f1f51c8-7da6-4308-8e0a-babb120e303d) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:40:30 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (bf5edbae-123c-41ce-ad05-6ba36ffbed39) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:40:30 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (bf5edbae-123c-41ce-ad05-6ba36ffbed39) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:40:30 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (5bd01a1e-e653-4328-bb49-1a979b1d0544) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:40:30 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Updating volume 8cc35ab1-363f-4ad6-9a37-239d2b1ea996 with {'scheduled_at': datetime.datetime(2023, 4, 17, 15, 40, 30, 456265), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 15:40:30 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Volume 8cc35ab1-363f-4ad6-9a37-239d2b1ea996: re-scheduling SchedulerAPI.create_volume attempt 3 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 15:40:30 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Volume 8cc35ab1-363f-4ad6-9a37-239d2b1ea996: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 15:40:30 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] deleating volume v_8cc35ab1-363f-4ad6-9a37-239d2b1ea996 {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 15:40:30 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] get snapshots for volume v_8cc35ab1-363f-4ad6-9a37-239d2b1ea996 {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 15:40:30 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_8cc35ab1-363f-4ad6-9a37-239d2b1ea996/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:40:30 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_8cc35ab1-363f-4ad6-9a37-239d2b1ea996/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:40:33 user cinder-volume[84696]: DEBUG cinder.utils [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 74.061(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:40:33 user cinder-volume[84696]: DEBUG cinder.utils [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:40:33 user cinder-volume[84696]: DEBUG cinder.utils [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.960(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:40:33 user cinder-volume[84696]: DEBUG cinder.utils [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_8cc35ab1-363f-4ad6-9a37-239d2b1ea996/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:40:33 user cinder-volume[84696]: DEBUG cinder.utils [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.124(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:40:33 user cinder-volume[84696]: DEBUG cinder.utils [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_a50e9659-3e2d-4f04-b0ac-ae78b8f61fad/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:40:33 user cinder-volume[84696]: DEBUG cinder.utils [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 11.826(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:40:33 user cinder-volume[84696]: DEBUG cinder.utils [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:40:33 user cinder-volume[84696]: DEBUG cinder.utils [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 11.909(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:40:33 user cinder-volume[84696]: DEBUG cinder.utils [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:40:33 user cinder-volume[84696]: DEBUG cinder.utils [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.110(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:40:33 user cinder-volume[84696]: DEBUG cinder.utils [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_57070f23-6b40-47b9-b9b4-e0d77240f544/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:40:33 user cinder-volume[84696]: DEBUG cinder.utils [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 74.235(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:40:33 user cinder-volume[84696]: DEBUG cinder.utils [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_44073502-102d-452a-8244-69a8233b9060/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:40:33 user cinder-volume[84696]: DEBUG cinder.utils [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 136.503(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:40:33 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:40:33 user cinder-volume[84696]: INFO cinder.message.api [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Creating message record for request_id = req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b Apr 17 15:40:33 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 15:40:33 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (83d63a53-b36b-4517-ad48-a188de963f10) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 15:40:33 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 15:40:33 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:38:17Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesSnapshotTestJSON-Volume-1408369126',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=7ce815ce-9603-4fd6-8cc8-9fd798b2f99d,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='aacddeb0a9ec4a0d8d832595382e1189',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:38:17Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:38:17Z,use_quota=True,user_id='fddc0a19102c4ee7920f657f195171c9',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 15:40:33 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:38:17Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesSnapshotTestJSON-Volume-1408369126',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=7ce815ce-9603-4fd6-8cc8-9fd798b2f99d,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='aacddeb0a9ec4a0d8d832595382e1189',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:38:17Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:38:17Z,use_quota=True,user_id='fddc0a19102c4ee7920f657f195171c9',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(7ce815ce-9603-4fd6-8cc8-9fd798b2f99d),volume_id=7ce815ce-9603-4fd6-8cc8-9fd798b2f99d,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': '7ce815ce-9603-4fd6-8cc8-9fd798b2f99d', 'volume_name': 'volume-7ce815ce-9603-4fd6-8cc8-9fd798b2f99d', 'volume_size': 1}} Apr 17 15:40:33 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'retry': {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']}, 'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': '7ce815ce-9603-4fd6-8cc8-9fd798b2f99d', 'volume': {'id': '7ce815ce-9603-4fd6-8cc8-9fd798b2f99d', '_name_id': None, 'ec2_id': None, 'user_id': 'fddc0a19102c4ee7920f657f195171c9', 'project_id': 'aacddeb0a9ec4a0d8d832595382e1189', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-VolumesSnapshotTestJSON-Volume-1408369126', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T15:38:16.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-7ce815ce-9603-4fd6-8cc8-9fd798b2f99d', 'name_id': '7ce815ce-9603-4fd6-8cc8-9fd798b2f99d', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumesSnapshotTestJSON-Volume-1408369126', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': 'aacddeb0a9ec4a0d8d832595382e1189', 'qos_specs': None, 'reservations': ['f313e2f7-2050-44c4-b79e-0ecee302728c', '287d06a9-6c9c-4bb1-ba13-85b7cf2d8b7e', '14fdea43-dc14-4800-b0fe-8defcfbc5c61', '5659b6c2-9b08-423d-bdcb-0a1bab01869d'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'fddc0a19102c4ee7920f657f195171c9', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumesSnapshotTestJSON-Volume-1408369126', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': 'aacddeb0a9ec4a0d8d832595382e1189', 'qos_specs': None, 'reservations': ['f313e2f7-2050-44c4-b79e-0ecee302728c', '287d06a9-6c9c-4bb1-ba13-85b7cf2d8b7e', '14fdea43-dc14-4800-b0fe-8defcfbc5c61', '5659b6c2-9b08-423d-bdcb-0a1bab01869d'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'fddc0a19102c4ee7920f657f195171c9', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': 'fddc0a19102c4ee7920f657f195171c9', 'metadata': {}, 'qos_specs': None}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(7ce815ce-9603-4fd6-8cc8-9fd798b2f99d),volume_id=7ce815ce-9603-4fd6-8cc8-9fd798b2f99d,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:38:17Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesSnapshotTestJSON-Volume-1408369126',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=7ce815ce-9603-4fd6-8cc8-9fd798b2f99d,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='aacddeb0a9ec4a0d8d832595382e1189',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:38:17Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:38:17Z,use_quota=True,user_id='fddc0a19102c4ee7920f657f195171c9',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 15:40:33 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:38:17Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesSnapshotTestJSON-Volume-1408369126',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=7ce815ce-9603-4fd6-8cc8-9fd798b2f99d,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='aacddeb0a9ec4a0d8d832595382e1189',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:38:17Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:38:17Z,use_quota=True,user_id='fddc0a19102c4ee7920f657f195171c9',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:38:17Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesSnapshotTestJSON-Volume-1408369126',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=7ce815ce-9603-4fd6-8cc8-9fd798b2f99d,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='aacddeb0a9ec4a0d8d832595382e1189',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:38:17Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:38:17Z,use_quota=True,user_id='fddc0a19102c4ee7920f657f195171c9',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 15:40:33 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 15:40:33 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 15:40:33 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:40:33 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 15:40:33 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 15:40:33 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 15:40:33 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 15:40:33 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 15:40:33 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 15:40:33 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 15:40:33 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 15:40:33 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 15:40:33 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 15:40:33 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 15:40:33 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 15:40:33 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 15:40:33 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 15:40:33 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 15:40:33 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 15:40:33 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 15:40:33 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (83d63a53-b36b-4517-ad48-a188de963f10) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:40:33 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (83d63a53-b36b-4517-ad48-a188de963f10) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:40:33 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (f6e9574c-c118-4aaa-a5fb-f10ae4e68fc7) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:40:33 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (f6e9574c-c118-4aaa-a5fb-f10ae4e68fc7) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:40:33 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (8d0ff060-f3a7-4bb1-ba43-2fd884642faa) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:40:33 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (8d0ff060-f3a7-4bb1-ba43-2fd884642faa) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:40:33 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (6b63b69f-daf6-4306-9c39-dcb57b647423) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:40:33 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Updating volume 7ce815ce-9603-4fd6-8cc8-9fd798b2f99d with {'scheduled_at': datetime.datetime(2023, 4, 17, 15, 40, 33, 512063), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 15:40:33 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Volume 7ce815ce-9603-4fd6-8cc8-9fd798b2f99d: re-scheduling SchedulerAPI.create_volume attempt 1 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 15:40:33 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Volume 7ce815ce-9603-4fd6-8cc8-9fd798b2f99d: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 15:40:33 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] deleating volume v_7ce815ce-9603-4fd6-8cc8-9fd798b2f99d {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 15:40:33 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] get snapshots for volume v_7ce815ce-9603-4fd6-8cc8-9fd798b2f99d {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 15:40:33 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_7ce815ce-9603-4fd6-8cc8-9fd798b2f99d/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:40:33 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_7ce815ce-9603-4fd6-8cc8-9fd798b2f99d/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:40:33 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Volume reschedule parameters: True retry: {'num_attempts': 2, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 15:40:33 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Flow 'volume_create_manager' (0a67e980-27da-43b1-b2f3-13d324bdb9a5) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:40:33 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (5a9b725b-64e1-4f2b-9db9-01fb473d3018) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:40:33 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (5a9b725b-64e1-4f2b-9db9-01fb473d3018) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:38:17Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesSnapshotTestJSON-Volume-1408369126',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=7ce815ce-9603-4fd6-8cc8-9fd798b2f99d,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='aacddeb0a9ec4a0d8d832595382e1189',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:40:34Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:40:34Z,use_quota=True,user_id='fddc0a19102c4ee7920f657f195171c9',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:40:33 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (018eee5d-5087-4932-97d7-887e2a16ecd9) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:40:33 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (018eee5d-5087-4932-97d7-887e2a16ecd9) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:40:33 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (f95baaef-aff1-42f9-a9bb-b8e18cec8961) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:40:33 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (f95baaef-aff1-42f9-a9bb-b8e18cec8961) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': '7ce815ce-9603-4fd6-8cc8-9fd798b2f99d', 'volume_name': 'volume-7ce815ce-9603-4fd6-8cc8-9fd798b2f99d', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:40:33 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (74005d33-c134-4cca-8c1a-d44af556cede) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:40:33 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (74005d33-c134-4cca-8c1a-d44af556cede) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:40:33 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (6f9fb967-ca8d-4eba-832e-081aac180111) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:40:33 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Volume 7ce815ce-9603-4fd6-8cc8-9fd798b2f99d: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-7ce815ce-9603-4fd6-8cc8-9fd798b2f99d', 'volume_size': 1} Apr 17 15:40:33 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] creating volume v_7ce815ce-9603-4fd6-8cc8-9fd798b2f99d. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 15:40:33 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] create volume {'name': 'v_7ce815ce-9603-4fd6-8cc8-9fd798b2f99d', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 15:40:33 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:40:33 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:40:36 user cinder-volume[84696]: DEBUG cinder.utils [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.806(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:40:36 user cinder-volume[84696]: DEBUG cinder.utils [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:40:36 user cinder-volume[84696]: DEBUG cinder.utils [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 138.887(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:40:36 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_5d9c86cd-3e22-4926-bfc9-e5a37f6ae76c/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:40:36 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (b04285b7-aa2c-4e25-91ee-7eaa61ab5ad3) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 15:40:36 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (3d64220f-2ad6-4808-add0-ce29ee542b1c) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:40:36 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (3d64220f-2ad6-4808-add0-ce29ee542b1c) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:40:36 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Flow 'volume_create_manager' (1de8cec6-3135-4047-a7e1-e72cadc5bdd2) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 15:40:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-a9831ecf-bf39-4cfe-ae2d-314dbfb31318 tempest-VolumesServicesTestJSON-1183731809 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 15:40:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 15:40:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 15:40:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 15:40:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 15:40:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 15:40:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 15:40:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 15:40:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 15:40:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 15:40:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 15:40:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 15:40:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 15:40:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 15:40:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 15:40:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 15:40:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 15:40:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 15:40:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 15:40:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 15:40:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 15:40:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 15:40:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 15:40:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 15:40:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:40:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 15:40:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 15:40:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 15:40:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 15:40:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 15:40:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 15:40:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 15:40:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 15:40:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 15:40:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 15:40:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 15:40:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 15:40:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 15:40:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 15:40:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 15:40:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 15:40:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 15:40:36 user cinder-volume[84696]: DEBUG cinder.utils [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.046(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:40:36 user cinder-volume[84696]: DEBUG cinder.utils [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_8cc35ab1-363f-4ad6-9a37-239d2b1ea996/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:40:36 user cinder-volume[84696]: DEBUG cinder.utils [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.990(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:40:36 user cinder-volume[84696]: DEBUG cinder.utils [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_7ce815ce-9603-4fd6-8cc8-9fd798b2f99d/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:40:36 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.519(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:40:36 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:40:39 user cinder-volume[84696]: DEBUG cinder.utils [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.041(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:40:39 user cinder-volume[84696]: DEBUG cinder.utils [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_7ce815ce-9603-4fd6-8cc8-9fd798b2f99d/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:40:39 user cinder-volume[84696]: DEBUG cinder.utils [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.878(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:40:39 user cinder-volume[84696]: DEBUG cinder.utils [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:40:42 user cinder-volume[84696]: DEBUG cinder.utils [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.170(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:40:42 user cinder-volume[84696]: DEBUG cinder.utils [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_8cc35ab1-363f-4ad6-9a37-239d2b1ea996/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:40:42 user cinder-volume[84696]: DEBUG cinder.utils [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.118(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:40:42 user cinder-volume[84696]: DEBUG cinder.utils [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:40:42 user cinder-volume[84696]: DEBUG cinder.utils [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.037(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:40:42 user cinder-volume[84696]: DEBUG cinder.utils [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:40:42 user cinder-volume[84696]: DEBUG cinder.utils [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.337(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:40:42 user cinder-volume[84696]: DEBUG cinder.utils [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_a50e9659-3e2d-4f04-b0ac-ae78b8f61fad/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:40:42 user cinder-volume[84696]: DEBUG cinder.utils [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.320(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:40:42 user cinder-volume[84696]: DEBUG cinder.utils [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_57070f23-6b40-47b9-b9b4-e0d77240f544/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:40:44 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=19,cinder:UPDATE=9,cinder:INSERT=4 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:40:44 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=26,cinder:UPDATE=6,cinder:INSERT=4 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:40:45 user cinder-volume[84696]: DEBUG cinder.utils [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.189(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:40:45 user cinder-volume[84696]: DEBUG cinder.utils [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_7ce815ce-9603-4fd6-8cc8-9fd798b2f99d/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:40:45 user cinder-volume[84696]: DEBUG cinder.utils [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 74.206(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:40:45 user cinder-volume[84696]: DEBUG cinder.utils [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_153b4df8-1f37-4118-8aa6-1c155ea5700c/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:40:45 user cinder-volume[84696]: DEBUG cinder.utils [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 73.998(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:40:45 user cinder-volume[84696]: DEBUG cinder.utils [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:40:45 user cinder-volume[84696]: DEBUG cinder.utils [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.027(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:40:45 user cinder-volume[84696]: DEBUG cinder.utils [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:40:48 user cinder-volume[84696]: DEBUG cinder.utils [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 138.861(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:40:48 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_44f909cf-7774-41f6-9f86-805456609ff8/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:40:48 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (ad21cf54-42e6-4d2d-bbc8-2aa4dc0604d7) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 15:40:48 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (a2daede2-93f4-4c4f-9805-dbb1f304ff7f) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:40:48 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (a2daede2-93f4-4c4f-9805-dbb1f304ff7f) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:40:48 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Flow 'volume_create_manager' (d15433fb-3b6e-47b8-8df1-aa4f8b82cd32) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 15:40:48 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-9c9edd59-8934-430a-a1fa-77dd5ef4d4df tempest-VolumesActionsTest-707453245 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 15:40:48 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 15:40:48 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 15:40:48 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 15:40:48 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 15:40:48 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 15:40:48 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 15:40:48 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 15:40:48 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 15:40:48 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 15:40:48 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 15:40:48 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 15:40:48 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 15:40:48 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 15:40:48 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 15:40:48 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 15:40:48 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 15:40:48 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 15:40:48 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 15:40:48 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 15:40:48 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 15:40:48 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 15:40:48 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 15:40:48 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 15:40:48 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:40:48 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 15:40:48 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 15:40:48 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 15:40:48 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 15:40:48 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 15:40:48 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 15:40:48 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 15:40:48 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 15:40:48 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 15:40:48 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 15:40:48 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 15:40:48 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 15:40:48 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 15:40:48 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 15:40:48 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 15:40:48 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 15:40:48 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 15:40:51 user cinder-volume[84696]: DEBUG cinder.utils [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.385(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:40:51 user cinder-volume[84696]: DEBUG cinder.utils [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_8cc35ab1-363f-4ad6-9a37-239d2b1ea996/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:40:54 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.928(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:40:54 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:40:54 user cinder-volume[84696]: DEBUG cinder.utils [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.237(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:40:54 user cinder-volume[84696]: DEBUG cinder.utils [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:40:54 user cinder-volume[84696]: DEBUG cinder.utils [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.403(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:40:54 user cinder-volume[84696]: DEBUG cinder.utils [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_7ce815ce-9603-4fd6-8cc8-9fd798b2f99d/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:40:58 user cinder-volume[84696]: DEBUG cinder.utils [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 138.920(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:40:58 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_0fed6b21-0520-4554-bee4-9540a03f9af0/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:40:58 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (08b4e6dc-1ce7-49f6-b5aa-72522a91a536) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 15:40:58 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (73e195f3-6579-445a-9aeb-28f7098f6f4d) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:40:58 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (73e195f3-6579-445a-9aeb-28f7098f6f4d) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:40:58 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Flow 'volume_create_manager' (cce6b286-e2af-4a14-85bc-d1251e04b47e) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 15:40:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-d0269988-5f86-4167-b5dd-78432d84593a tempest-VolumesCloneTest-1847825601 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 15:40:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 15:40:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 15:40:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 15:40:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 15:40:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 15:40:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 15:40:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 15:40:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 15:40:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 15:40:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 15:40:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 15:40:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 15:40:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 15:40:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 15:40:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 15:40:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 15:40:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 15:40:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 15:40:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 15:40:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 15:40:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 15:40:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 15:40:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 15:40:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:40:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 15:40:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1237, in execute Apr 17 15:40:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_from_image(context, Apr 17 15:40:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/utils.py", line 670, in _wrapper Apr 17 15:40:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return r(f, *args, **kwargs) Apr 17 15:40:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/tenacity/__init__.py", line 423, in __call__ Apr 17 15:40:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server do = self.iter(retry_state=retry_state) Apr 17 15:40:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/tenacity/__init__.py", line 360, in iter Apr 17 15:40:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return fut.result() Apr 17 15:40:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/lib/python3.10/concurrent/futures/_base.py", line 451, in result Apr 17 15:40:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self.__get_result() Apr 17 15:40:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result Apr 17 15:40:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self._exception Apr 17 15:40:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/tenacity/__init__.py", line 426, in __call__ Apr 17 15:40:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = fn(*args, **kwargs) Apr 17 15:40:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1125, in _create_from_image Apr 17 15:40:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_from_image_cache_or_download( Apr 17 15:40:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 944, in _create_from_image_cache_or_download Apr 17 15:40:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update, cloned = self._create_from_image_cache( Apr 17 15:40:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 843, in _create_from_image_cache Apr 17 15:40:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_from_source_volume( Apr 17 15:40:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 659, in _create_from_source_volume Apr 17 15:40:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self.driver.create_cloned_volume(volume, srcvol_ref) Apr 17 15:40:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 525, in create_cloned_volume Apr 17 15:40:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self._clone_object(vname, cvname) Apr 17 15:40:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 461, in _clone_object Apr 17 15:40:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_snapshot(oname, coname) Apr 17 15:40:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 639, in create_snapshot Apr 17 15:40:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201, 204): Apr 17 15:40:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 15:40:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 15:41:01 user cinder-volume[84696]: DEBUG cinder.utils [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.547(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:41:01 user cinder-volume[84696]: DEBUG cinder.utils [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:41:01 user cinder-volume[84696]: DEBUG cinder.utils [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.467(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:41:01 user cinder-volume[84696]: DEBUG cinder.utils [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:41:01 user cinder-volume[84696]: DEBUG cinder.utils [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.750(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:41:01 user cinder-volume[84696]: DEBUG cinder.utils [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_57070f23-6b40-47b9-b9b4-e0d77240f544/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:41:01 user cinder-volume[84696]: DEBUG cinder.utils [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.768(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:41:01 user cinder-volume[84696]: DEBUG cinder.utils [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_a50e9659-3e2d-4f04-b0ac-ae78b8f61fad/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:41:04 user cinder-volume[84696]: DEBUG cinder.utils [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 136.818(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:41:04 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:41:04 user cinder-volume[84696]: INFO cinder.message.api [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Creating message record for request_id = req-52df28ef-59c1-47f6-831b-cf97d2a31df8 Apr 17 15:41:04 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 15:41:04 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (10330e8b-f1b7-43a7-9551-0888eeb88387) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 15:41:04 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 15:41:04 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:38:47Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-ServersNegativeTestJSON-volume-2082564469',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=a27d0ff3-5de9-4feb-82f7-d4571af88bb9,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='d2dd48f279404859bebcc19bcc272915',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:38:47Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:38:47Z,use_quota=True,user_id='b0075b26d6044a50981d29df0d863959',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 15:41:04 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:38:47Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-ServersNegativeTestJSON-volume-2082564469',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=a27d0ff3-5de9-4feb-82f7-d4571af88bb9,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='d2dd48f279404859bebcc19bcc272915',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:38:47Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:38:47Z,use_quota=True,user_id='b0075b26d6044a50981d29df0d863959',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(a27d0ff3-5de9-4feb-82f7-d4571af88bb9),volume_id=a27d0ff3-5de9-4feb-82f7-d4571af88bb9,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': 'a27d0ff3-5de9-4feb-82f7-d4571af88bb9', 'volume_name': 'volume-a27d0ff3-5de9-4feb-82f7-d4571af88bb9', 'volume_size': 1}} Apr 17 15:41:04 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'retry': {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']}, 'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': 'a27d0ff3-5de9-4feb-82f7-d4571af88bb9', 'volume': {'id': 'a27d0ff3-5de9-4feb-82f7-d4571af88bb9', '_name_id': None, 'ec2_id': None, 'user_id': 'b0075b26d6044a50981d29df0d863959', 'project_id': 'd2dd48f279404859bebcc19bcc272915', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-ServersNegativeTestJSON-volume-2082564469', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T15:38:47.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-a27d0ff3-5de9-4feb-82f7-d4571af88bb9', 'name_id': 'a27d0ff3-5de9-4feb-82f7-d4571af88bb9', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-ServersNegativeTestJSON-volume-2082564469', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': 'd2dd48f279404859bebcc19bcc272915', 'qos_specs': None, 'reservations': ['5c815076-6ae9-452c-aa1e-9825025560c9', 'b96b1904-2fdf-4b5b-9c8d-61562d250d61', '569a260d-012a-4ff0-b9af-61caad629af7', 'c933f679-b335-4a11-b551-a5c307704f2e'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'b0075b26d6044a50981d29df0d863959', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-ServersNegativeTestJSON-volume-2082564469', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': 'd2dd48f279404859bebcc19bcc272915', 'qos_specs': None, 'reservations': ['5c815076-6ae9-452c-aa1e-9825025560c9', 'b96b1904-2fdf-4b5b-9c8d-61562d250d61', '569a260d-012a-4ff0-b9af-61caad629af7', 'c933f679-b335-4a11-b551-a5c307704f2e'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'b0075b26d6044a50981d29df0d863959', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': 'b0075b26d6044a50981d29df0d863959', 'metadata': {}, 'qos_specs': None}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(a27d0ff3-5de9-4feb-82f7-d4571af88bb9),volume_id=a27d0ff3-5de9-4feb-82f7-d4571af88bb9,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:38:47Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-ServersNegativeTestJSON-volume-2082564469',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=a27d0ff3-5de9-4feb-82f7-d4571af88bb9,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='d2dd48f279404859bebcc19bcc272915',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:38:47Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:38:47Z,use_quota=True,user_id='b0075b26d6044a50981d29df0d863959',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 15:41:04 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:38:47Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-ServersNegativeTestJSON-volume-2082564469',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=a27d0ff3-5de9-4feb-82f7-d4571af88bb9,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='d2dd48f279404859bebcc19bcc272915',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:38:47Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:38:47Z,use_quota=True,user_id='b0075b26d6044a50981d29df0d863959',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:38:47Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-ServersNegativeTestJSON-volume-2082564469',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=a27d0ff3-5de9-4feb-82f7-d4571af88bb9,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='d2dd48f279404859bebcc19bcc272915',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:38:47Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:38:47Z,use_quota=True,user_id='b0075b26d6044a50981d29df0d863959',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 15:41:04 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 15:41:04 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 15:41:04 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:41:04 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 15:41:04 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 15:41:04 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 15:41:04 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 15:41:04 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 15:41:04 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 15:41:04 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 15:41:04 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 15:41:04 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 15:41:04 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 15:41:04 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 15:41:04 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 15:41:04 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 15:41:04 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 15:41:04 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 15:41:04 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 15:41:04 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 15:41:04 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (10330e8b-f1b7-43a7-9551-0888eeb88387) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:41:04 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (10330e8b-f1b7-43a7-9551-0888eeb88387) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:41:04 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (1de34f58-5d68-47ef-93aa-81234c771231) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:41:04 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (1de34f58-5d68-47ef-93aa-81234c771231) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:41:04 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (f8e2c763-d9fc-47d6-a6f5-0ebd48e8f47b) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:41:04 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (f8e2c763-d9fc-47d6-a6f5-0ebd48e8f47b) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:41:04 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (314a2907-5948-4ac0-85b7-c7ec1690c5b5) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:41:04 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Updating volume a27d0ff3-5de9-4feb-82f7-d4571af88bb9 with {'scheduled_at': datetime.datetime(2023, 4, 17, 15, 41, 4, 221053), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 15:41:04 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Volume a27d0ff3-5de9-4feb-82f7-d4571af88bb9: re-scheduling SchedulerAPI.create_volume attempt 1 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 15:41:04 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Volume a27d0ff3-5de9-4feb-82f7-d4571af88bb9: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 15:41:04 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] deleating volume v_a27d0ff3-5de9-4feb-82f7-d4571af88bb9 {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 15:41:04 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] get snapshots for volume v_a27d0ff3-5de9-4feb-82f7-d4571af88bb9 {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 15:41:04 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_a27d0ff3-5de9-4feb-82f7-d4571af88bb9/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:41:04 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_a27d0ff3-5de9-4feb-82f7-d4571af88bb9/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:41:04 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Volume reschedule parameters: True retry: {'num_attempts': 2, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 15:41:04 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Flow 'volume_create_manager' (a9a0f28b-5a79-46cf-a810-f69068e92f0a) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:41:04 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (50a78af3-ba39-4893-a01a-240aa1cf974f) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:41:04 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (50a78af3-ba39-4893-a01a-240aa1cf974f) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:38:47Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-ServersNegativeTestJSON-volume-2082564469',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=a27d0ff3-5de9-4feb-82f7-d4571af88bb9,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='d2dd48f279404859bebcc19bcc272915',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:41:04Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:41:04Z,use_quota=True,user_id='b0075b26d6044a50981d29df0d863959',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:41:04 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (88bc4781-d148-4199-87b5-ca7f3c1d508f) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:41:04 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (88bc4781-d148-4199-87b5-ca7f3c1d508f) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:41:04 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (2b8ca4fa-2bf5-40a7-af8c-19e1e1cabd0b) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:41:04 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (2b8ca4fa-2bf5-40a7-af8c-19e1e1cabd0b) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': 'a27d0ff3-5de9-4feb-82f7-d4571af88bb9', 'volume_name': 'volume-a27d0ff3-5de9-4feb-82f7-d4571af88bb9', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:41:04 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (91ad3650-32d3-4064-a30b-91e6b806aee8) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:41:04 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (91ad3650-32d3-4064-a30b-91e6b806aee8) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:41:04 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (f299f18a-17e0-4661-9805-d82f9894a488) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:41:04 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Volume a27d0ff3-5de9-4feb-82f7-d4571af88bb9: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-a27d0ff3-5de9-4feb-82f7-d4571af88bb9', 'volume_size': 1} Apr 17 15:41:04 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] creating volume v_a27d0ff3-5de9-4feb-82f7-d4571af88bb9. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 15:41:04 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] create volume {'name': 'v_a27d0ff3-5de9-4feb-82f7-d4571af88bb9', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 15:41:04 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:41:04 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:41:07 user cinder-volume[84696]: DEBUG cinder.utils [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.980(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:41:07 user cinder-volume[84696]: DEBUG cinder.utils [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_a27d0ff3-5de9-4feb-82f7-d4571af88bb9/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:41:07 user cinder-volume[84696]: DEBUG cinder.utils [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.751(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:41:07 user cinder-volume[84696]: DEBUG cinder.utils [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:41:10 user cinder-volume[84696]: DEBUG cinder.utils [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.821(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:41:10 user cinder-volume[84696]: DEBUG cinder.utils [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_8cc35ab1-363f-4ad6-9a37-239d2b1ea996/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:41:10 user cinder-volume[84696]: DEBUG cinder.utils [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.826(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:41:10 user cinder-volume[84696]: DEBUG cinder.utils [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:41:10 user cinder-volume[84696]: DEBUG cinder.utils [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.058(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:41:10 user cinder-volume[84696]: DEBUG cinder.utils [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_a27d0ff3-5de9-4feb-82f7-d4571af88bb9/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:41:13 user cinder-volume[84696]: DEBUG cinder.utils [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 138.563(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:41:13 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:41:13 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (36490734-f2a7-4d3f-81d9-412a304fb3bb) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 15:41:13 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 15:41:13 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:34:17Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TaggedBootDevicesTest-volume-1099737906',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=e80c78b2-20c9-4f9b-8898-6ab131d1eb06,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='75ab8873f7064d3fbe954746c071cb72',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:38:55Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:38:55Z,use_quota=True,user_id='39328f4ed6da4a85ac312bc5637b1500',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 15:41:13 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:34:17Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TaggedBootDevicesTest-volume-1099737906',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=e80c78b2-20c9-4f9b-8898-6ab131d1eb06,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='75ab8873f7064d3fbe954746c071cb72',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:38:55Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:38:55Z,use_quota=True,user_id='39328f4ed6da4a85ac312bc5637b1500',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=b3f0dec1-7758-44a8-82e0-76ada8837428,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(e80c78b2-20c9-4f9b-8898-6ab131d1eb06),volume_id=e80c78b2-20c9-4f9b-8898-6ab131d1eb06,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'image', 'volume_id': 'e80c78b2-20c9-4f9b-8898-6ab131d1eb06', 'volume_name': 'volume-e80c78b2-20c9-4f9b-8898-6ab131d1eb06', 'volume_size': 1, 'image_id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'image_location': (None, None), 'image_meta': {'name': 'cirros-0.5.2-x86_64-disk', 'disk_format': 'qcow2', 'container_format': 'bare', 'visibility': 'public', 'size': 16300544, 'virtual_size': 117440512, 'status': 'active', 'checksum': 'b874c39491a2377b8490f5f1e89761a4', 'protected': False, 'min_ram': 0, 'min_disk': 0, 'owner': 'a43c211b4ed542538799349489b1d884', 'os_hidden': False, 'os_hash_algo': 'sha512', 'os_hash_value': '6b813aa46bb90b4da216a4d19376593fa3f4fc7e617f03a92b7fe11e9a3981cbe8f0959dbebe36225e5f53dc4492341a4863cac4ed1ee0909f3fc78ef9c3e869', 'id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'created_at': datetime.datetime(2023, 4, 17, 15, 19, 53, tzinfo=datetime.timezone.utc), 'updated_at': datetime.datetime(2023, 4, 17, 15, 19, 55, tzinfo=datetime.timezone.utc), 'tags': [], 'file': '/v2/images/b3f0dec1-7758-44a8-82e0-76ada8837428/file', 'properties': {'hw_rng_model': 'virtio', 'owner_specified.openstack.md5': '', 'owner_specified.openstack.object': 'images/cirros-0.5.2-x86_64-disk', 'owner_specified.openstack.sha256': ''}}, 'image_service': }} Apr 17 15:41:13 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'snapshot_id': None, 'source_volid': None, 'volume_id': 'e80c78b2-20c9-4f9b-8898-6ab131d1eb06', 'volume': {'id': 'e80c78b2-20c9-4f9b-8898-6ab131d1eb06', '_name_id': None, 'ec2_id': None, 'user_id': '39328f4ed6da4a85ac312bc5637b1500', 'project_id': '75ab8873f7064d3fbe954746c071cb72', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-TaggedBootDevicesTest-volume-1099737906', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'admin_metadata': {}, 'glance_metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T15:34:16.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-e80c78b2-20c9-4f9b-8898-6ab131d1eb06', 'name_id': 'e80c78b2-20c9-4f9b-8898-6ab131d1eb06', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-TaggedBootDevicesTest-volume-1099737906', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '75ab8873f7064d3fbe954746c071cb72', 'qos_specs': None, 'reservations': ['d6688115-7e3d-438c-8fc0-69b8ec7a1cde', '8ca9607c-d2b9-47d7-8e24-f752a5ed3e59', 'a110fbe0-e00b-4d63-a1a6-1c96ea375721', '06947137-4319-40d9-b8a3-4db09cd63ce4'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': '39328f4ed6da4a85ac312bc5637b1500', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-TaggedBootDevicesTest-volume-1099737906', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '75ab8873f7064d3fbe954746c071cb72', 'qos_specs': None, 'reservations': ['d6688115-7e3d-438c-8fc0-69b8ec7a1cde', '8ca9607c-d2b9-47d7-8e24-f752a5ed3e59', 'a110fbe0-e00b-4d63-a1a6-1c96ea375721', '06947137-4319-40d9-b8a3-4db09cd63ce4'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': '39328f4ed6da4a85ac312bc5637b1500', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': '39328f4ed6da4a85ac312bc5637b1500', 'metadata': {}, 'qos_specs': None, 'retry': {'num_attempts': 3, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']}}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=b3f0dec1-7758-44a8-82e0-76ada8837428,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(e80c78b2-20c9-4f9b-8898-6ab131d1eb06),volume_id=e80c78b2-20c9-4f9b-8898-6ab131d1eb06,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:34:17Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TaggedBootDevicesTest-volume-1099737906',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=e80c78b2-20c9-4f9b-8898-6ab131d1eb06,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='75ab8873f7064d3fbe954746c071cb72',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:38:55Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:38:55Z,use_quota=True,user_id='39328f4ed6da4a85ac312bc5637b1500',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 15:41:13 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:34:17Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TaggedBootDevicesTest-volume-1099737906',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=e80c78b2-20c9-4f9b-8898-6ab131d1eb06,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='75ab8873f7064d3fbe954746c071cb72',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:38:55Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:38:55Z,use_quota=True,user_id='39328f4ed6da4a85ac312bc5637b1500',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:34:17Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TaggedBootDevicesTest-volume-1099737906',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=e80c78b2-20c9-4f9b-8898-6ab131d1eb06,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='75ab8873f7064d3fbe954746c071cb72',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:38:55Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:38:55Z,use_quota=True,user_id='39328f4ed6da4a85ac312bc5637b1500',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 15:41:13 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 15:41:13 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 15:41:13 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:41:13 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 15:41:13 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1237, in execute Apr 17 15:41:13 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_from_image(context, Apr 17 15:41:13 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/utils.py", line 670, in _wrapper Apr 17 15:41:13 user cinder-volume[84696]: ERROR cinder.volume.manager return r(f, *args, **kwargs) Apr 17 15:41:13 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/tenacity/__init__.py", line 423, in __call__ Apr 17 15:41:13 user cinder-volume[84696]: ERROR cinder.volume.manager do = self.iter(retry_state=retry_state) Apr 17 15:41:13 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/tenacity/__init__.py", line 360, in iter Apr 17 15:41:13 user cinder-volume[84696]: ERROR cinder.volume.manager return fut.result() Apr 17 15:41:13 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/lib/python3.10/concurrent/futures/_base.py", line 451, in result Apr 17 15:41:13 user cinder-volume[84696]: ERROR cinder.volume.manager return self.__get_result() Apr 17 15:41:13 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result Apr 17 15:41:13 user cinder-volume[84696]: ERROR cinder.volume.manager raise self._exception Apr 17 15:41:13 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/tenacity/__init__.py", line 426, in __call__ Apr 17 15:41:13 user cinder-volume[84696]: ERROR cinder.volume.manager result = fn(*args, **kwargs) Apr 17 15:41:13 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1125, in _create_from_image Apr 17 15:41:13 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_from_image_cache_or_download( Apr 17 15:41:13 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 944, in _create_from_image_cache_or_download Apr 17 15:41:13 user cinder-volume[84696]: ERROR cinder.volume.manager model_update, cloned = self._create_from_image_cache( Apr 17 15:41:13 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 843, in _create_from_image_cache Apr 17 15:41:13 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_from_source_volume( Apr 17 15:41:13 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 659, in _create_from_source_volume Apr 17 15:41:13 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self.driver.create_cloned_volume(volume, srcvol_ref) Apr 17 15:41:13 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 525, in create_cloned_volume Apr 17 15:41:13 user cinder-volume[84696]: ERROR cinder.volume.manager self._clone_object(vname, cvname) Apr 17 15:41:13 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 461, in _clone_object Apr 17 15:41:13 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_snapshot(oname, coname) Apr 17 15:41:13 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 639, in create_snapshot Apr 17 15:41:13 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201, 204): Apr 17 15:41:13 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 15:41:13 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 15:41:13 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (36490734-f2a7-4d3f-81d9-412a304fb3bb) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:41:13 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (36490734-f2a7-4d3f-81d9-412a304fb3bb) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:41:13 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (37a3045e-bf49-40c1-bd38-69ade0379faa) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:41:13 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (37a3045e-bf49-40c1-bd38-69ade0379faa) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:41:13 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (3824494b-4da4-4a57-b785-2f6d29ed7b48) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:41:13 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (3824494b-4da4-4a57-b785-2f6d29ed7b48) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:41:13 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (a9adefa1-d40c-47d2-91d7-eca949dad911) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:41:13 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Updating volume e80c78b2-20c9-4f9b-8898-6ab131d1eb06 with {'scheduled_at': datetime.datetime(2023, 4, 17, 15, 41, 13, 437977), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 15:41:13 user cinder-volume[84696]: DEBUG cinder.utils [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.902(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:41:13 user cinder-volume[84696]: DEBUG cinder.utils [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_7ce815ce-9603-4fd6-8cc8-9fd798b2f99d/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:41:13 user cinder-volume[84696]: DEBUG cinder.utils [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.739(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:41:13 user cinder-volume[84696]: DEBUG cinder.utils [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:41:13 user cinder-volume[84696]: DEBUG cinder.utils [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 138.976(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:41:13 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_e80c78b2-20c9-4f9b-8898-6ab131d1eb06/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:41:13 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (fe973769-6b51-4a99-b47e-a6ce4a1b7f6a) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 15:41:13 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (085f0fc6-3c9a-444a-9e38-9707c821a736) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:41:13 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (085f0fc6-3c9a-444a-9e38-9707c821a736) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:41:13 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Flow 'volume_create_manager' (48ffe1e2-23fc-4875-b092-e00508064396) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 15:41:13 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 15:41:13 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 15:41:13 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 15:41:13 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 15:41:13 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 15:41:13 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 15:41:13 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 15:41:13 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 15:41:13 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 15:41:13 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 15:41:13 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 15:41:13 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 15:41:13 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 15:41:13 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 15:41:13 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 15:41:13 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 15:41:13 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 15:41:13 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 15:41:13 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 15:41:13 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 15:41:13 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 15:41:13 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 15:41:13 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 15:41:13 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 15:41:13 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:41:13 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 15:41:13 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1237, in execute Apr 17 15:41:13 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_from_image(context, Apr 17 15:41:13 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/utils.py", line 670, in _wrapper Apr 17 15:41:13 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return r(f, *args, **kwargs) Apr 17 15:41:13 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/tenacity/__init__.py", line 423, in __call__ Apr 17 15:41:13 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server do = self.iter(retry_state=retry_state) Apr 17 15:41:13 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/tenacity/__init__.py", line 360, in iter Apr 17 15:41:13 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return fut.result() Apr 17 15:41:13 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/lib/python3.10/concurrent/futures/_base.py", line 451, in result Apr 17 15:41:13 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self.__get_result() Apr 17 15:41:13 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result Apr 17 15:41:13 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self._exception Apr 17 15:41:13 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/tenacity/__init__.py", line 426, in __call__ Apr 17 15:41:13 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = fn(*args, **kwargs) Apr 17 15:41:13 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1125, in _create_from_image Apr 17 15:41:13 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_from_image_cache_or_download( Apr 17 15:41:13 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 944, in _create_from_image_cache_or_download Apr 17 15:41:13 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update, cloned = self._create_from_image_cache( Apr 17 15:41:13 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 843, in _create_from_image_cache Apr 17 15:41:13 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_from_source_volume( Apr 17 15:41:13 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 659, in _create_from_source_volume Apr 17 15:41:13 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self.driver.create_cloned_volume(volume, srcvol_ref) Apr 17 15:41:13 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 525, in create_cloned_volume Apr 17 15:41:13 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self._clone_object(vname, cvname) Apr 17 15:41:13 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 461, in _clone_object Apr 17 15:41:13 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_snapshot(oname, coname) Apr 17 15:41:13 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 639, in create_snapshot Apr 17 15:41:13 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201, 204): Apr 17 15:41:13 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 15:41:13 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 15:41:13 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Volume e80c78b2-20c9-4f9b-8898-6ab131d1eb06: re-scheduling SchedulerAPI.create_volume attempt 3 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 15:41:13 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Volume e80c78b2-20c9-4f9b-8898-6ab131d1eb06: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 15:41:13 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] deleating volume v_e80c78b2-20c9-4f9b-8898-6ab131d1eb06 {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 15:41:13 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] get snapshots for volume v_e80c78b2-20c9-4f9b-8898-6ab131d1eb06 {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 15:41:13 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_e80c78b2-20c9-4f9b-8898-6ab131d1eb06/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:41:13 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_e80c78b2-20c9-4f9b-8898-6ab131d1eb06/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:41:16 user cinder-volume[84696]: DEBUG cinder.utils [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.954(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:41:16 user cinder-volume[84696]: DEBUG cinder.utils [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_e80c78b2-20c9-4f9b-8898-6ab131d1eb06/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:41:16 user cinder-volume[84696]: DEBUG cinder.utils [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 11.971(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:41:16 user cinder-volume[84696]: DEBUG cinder.utils [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:41:16 user cinder-volume[84696]: DEBUG cinder.utils [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.202(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:41:16 user cinder-volume[84696]: DEBUG cinder.utils [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_a27d0ff3-5de9-4feb-82f7-d4571af88bb9/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:41:18 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=14,cinder:UPDATE=2 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:41:18 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=10,cinder:UPDATE=7,cinder:INSERT=3 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:41:18 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Volume reschedule parameters: True retry: {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 15:41:18 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Flow 'volume_create_manager' (29988406-5462-4c36-9fe1-ce18873db3ae) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:41:18 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (dde3ea60-4edc-4d71-bbb6-e3308a9eae12) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:41:18 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (dde3ea60-4edc-4d71-bbb6-e3308a9eae12) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:41:18Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesActionsTest-Volume-1624790909',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=441910f8-d29c-4c9e-bc5b-780ad228c85d,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='a8b97ff6ed6047f0a33e74684a1c162b',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:41:19Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:41:19Z,use_quota=True,user_id='c993ceeb4a5c49eea14057e2125518ad',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:41:18 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (8f2b47e7-826d-42cd-b35a-0914962c14c7) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:41:18 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (8f2b47e7-826d-42cd-b35a-0914962c14c7) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:41:18 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (7e91bd90-72f9-4965-b566-ea790ae99d85) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:41:18 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (7e91bd90-72f9-4965-b566-ea790ae99d85) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': '441910f8-d29c-4c9e-bc5b-780ad228c85d', 'volume_name': 'volume-441910f8-d29c-4c9e-bc5b-780ad228c85d', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:41:18 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (5937c4d5-cbe0-4096-9f56-e22a462541c1) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:41:18 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (5937c4d5-cbe0-4096-9f56-e22a462541c1) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:41:18 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (dc7ba6cf-31c7-40fc-9aa4-64eb819b469b) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:41:18 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Volume 441910f8-d29c-4c9e-bc5b-780ad228c85d: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-441910f8-d29c-4c9e-bc5b-780ad228c85d', 'volume_size': 1} Apr 17 15:41:18 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] creating volume v_441910f8-d29c-4c9e-bc5b-780ad228c85d. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 15:41:18 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] create volume {'name': 'v_441910f8-d29c-4c9e-bc5b-780ad228c85d', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 15:41:18 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:41:18 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:41:19 user cinder-volume[84696]: DEBUG cinder.utils [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.026(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:41:19 user cinder-volume[84696]: DEBUG cinder.utils [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_e80c78b2-20c9-4f9b-8898-6ab131d1eb06/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:41:19 user cinder-volume[84696]: DEBUG cinder.utils [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 0.743(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:41:19 user cinder-volume[84696]: DEBUG cinder.utils [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:41:22 user cinder-volume[84696]: DEBUG cinder.utils [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 3.808(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:41:22 user cinder-volume[84696]: DEBUG cinder.utils [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:41:25 user cinder-volume[84696]: DEBUG cinder.utils [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.166(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:41:25 user cinder-volume[84696]: DEBUG cinder.utils [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_e80c78b2-20c9-4f9b-8898-6ab131d1eb06/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:41:25 user cinder-volume[84696]: DEBUG cinder.utils [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.183(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:41:25 user cinder-volume[84696]: DEBUG cinder.utils [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:41:25 user cinder-volume[84696]: DEBUG cinder.utils [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.414(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:41:25 user cinder-volume[84696]: DEBUG cinder.utils [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_a27d0ff3-5de9-4feb-82f7-d4571af88bb9/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:41:28 user cinder-volume[84696]: DEBUG cinder.utils [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 9.952(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:41:28 user cinder-volume[84696]: DEBUG cinder.utils [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:41:28 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 73.721(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:41:28 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:41:34 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=1,cinder:UPDATE=1 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:41:34 user cinder-volume[84696]: DEBUG cinder.utils [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 73.624(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:41:34 user cinder-volume[84696]: DEBUG cinder.utils [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_a50e9659-3e2d-4f04-b0ac-ae78b8f61fad/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:41:34 user cinder-volume[84696]: DEBUG cinder.utils [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 73.608(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:41:34 user cinder-volume[84696]: DEBUG cinder.utils [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_57070f23-6b40-47b9-b9b4-e0d77240f544/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:41:34 user cinder-volume[84696]: DEBUG cinder.utils [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 73.409(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:41:34 user cinder-volume[84696]: DEBUG cinder.utils [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:41:34 user cinder-volume[84696]: DEBUG cinder.utils [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 73.328(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:41:34 user cinder-volume[84696]: DEBUG cinder.utils [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:41:34 user cinder-volume[84696]: DEBUG cinder.utils [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.454(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:41:34 user cinder-volume[84696]: DEBUG cinder.utils [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_e80c78b2-20c9-4f9b-8898-6ab131d1eb06/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:41:34 user cinder-volume[84696]: DEBUG cinder.utils [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 138.634(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:41:34 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:41:34 user cinder-volume[84696]: INFO cinder.message.api [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Creating message record for request_id = req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 Apr 17 15:41:34 user cinder-volume[84696]: DEBUG cinder.utils [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 138.811(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:41:34 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_29d9e55e-0b8b-4f70-b5c4-830d39f10fcd/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:41:34 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (da3db1cb-1b53-4ead-9ae4-b6be31938c54) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 15:41:34 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (1207f657-5807-48c5-b093-65262d08fd13) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:41:34 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (1207f657-5807-48c5-b093-65262d08fd13) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:41:34 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Flow 'volume_create_manager' (8f26591c-ce7d-4cf5-986c-6e45832cded6) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 15:41:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 15:41:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 15:41:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 15:41:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 15:41:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 15:41:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 15:41:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 15:41:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 15:41:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 15:41:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 15:41:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 15:41:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 15:41:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 15:41:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 15:41:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 15:41:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 15:41:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 15:41:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 15:41:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 15:41:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 15:41:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 15:41:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 15:41:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 15:41:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 15:41:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:41:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 15:41:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 15:41:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 15:41:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 15:41:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 15:41:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 15:41:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 15:41:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 15:41:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 15:41:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 15:41:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 15:41:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 15:41:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 15:41:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 15:41:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 15:41:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 15:41:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 15:41:34 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 15:41:34 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (cdcc8d8b-051c-4af3-91ac-bd03fc2b1945) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 15:41:34 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 15:41:34 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:34:44Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesSnapshotNegativeTestJSON-Volume-379445405',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=29d9e55e-0b8b-4f70-b5c4-830d39f10fcd,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='0b21bda55be444fc9f1a9c8c37b09224',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:39:16Z,service_uuid=None,shared_targets=True,size=2,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:39:16Z,use_quota=True,user_id='df17d7972cc649eb829bde41b8a718e4',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 15:41:34 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:34:44Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesSnapshotNegativeTestJSON-Volume-379445405',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=29d9e55e-0b8b-4f70-b5c4-830d39f10fcd,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='0b21bda55be444fc9f1a9c8c37b09224',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:39:16Z,service_uuid=None,shared_targets=True,size=2,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:39:16Z,use_quota=True,user_id='df17d7972cc649eb829bde41b8a718e4',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(29d9e55e-0b8b-4f70-b5c4-830d39f10fcd),volume_id=29d9e55e-0b8b-4f70-b5c4-830d39f10fcd,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': '29d9e55e-0b8b-4f70-b5c4-830d39f10fcd', 'volume_name': 'volume-29d9e55e-0b8b-4f70-b5c4-830d39f10fcd', 'volume_size': 2}} Apr 17 15:41:34 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': '29d9e55e-0b8b-4f70-b5c4-830d39f10fcd', 'volume': {'id': '29d9e55e-0b8b-4f70-b5c4-830d39f10fcd', '_name_id': None, 'ec2_id': None, 'user_id': 'df17d7972cc649eb829bde41b8a718e4', 'project_id': '0b21bda55be444fc9f1a9c8c37b09224', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 2, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-VolumesSnapshotNegativeTestJSON-Volume-379445405', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'admin_metadata': {}, 'glance_metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T15:34:44.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-29d9e55e-0b8b-4f70-b5c4-830d39f10fcd', 'name_id': '29d9e55e-0b8b-4f70-b5c4-830d39f10fcd', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumesSnapshotNegativeTestJSON-Volume-379445405', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '0b21bda55be444fc9f1a9c8c37b09224', 'qos_specs': None, 'reservations': ['da4aff04-a876-4157-b666-0fcc804a5076', '8e06fcb4-c53a-4537-9e40-36798fd5109c', '6bf081db-b8e9-44ff-ab74-ff0b0db7654a', 'ea5f14dc-ad5d-406a-8281-815b78e4eeb3'], 'size': 2, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'df17d7972cc649eb829bde41b8a718e4', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumesSnapshotNegativeTestJSON-Volume-379445405', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '0b21bda55be444fc9f1a9c8c37b09224', 'qos_specs': None, 'reservations': ['da4aff04-a876-4157-b666-0fcc804a5076', '8e06fcb4-c53a-4537-9e40-36798fd5109c', '6bf081db-b8e9-44ff-ab74-ff0b0db7654a', 'ea5f14dc-ad5d-406a-8281-815b78e4eeb3'], 'size': 2, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'df17d7972cc649eb829bde41b8a718e4', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 2, 'availability_zone': 'nova', 'user_id': 'df17d7972cc649eb829bde41b8a718e4', 'metadata': {}, 'qos_specs': None, 'retry': {'num_attempts': 3, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']}}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(29d9e55e-0b8b-4f70-b5c4-830d39f10fcd),volume_id=29d9e55e-0b8b-4f70-b5c4-830d39f10fcd,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:34:44Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesSnapshotNegativeTestJSON-Volume-379445405',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=29d9e55e-0b8b-4f70-b5c4-830d39f10fcd,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='0b21bda55be444fc9f1a9c8c37b09224',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:39:16Z,service_uuid=None,shared_targets=True,size=2,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:39:16Z,use_quota=True,user_id='df17d7972cc649eb829bde41b8a718e4',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 15:41:34 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:34:44Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesSnapshotNegativeTestJSON-Volume-379445405',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=29d9e55e-0b8b-4f70-b5c4-830d39f10fcd,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='0b21bda55be444fc9f1a9c8c37b09224',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:39:16Z,service_uuid=None,shared_targets=True,size=2,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:39:16Z,use_quota=True,user_id='df17d7972cc649eb829bde41b8a718e4',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:34:44Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesSnapshotNegativeTestJSON-Volume-379445405',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=29d9e55e-0b8b-4f70-b5c4-830d39f10fcd,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='0b21bda55be444fc9f1a9c8c37b09224',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:39:16Z,service_uuid=None,shared_targets=True,size=2,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:39:16Z,use_quota=True,user_id='df17d7972cc649eb829bde41b8a718e4',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 15:41:34 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 15:41:34 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 15:41:34 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:41:34 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 15:41:34 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 15:41:34 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 15:41:34 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 15:41:34 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 15:41:34 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 15:41:34 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 15:41:34 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 15:41:34 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 15:41:34 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 15:41:34 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 15:41:34 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 15:41:34 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 15:41:34 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 15:41:34 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 15:41:34 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 15:41:34 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 15:41:35 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (cdcc8d8b-051c-4af3-91ac-bd03fc2b1945) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:41:35 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (cdcc8d8b-051c-4af3-91ac-bd03fc2b1945) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:41:35 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (5fe25169-c5dd-4130-aee6-d40987ae3394) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:41:35 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (5fe25169-c5dd-4130-aee6-d40987ae3394) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:41:35 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (805eeac4-44a5-47ea-a5f9-0075b41b4116) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:41:35 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (805eeac4-44a5-47ea-a5f9-0075b41b4116) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:41:35 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (993cffe6-ba6b-4465-b578-d70fa9f61aa4) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:41:35 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Updating volume 29d9e55e-0b8b-4f70-b5c4-830d39f10fcd with {'scheduled_at': datetime.datetime(2023, 4, 17, 15, 41, 35, 33087), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 15:41:35 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Volume 29d9e55e-0b8b-4f70-b5c4-830d39f10fcd: re-scheduling SchedulerAPI.create_volume attempt 3 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 15:41:35 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Volume 29d9e55e-0b8b-4f70-b5c4-830d39f10fcd: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 15:41:35 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] deleating volume v_29d9e55e-0b8b-4f70-b5c4-830d39f10fcd {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 15:41:35 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] get snapshots for volume v_29d9e55e-0b8b-4f70-b5c4-830d39f10fcd {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 15:41:35 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_29d9e55e-0b8b-4f70-b5c4-830d39f10fcd/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:41:35 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_29d9e55e-0b8b-4f70-b5c4-830d39f10fcd/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:41:38 user cinder-volume[84696]: DEBUG cinder.utils [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 138.632(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:41:38 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:41:38 user cinder-volume[84696]: INFO cinder.message.api [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Creating message record for request_id = req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 Apr 17 15:41:38 user cinder-volume[84696]: DEBUG cinder.utils [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 138.803(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:41:38 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_44073502-102d-452a-8244-69a8233b9060/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:41:38 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (d1e1c597-b5d4-4f26-a4cf-5d131084fa80) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 15:41:38 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (474a3bd3-757a-410a-b2be-a4c346e7dc7d) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:41:38 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (474a3bd3-757a-410a-b2be-a4c346e7dc7d) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:41:38 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Flow 'volume_create_manager' (1353dfbd-eba4-4318-976f-a758c44f9eb2) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 15:41:38 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 15:41:38 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 15:41:38 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 15:41:38 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 15:41:38 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 15:41:38 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 15:41:38 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 15:41:38 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 15:41:38 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 15:41:38 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 15:41:38 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 15:41:38 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 15:41:38 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 15:41:38 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 15:41:38 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 15:41:38 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 15:41:38 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 15:41:38 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 15:41:38 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 15:41:38 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 15:41:38 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 15:41:38 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 15:41:38 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 15:41:38 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 15:41:38 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:41:38 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 15:41:38 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 15:41:38 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 15:41:38 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 15:41:38 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 15:41:38 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 15:41:38 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 15:41:38 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 15:41:38 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 15:41:38 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 15:41:38 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 15:41:38 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 15:41:38 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 15:41:38 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 15:41:38 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 15:41:38 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 15:41:38 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 15:41:38 user cinder-volume[84696]: DEBUG cinder.utils [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 19.253(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:41:38 user cinder-volume[84696]: DEBUG cinder.utils [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:41:38 user cinder-volume[84696]: DEBUG cinder.utils [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.971(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:41:38 user cinder-volume[84696]: DEBUG cinder.utils [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_29d9e55e-0b8b-4f70-b5c4-830d39f10fcd/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:41:38 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 15:41:38 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (f06dc9e1-67eb-4807-90dd-151e38da0ccf) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 15:41:38 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 15:41:38 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:34:45Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesActionsTest-Volume-737622743',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=44073502-102d-452a-8244-69a8233b9060,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='a8b97ff6ed6047f0a33e74684a1c162b',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:39:19Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:39:19Z,use_quota=True,user_id='c993ceeb4a5c49eea14057e2125518ad',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 15:41:38 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:34:45Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesActionsTest-Volume-737622743',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=44073502-102d-452a-8244-69a8233b9060,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='a8b97ff6ed6047f0a33e74684a1c162b',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:39:19Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:39:19Z,use_quota=True,user_id='c993ceeb4a5c49eea14057e2125518ad',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(44073502-102d-452a-8244-69a8233b9060),volume_id=44073502-102d-452a-8244-69a8233b9060,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': '44073502-102d-452a-8244-69a8233b9060', 'volume_name': 'volume-44073502-102d-452a-8244-69a8233b9060', 'volume_size': 1}} Apr 17 15:41:38 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': '44073502-102d-452a-8244-69a8233b9060', 'volume': {'id': '44073502-102d-452a-8244-69a8233b9060', '_name_id': None, 'ec2_id': None, 'user_id': 'c993ceeb4a5c49eea14057e2125518ad', 'project_id': 'a8b97ff6ed6047f0a33e74684a1c162b', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-VolumesActionsTest-Volume-737622743', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'admin_metadata': {}, 'glance_metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T15:34:45.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-44073502-102d-452a-8244-69a8233b9060', 'name_id': '44073502-102d-452a-8244-69a8233b9060', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumesActionsTest-Volume-737622743', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': 'a8b97ff6ed6047f0a33e74684a1c162b', 'qos_specs': None, 'reservations': ['bb018cba-ad93-4b64-acaa-8608d0a298cb', 'c3b20513-152c-40a3-a11d-aac4f6530172', '567445e0-7c93-4f93-b254-4b34cc804061', '988486ce-3f26-4167-a9e9-e6aa5ffe2065'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'c993ceeb4a5c49eea14057e2125518ad', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumesActionsTest-Volume-737622743', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': 'a8b97ff6ed6047f0a33e74684a1c162b', 'qos_specs': None, 'reservations': ['bb018cba-ad93-4b64-acaa-8608d0a298cb', 'c3b20513-152c-40a3-a11d-aac4f6530172', '567445e0-7c93-4f93-b254-4b34cc804061', '988486ce-3f26-4167-a9e9-e6aa5ffe2065'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'c993ceeb4a5c49eea14057e2125518ad', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': 'c993ceeb4a5c49eea14057e2125518ad', 'metadata': {}, 'qos_specs': None, 'retry': {'num_attempts': 3, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']}}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(44073502-102d-452a-8244-69a8233b9060),volume_id=44073502-102d-452a-8244-69a8233b9060,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:34:45Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesActionsTest-Volume-737622743',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=44073502-102d-452a-8244-69a8233b9060,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='a8b97ff6ed6047f0a33e74684a1c162b',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:39:19Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:39:19Z,use_quota=True,user_id='c993ceeb4a5c49eea14057e2125518ad',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 15:41:38 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:34:45Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesActionsTest-Volume-737622743',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=44073502-102d-452a-8244-69a8233b9060,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='a8b97ff6ed6047f0a33e74684a1c162b',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:39:19Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:39:19Z,use_quota=True,user_id='c993ceeb4a5c49eea14057e2125518ad',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:34:45Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesActionsTest-Volume-737622743',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=44073502-102d-452a-8244-69a8233b9060,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='a8b97ff6ed6047f0a33e74684a1c162b',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:39:19Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:39:19Z,use_quota=True,user_id='c993ceeb4a5c49eea14057e2125518ad',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 15:41:38 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 15:41:38 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 15:41:38 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:41:38 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 15:41:38 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 15:41:38 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 15:41:38 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 15:41:38 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 15:41:38 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 15:41:38 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 15:41:38 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 15:41:38 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 15:41:38 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 15:41:38 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 15:41:38 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 15:41:38 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 15:41:38 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 15:41:38 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 15:41:38 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 15:41:38 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 15:41:38 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (f06dc9e1-67eb-4807-90dd-151e38da0ccf) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:41:38 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (f06dc9e1-67eb-4807-90dd-151e38da0ccf) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:41:38 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (15d255df-caf9-495f-a84c-2da06d90d80c) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:41:38 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (15d255df-caf9-495f-a84c-2da06d90d80c) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:41:38 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (06ede640-5cb3-4469-a8a1-bb96c7244b71) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:41:38 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (06ede640-5cb3-4469-a8a1-bb96c7244b71) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:41:38 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (3ce1e426-0c16-497b-adfe-a54970dc7680) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:41:38 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Updating volume 44073502-102d-452a-8244-69a8233b9060 with {'scheduled_at': datetime.datetime(2023, 4, 17, 15, 41, 38, 91973), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 15:41:38 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Volume 44073502-102d-452a-8244-69a8233b9060: re-scheduling SchedulerAPI.create_volume attempt 3 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 15:41:38 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Volume 44073502-102d-452a-8244-69a8233b9060: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 15:41:38 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] deleating volume v_44073502-102d-452a-8244-69a8233b9060 {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 15:41:38 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] get snapshots for volume v_44073502-102d-452a-8244-69a8233b9060 {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 15:41:38 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_44073502-102d-452a-8244-69a8233b9060/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:41:38 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_44073502-102d-452a-8244-69a8233b9060/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:41:41 user cinder-volume[84696]: DEBUG cinder.utils [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.964(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:41:41 user cinder-volume[84696]: DEBUG cinder.utils [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_44073502-102d-452a-8244-69a8233b9060/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:41:41 user cinder-volume[84696]: DEBUG cinder.utils [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.023(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:41:41 user cinder-volume[84696]: DEBUG cinder.utils [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_29d9e55e-0b8b-4f70-b5c4-830d39f10fcd/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:41:44 user cinder-volume[84696]: DEBUG cinder.utils [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.677(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:41:44 user cinder-volume[84696]: DEBUG cinder.utils [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:41:44 user cinder-volume[84696]: DEBUG cinder.utils [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.909(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:41:44 user cinder-volume[84696]: DEBUG cinder.utils [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_a27d0ff3-5de9-4feb-82f7-d4571af88bb9/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:41:44 user cinder-volume[84696]: DEBUG cinder.utils [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.038(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:41:44 user cinder-volume[84696]: DEBUG cinder.utils [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_44073502-102d-452a-8244-69a8233b9060/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:41:44 user cinder-volume[84696]: DEBUG cinder.utils [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 73.676(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:41:44 user cinder-volume[84696]: DEBUG cinder.utils [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_8cc35ab1-363f-4ad6-9a37-239d2b1ea996/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:41:47 user cinder-volume[84696]: DEBUG cinder.utils [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 73.689(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:41:47 user cinder-volume[84696]: DEBUG cinder.utils [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_7ce815ce-9603-4fd6-8cc8-9fd798b2f99d/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:41:47 user cinder-volume[84696]: DEBUG cinder.utils [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.167(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:41:47 user cinder-volume[84696]: DEBUG cinder.utils [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_29d9e55e-0b8b-4f70-b5c4-830d39f10fcd/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:41:47 user cinder-volume[84696]: DEBUG cinder.utils [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 73.526(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:41:47 user cinder-volume[84696]: DEBUG cinder.utils [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:41:48 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=15,cinder:UPDATE=5,cinder:INSERT=1 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:41:50 user cinder-volume[84696]: DEBUG cinder.utils [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.184(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:41:50 user cinder-volume[84696]: DEBUG cinder.utils [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_44073502-102d-452a-8244-69a8233b9060/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:41:50 user cinder-volume[84696]: DEBUG cinder.utils [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 138.782(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:41:50 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_153b4df8-1f37-4118-8aa6-1c155ea5700c/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:41:50 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (65c9e1e6-5853-421e-acca-4c4add9c4e18) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 15:41:50 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (85f5f5dc-ad23-4286-b30c-b4b3a55cf7ac) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:41:50 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (85f5f5dc-ad23-4286-b30c-b4b3a55cf7ac) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:41:50 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Flow 'volume_create_manager' (a5e567c6-c297-4177-8116-27fb5330b229) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 15:41:50 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 15:41:50 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 15:41:50 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 15:41:50 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 15:41:50 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 15:41:50 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 15:41:50 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 15:41:50 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 15:41:50 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 15:41:50 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 15:41:50 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 15:41:50 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 15:41:50 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 15:41:50 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 15:41:50 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 15:41:50 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 15:41:50 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 15:41:50 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 15:41:50 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 15:41:50 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 15:41:50 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 15:41:50 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 15:41:50 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 15:41:50 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 15:41:50 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:41:50 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 15:41:50 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 15:41:50 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 15:41:50 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 15:41:50 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 15:41:50 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 15:41:50 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 15:41:50 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 15:41:50 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 15:41:50 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 15:41:50 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 15:41:50 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 15:41:50 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 15:41:50 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 15:41:50 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 15:41:50 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 15:41:50 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 15:41:50 user cinder-volume[84696]: DEBUG cinder.utils [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 138.590(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:41:50 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:41:50 user cinder-volume[84696]: INFO cinder.message.api [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Creating message record for request_id = req-de22aca9-c3f8-4015-be3e-9d70b59359d2 Apr 17 15:41:50 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 15:41:50 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (d0d80f8c-1637-4101-a116-fabe4281bba4) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 15:41:50 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 15:41:50 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:34:57Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesCloneTest-Volume-2030679513',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=153b4df8-1f37-4118-8aa6-1c155ea5700c,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='82fce1724afa40dd914e0791c3ee69a3',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:39:32Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:39:32Z,use_quota=True,user_id='773ce6c2a0f844edaf666e190504e258',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 15:41:50 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:34:57Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesCloneTest-Volume-2030679513',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=153b4df8-1f37-4118-8aa6-1c155ea5700c,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='82fce1724afa40dd914e0791c3ee69a3',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:39:32Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:39:32Z,use_quota=True,user_id='773ce6c2a0f844edaf666e190504e258',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(153b4df8-1f37-4118-8aa6-1c155ea5700c),volume_id=153b4df8-1f37-4118-8aa6-1c155ea5700c,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': '153b4df8-1f37-4118-8aa6-1c155ea5700c', 'volume_name': 'volume-153b4df8-1f37-4118-8aa6-1c155ea5700c', 'volume_size': 1}} Apr 17 15:41:50 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': '153b4df8-1f37-4118-8aa6-1c155ea5700c', 'volume': {'id': '153b4df8-1f37-4118-8aa6-1c155ea5700c', '_name_id': None, 'ec2_id': None, 'user_id': '773ce6c2a0f844edaf666e190504e258', 'project_id': '82fce1724afa40dd914e0791c3ee69a3', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-VolumesCloneTest-Volume-2030679513', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'admin_metadata': {}, 'glance_metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T15:34:57.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-153b4df8-1f37-4118-8aa6-1c155ea5700c', 'name_id': '153b4df8-1f37-4118-8aa6-1c155ea5700c', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumesCloneTest-Volume-2030679513', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '82fce1724afa40dd914e0791c3ee69a3', 'qos_specs': None, 'reservations': ['4e0e5a8a-460c-4231-830c-7500b8b4d874', '4e75ce56-bc87-4771-a3a2-a212a9299682', '1e95db6e-9eea-496d-9f91-b9dc2fd1660c', 'd2d362d2-ccef-4207-b0ce-c2abf0695375'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': '773ce6c2a0f844edaf666e190504e258', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumesCloneTest-Volume-2030679513', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '82fce1724afa40dd914e0791c3ee69a3', 'qos_specs': None, 'reservations': ['4e0e5a8a-460c-4231-830c-7500b8b4d874', '4e75ce56-bc87-4771-a3a2-a212a9299682', '1e95db6e-9eea-496d-9f91-b9dc2fd1660c', 'd2d362d2-ccef-4207-b0ce-c2abf0695375'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': '773ce6c2a0f844edaf666e190504e258', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': '773ce6c2a0f844edaf666e190504e258', 'metadata': {}, 'qos_specs': None, 'retry': {'num_attempts': 3, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']}}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(153b4df8-1f37-4118-8aa6-1c155ea5700c),volume_id=153b4df8-1f37-4118-8aa6-1c155ea5700c,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:34:57Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesCloneTest-Volume-2030679513',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=153b4df8-1f37-4118-8aa6-1c155ea5700c,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='82fce1724afa40dd914e0791c3ee69a3',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:39:32Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:39:32Z,use_quota=True,user_id='773ce6c2a0f844edaf666e190504e258',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 15:41:50 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:34:57Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesCloneTest-Volume-2030679513',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=153b4df8-1f37-4118-8aa6-1c155ea5700c,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='82fce1724afa40dd914e0791c3ee69a3',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:39:32Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:39:32Z,use_quota=True,user_id='773ce6c2a0f844edaf666e190504e258',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:34:57Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesCloneTest-Volume-2030679513',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=153b4df8-1f37-4118-8aa6-1c155ea5700c,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='82fce1724afa40dd914e0791c3ee69a3',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:39:32Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:39:32Z,use_quota=True,user_id='773ce6c2a0f844edaf666e190504e258',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 15:41:50 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 15:41:50 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 15:41:50 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:41:50 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 15:41:50 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 15:41:50 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 15:41:50 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 15:41:50 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 15:41:50 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 15:41:50 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 15:41:50 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 15:41:50 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 15:41:50 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 15:41:50 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 15:41:50 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 15:41:50 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 15:41:50 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 15:41:50 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 15:41:50 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 15:41:50 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 15:41:50 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (d0d80f8c-1637-4101-a116-fabe4281bba4) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:41:50 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (d0d80f8c-1637-4101-a116-fabe4281bba4) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:41:50 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (8094f8b2-c537-494d-a389-b4d5a24ce4bf) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:41:50 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (8094f8b2-c537-494d-a389-b4d5a24ce4bf) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:41:50 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (d8c97eff-b5fc-498b-b8d0-f31782630c29) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:41:50 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (d8c97eff-b5fc-498b-b8d0-f31782630c29) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:41:50 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (7c086dcf-f653-40fe-aa4b-b1e6bb7df085) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:41:50 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Updating volume 153b4df8-1f37-4118-8aa6-1c155ea5700c with {'scheduled_at': datetime.datetime(2023, 4, 17, 15, 41, 50, 386622), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 15:41:50 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Volume 153b4df8-1f37-4118-8aa6-1c155ea5700c: re-scheduling SchedulerAPI.create_volume attempt 3 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 15:41:50 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Volume 153b4df8-1f37-4118-8aa6-1c155ea5700c: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 15:41:50 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] deleating volume v_153b4df8-1f37-4118-8aa6-1c155ea5700c {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 15:41:50 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] get snapshots for volume v_153b4df8-1f37-4118-8aa6-1c155ea5700c {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 15:41:50 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_153b4df8-1f37-4118-8aa6-1c155ea5700c/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:41:50 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_153b4df8-1f37-4118-8aa6-1c155ea5700c/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:41:53 user cinder-volume[84696]: DEBUG cinder.utils [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.962(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:41:53 user cinder-volume[84696]: DEBUG cinder.utils [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_153b4df8-1f37-4118-8aa6-1c155ea5700c/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:41:53 user cinder-volume[84696]: DEBUG cinder.utils [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.879(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:41:53 user cinder-volume[84696]: DEBUG cinder.utils [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_e80c78b2-20c9-4f9b-8898-6ab131d1eb06/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:41:56 user cinder-volume[84696]: DEBUG cinder.utils [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 37.696(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:41:56 user cinder-volume[84696]: DEBUG cinder.utils [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:41:56 user cinder-volume[84696]: DEBUG cinder.utils [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.415(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:41:56 user cinder-volume[84696]: DEBUG cinder.utils [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_29d9e55e-0b8b-4f70-b5c4-830d39f10fcd/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:41:56 user cinder-volume[84696]: DEBUG cinder.utils [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.068(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:41:56 user cinder-volume[84696]: DEBUG cinder.utils [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_153b4df8-1f37-4118-8aa6-1c155ea5700c/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:41:59 user cinder-volume[84696]: DEBUG cinder.utils [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.428(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:41:59 user cinder-volume[84696]: DEBUG cinder.utils [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_44073502-102d-452a-8244-69a8233b9060/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:42:02 user cinder-volume[84696]: DEBUG cinder.utils [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.210(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:42:02 user cinder-volume[84696]: DEBUG cinder.utils [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_153b4df8-1f37-4118-8aa6-1c155ea5700c/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:42:04 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:INSERT=3,cinder:UPDATE=3,cinder:SELECT=9 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:42:11 user cinder-volume[84696]: DEBUG cinder.utils [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.554(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:42:11 user cinder-volume[84696]: DEBUG cinder.utils [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_153b4df8-1f37-4118-8aa6-1c155ea5700c/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:42:15 user cinder-volume[84696]: DEBUG cinder.utils [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.974(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:42:15 user cinder-volume[84696]: DEBUG cinder.utils [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_29d9e55e-0b8b-4f70-b5c4-830d39f10fcd/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:42:18 user cinder-volume[84696]: DEBUG cinder.utils [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.988(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:42:18 user cinder-volume[84696]: DEBUG cinder.utils [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_44073502-102d-452a-8244-69a8233b9060/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:42:18 user cinder-volume[84696]: DEBUG cinder.utils [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 73.630(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:42:18 user cinder-volume[84696]: DEBUG cinder.utils [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:42:18 user cinder-volume[84696]: DEBUG cinder.utils [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 73.862(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:42:18 user cinder-volume[84696]: DEBUG cinder.utils [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_a27d0ff3-5de9-4feb-82f7-d4571af88bb9/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:42:24 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:INSERT=2,cinder:UPDATE=3,cinder:SELECT=7 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:42:24 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=1,cinder:UPDATE=1 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:42:28 user cinder-volume[84696]: DEBUG cinder.utils [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 37.650(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:42:28 user cinder-volume[84696]: DEBUG cinder.utils [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_153b4df8-1f37-4118-8aa6-1c155ea5700c/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:42:28 user cinder-volume[84696]: DEBUG cinder.utils [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 74.568(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:42:28 user cinder-volume[84696]: DEBUG cinder.utils [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_e80c78b2-20c9-4f9b-8898-6ab131d1eb06/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:42:31 user cinder-volume[84696]: DEBUG cinder.utils [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 72.352(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:42:31 user cinder-volume[84696]: DEBUG cinder.utils [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:42:34 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 139.192(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:42:34 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:42:34 user cinder-volume[84696]: ERROR oslo_service.periodic_task [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Error during VolumeManager.publish_service_capabilities: TypeError: 'NoneType' object is not subscriptable Apr 17 15:42:34 user cinder-volume[84696]: ERROR oslo_service.periodic_task Traceback (most recent call last): Apr 17 15:42:34 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/usr/local/lib/python3.10/dist-packages/oslo_service/periodic_task.py", line 216, in run_periodic_tasks Apr 17 15:42:34 user cinder-volume[84696]: ERROR oslo_service.periodic_task task(self, context) Apr 17 15:42:34 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/manager.py", line 2876, in publish_service_capabilities Apr 17 15:42:34 user cinder-volume[84696]: ERROR oslo_service.periodic_task self._report_driver_status(context) Apr 17 15:42:34 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/manager.py", line 2755, in _report_driver_status Apr 17 15:42:34 user cinder-volume[84696]: ERROR oslo_service.periodic_task volume_stats = get_stats() Apr 17 15:42:34 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/usr/local/lib/python3.10/dist-packages/oslo_utils/timeutils.py", line 319, in wrapper Apr 17 15:42:34 user cinder-volume[84696]: ERROR oslo_service.periodic_task result = func(*args, **kwargs) Apr 17 15:42:34 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/manager.py", line 2753, in get_stats Apr 17 15:42:34 user cinder-volume[84696]: ERROR oslo_service.periodic_task return self.driver.get_volume_stats(refresh=True) Apr 17 15:42:34 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/driver.py", line 666, in get_volume_stats Apr 17 15:42:34 user cinder-volume[84696]: ERROR oslo_service.periodic_task self._update_volume_stats() Apr 17 15:42:34 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 726, in _update_volume_stats Apr 17 15:42:34 user cinder-volume[84696]: ERROR oslo_service.periodic_task pool_stats = self.ra.get_pool_stats() Apr 17 15:42:34 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 897, in get_pool_stats Apr 17 15:42:34 user cinder-volume[84696]: ERROR oslo_service.periodic_task if not resp["error"] and resp["code"] == 200: Apr 17 15:42:34 user cinder-volume[84696]: ERROR oslo_service.periodic_task TypeError: 'NoneType' object is not subscriptable Apr 17 15:42:34 user cinder-volume[84696]: ERROR oslo_service.periodic_task Apr 17 15:42:34 user cinder-volume[84696]: WARNING oslo.service.loopingcall [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Function 'cinder.service.Service.periodic_tasks' run outlasted interval by 79.20 sec Apr 17 15:42:34 user cinder-volume[84696]: DEBUG oslo_service.periodic_task [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Running periodic task VolumeManager.publish_service_capabilities {{(pid=84696) run_periodic_tasks /usr/local/lib/python3.10/dist-packages/oslo_service/periodic_task.py:210}} Apr 17 15:42:34 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Updating volume stats {{(pid=84696) _update_volume_stats /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:724}} Apr 17 15:42:34 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Get pool Pool-0 fsprops {{(pid=84696) get_pool_stats /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:894}} Apr 17 15:42:34 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:42:34 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:42:37 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 3.067(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:42:37 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:42:40 user cinder-volume[84696]: DEBUG cinder.utils [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 139.028(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:42:40 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_a50e9659-3e2d-4f04-b0ac-ae78b8f61fad/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:42:40 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (d0b2a951-fe4a-461f-9929-7e538f0f603f) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 15:42:40 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (b3b3ed96-55cb-4b63-b5cf-dbf33ab64d47) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:42:40 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (b3b3ed96-55cb-4b63-b5cf-dbf33ab64d47) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:42:40 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Flow 'volume_create_manager' (f343c56c-018a-4dcb-8100-2d310c2bc132) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 15:42:40 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 15:42:40 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 15:42:40 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 15:42:40 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 15:42:40 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 15:42:40 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 15:42:40 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 15:42:40 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 15:42:40 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 15:42:40 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 15:42:40 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 15:42:40 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 15:42:40 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 15:42:40 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 15:42:40 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 15:42:40 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 15:42:40 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 15:42:40 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 15:42:40 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 15:42:40 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 15:42:40 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 15:42:40 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 15:42:40 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 15:42:40 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 15:42:40 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:42:40 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 15:42:40 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 15:42:40 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 15:42:40 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 15:42:40 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 15:42:40 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 15:42:40 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 15:42:40 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 15:42:40 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 15:42:40 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 15:42:40 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 15:42:40 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 15:42:40 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 15:42:40 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 15:42:40 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 15:42:40 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 15:42:40 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 15:42:40 user cinder-volume[84696]: DEBUG cinder.utils [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 138.851(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:42:40 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:42:40 user cinder-volume[84696]: INFO cinder.message.api [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Creating message record for request_id = req-e0039512-b662-485b-95d9-48de5642a75d Apr 17 15:42:40 user cinder-volume[84696]: DEBUG cinder.utils [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 139.056(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:42:40 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_57070f23-6b40-47b9-b9b4-e0d77240f544/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:42:40 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (a7ae420e-549f-49e0-a4be-a45df8a475f7) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 15:42:40 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (c4aa30a2-b694-4dcd-b60a-96ad11091b1c) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:42:40 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (c4aa30a2-b694-4dcd-b60a-96ad11091b1c) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:42:40 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Flow 'volume_create_manager' (41dfb63e-f309-418c-ab2b-9c3f4f759af8) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 15:42:40 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 15:42:40 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 15:42:40 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 15:42:40 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 15:42:40 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 15:42:40 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 15:42:40 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 15:42:40 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 15:42:40 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 15:42:40 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 15:42:40 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 15:42:40 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 15:42:40 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 15:42:40 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 15:42:40 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 15:42:40 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 15:42:40 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 15:42:40 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 15:42:40 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 15:42:40 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 15:42:40 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 15:42:40 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 15:42:40 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 15:42:40 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 15:42:40 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:42:40 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 15:42:40 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 15:42:40 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 15:42:40 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 15:42:40 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 15:42:40 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 15:42:40 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 15:42:40 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 15:42:40 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 15:42:40 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 15:42:40 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 15:42:40 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 15:42:40 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 15:42:40 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 15:42:40 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 15:42:40 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 15:42:40 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 15:42:40 user cinder-volume[84696]: DEBUG cinder.utils [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 138.795(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:42:40 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:42:40 user cinder-volume[84696]: INFO cinder.message.api [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Creating message record for request_id = req-e55035f7-bed9-4dad-9cbc-b8b208b454bf Apr 17 15:42:40 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.209(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:42:40 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:42:40 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 15:42:40 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 15:42:40 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (c66f5f28-b7f0-46ea-8fa5-69e971fbfe72) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 15:42:40 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 15:42:40 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:38:02Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-CreateVolumesFromSourceVolumeTest-Volume-609446595',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=a50e9659-3e2d-4f04-b0ac-ae78b8f61fad,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='99747be727ae4592ae1b6176b488719a',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:40:21Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:40:21Z,use_quota=True,user_id='0696808ea0b14c0794e8664a37d9e2c3',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 15:42:40 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:38:02Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-CreateVolumesFromSourceVolumeTest-Volume-609446595',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=a50e9659-3e2d-4f04-b0ac-ae78b8f61fad,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='99747be727ae4592ae1b6176b488719a',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:40:21Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:40:21Z,use_quota=True,user_id='0696808ea0b14c0794e8664a37d9e2c3',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(a50e9659-3e2d-4f04-b0ac-ae78b8f61fad),volume_id=a50e9659-3e2d-4f04-b0ac-ae78b8f61fad,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': 'a50e9659-3e2d-4f04-b0ac-ae78b8f61fad', 'volume_name': 'volume-a50e9659-3e2d-4f04-b0ac-ae78b8f61fad', 'volume_size': 1}} Apr 17 15:42:40 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': 'a50e9659-3e2d-4f04-b0ac-ae78b8f61fad', 'volume': {'id': 'a50e9659-3e2d-4f04-b0ac-ae78b8f61fad', '_name_id': None, 'ec2_id': None, 'user_id': '0696808ea0b14c0794e8664a37d9e2c3', 'project_id': '99747be727ae4592ae1b6176b488719a', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-CreateVolumesFromSourceVolumeTest-Volume-609446595', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'admin_metadata': {}, 'glance_metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T15:38:02.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-a50e9659-3e2d-4f04-b0ac-ae78b8f61fad', 'name_id': 'a50e9659-3e2d-4f04-b0ac-ae78b8f61fad', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-CreateVolumesFromSourceVolumeTest-Volume-609446595', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '99747be727ae4592ae1b6176b488719a', 'qos_specs': None, 'reservations': ['e9df84e9-0002-4c5e-9db7-eae2489ad0e9', 'd438171a-8bd2-47cf-81e3-e4af8b2d355c', 'cf24dd26-1a0b-48b2-95ca-4eceaffb6e37', 'e9528d9a-1681-4b42-a96b-ae681a21177b'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': '0696808ea0b14c0794e8664a37d9e2c3', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-CreateVolumesFromSourceVolumeTest-Volume-609446595', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '99747be727ae4592ae1b6176b488719a', 'qos_specs': None, 'reservations': ['e9df84e9-0002-4c5e-9db7-eae2489ad0e9', 'd438171a-8bd2-47cf-81e3-e4af8b2d355c', 'cf24dd26-1a0b-48b2-95ca-4eceaffb6e37', 'e9528d9a-1681-4b42-a96b-ae681a21177b'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': '0696808ea0b14c0794e8664a37d9e2c3', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': '0696808ea0b14c0794e8664a37d9e2c3', 'metadata': {}, 'qos_specs': None, 'retry': {'num_attempts': 2, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']}}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(a50e9659-3e2d-4f04-b0ac-ae78b8f61fad),volume_id=a50e9659-3e2d-4f04-b0ac-ae78b8f61fad,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:38:02Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-CreateVolumesFromSourceVolumeTest-Volume-609446595',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=a50e9659-3e2d-4f04-b0ac-ae78b8f61fad,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='99747be727ae4592ae1b6176b488719a',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:40:21Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:40:21Z,use_quota=True,user_id='0696808ea0b14c0794e8664a37d9e2c3',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 15:42:40 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:38:02Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-CreateVolumesFromSourceVolumeTest-Volume-609446595',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=a50e9659-3e2d-4f04-b0ac-ae78b8f61fad,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='99747be727ae4592ae1b6176b488719a',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:40:21Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:40:21Z,use_quota=True,user_id='0696808ea0b14c0794e8664a37d9e2c3',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:38:02Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-CreateVolumesFromSourceVolumeTest-Volume-609446595',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=a50e9659-3e2d-4f04-b0ac-ae78b8f61fad,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='99747be727ae4592ae1b6176b488719a',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:40:21Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:40:21Z,use_quota=True,user_id='0696808ea0b14c0794e8664a37d9e2c3',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 15:42:40 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 15:42:40 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 15:42:40 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:42:40 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 15:42:40 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 15:42:40 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 15:42:40 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 15:42:40 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 15:42:40 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 15:42:40 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 15:42:40 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 15:42:40 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 15:42:40 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 15:42:40 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 15:42:40 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 15:42:40 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 15:42:40 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 15:42:40 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 15:42:40 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 15:42:40 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 15:42:40 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (c66f5f28-b7f0-46ea-8fa5-69e971fbfe72) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:42:40 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (c66f5f28-b7f0-46ea-8fa5-69e971fbfe72) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:42:40 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (40dd90f2-4132-4175-ba5a-9be4760756f3) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:42:40 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (40dd90f2-4132-4175-ba5a-9be4760756f3) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:42:40 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (98989c1a-5627-4523-b1dc-df8a73e72b21) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:42:40 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (98989c1a-5627-4523-b1dc-df8a73e72b21) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:42:40 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (a0c6b43b-6bc3-4e94-973e-6e446ec021f2) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:42:40 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Updating volume a50e9659-3e2d-4f04-b0ac-ae78b8f61fad with {'scheduled_at': datetime.datetime(2023, 4, 17, 15, 42, 40, 489770), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 15:42:40 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (1a21b330-e2f1-48db-a8b5-679e9cc0476b) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 15:42:40 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 15:42:40 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:38:02Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesActionsTest-Volume-1957591186',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=57070f23-6b40-47b9-b9b4-e0d77240f544,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='a8b97ff6ed6047f0a33e74684a1c162b',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:40:21Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:40:21Z,use_quota=True,user_id='c993ceeb4a5c49eea14057e2125518ad',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 15:42:40 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:38:02Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesActionsTest-Volume-1957591186',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=57070f23-6b40-47b9-b9b4-e0d77240f544,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='a8b97ff6ed6047f0a33e74684a1c162b',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:40:21Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:40:21Z,use_quota=True,user_id='c993ceeb4a5c49eea14057e2125518ad',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(57070f23-6b40-47b9-b9b4-e0d77240f544),volume_id=57070f23-6b40-47b9-b9b4-e0d77240f544,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': '57070f23-6b40-47b9-b9b4-e0d77240f544', 'volume_name': 'volume-57070f23-6b40-47b9-b9b4-e0d77240f544', 'volume_size': 1}} Apr 17 15:42:40 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': '57070f23-6b40-47b9-b9b4-e0d77240f544', 'volume': {'id': '57070f23-6b40-47b9-b9b4-e0d77240f544', '_name_id': None, 'ec2_id': None, 'user_id': 'c993ceeb4a5c49eea14057e2125518ad', 'project_id': 'a8b97ff6ed6047f0a33e74684a1c162b', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-VolumesActionsTest-Volume-1957591186', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'admin_metadata': {}, 'glance_metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T15:38:02.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-57070f23-6b40-47b9-b9b4-e0d77240f544', 'name_id': '57070f23-6b40-47b9-b9b4-e0d77240f544', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumesActionsTest-Volume-1957591186', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': 'a8b97ff6ed6047f0a33e74684a1c162b', 'qos_specs': None, 'reservations': ['24ba47c4-9401-42c6-8d85-7f78767497d2', '1d6429be-3a45-4be3-a3f2-22a585792470', 'd0b8f941-f03c-4f32-b89a-dd9df5910de8', '4cdcf710-02d8-4b32-903f-875263491ca7'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'c993ceeb4a5c49eea14057e2125518ad', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumesActionsTest-Volume-1957591186', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': 'a8b97ff6ed6047f0a33e74684a1c162b', 'qos_specs': None, 'reservations': ['24ba47c4-9401-42c6-8d85-7f78767497d2', '1d6429be-3a45-4be3-a3f2-22a585792470', 'd0b8f941-f03c-4f32-b89a-dd9df5910de8', '4cdcf710-02d8-4b32-903f-875263491ca7'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'c993ceeb4a5c49eea14057e2125518ad', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': 'c993ceeb4a5c49eea14057e2125518ad', 'metadata': {}, 'qos_specs': None, 'retry': {'num_attempts': 2, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']}}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(57070f23-6b40-47b9-b9b4-e0d77240f544),volume_id=57070f23-6b40-47b9-b9b4-e0d77240f544,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:38:02Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesActionsTest-Volume-1957591186',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=57070f23-6b40-47b9-b9b4-e0d77240f544,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='a8b97ff6ed6047f0a33e74684a1c162b',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:40:21Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:40:21Z,use_quota=True,user_id='c993ceeb4a5c49eea14057e2125518ad',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 15:42:40 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:38:02Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesActionsTest-Volume-1957591186',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=57070f23-6b40-47b9-b9b4-e0d77240f544,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='a8b97ff6ed6047f0a33e74684a1c162b',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:40:21Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:40:21Z,use_quota=True,user_id='c993ceeb4a5c49eea14057e2125518ad',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:38:02Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesActionsTest-Volume-1957591186',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=57070f23-6b40-47b9-b9b4-e0d77240f544,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='a8b97ff6ed6047f0a33e74684a1c162b',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:40:21Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:40:21Z,use_quota=True,user_id='c993ceeb4a5c49eea14057e2125518ad',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 15:42:40 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 15:42:40 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 15:42:40 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:42:40 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 15:42:40 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 15:42:40 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 15:42:40 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 15:42:40 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 15:42:40 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 15:42:40 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 15:42:40 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 15:42:40 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 15:42:40 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 15:42:40 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 15:42:40 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 15:42:40 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 15:42:40 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 15:42:40 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 15:42:40 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 15:42:40 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 15:42:40 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (1a21b330-e2f1-48db-a8b5-679e9cc0476b) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:42:40 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (1a21b330-e2f1-48db-a8b5-679e9cc0476b) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:42:40 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (e0548b12-e511-42f0-8807-129eb0fc0344) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:42:40 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (e0548b12-e511-42f0-8807-129eb0fc0344) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:42:40 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (99c4cf9e-b630-4d9c-a4ce-df04867da0f5) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:42:40 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (99c4cf9e-b630-4d9c-a4ce-df04867da0f5) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:42:40 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (f52a7c8f-8888-422c-a059-bcd722bb11a9) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:42:40 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Updating volume 57070f23-6b40-47b9-b9b4-e0d77240f544 with {'scheduled_at': datetime.datetime(2023, 4, 17, 15, 42, 40, 540265), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 15:42:40 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Volume 57070f23-6b40-47b9-b9b4-e0d77240f544: re-scheduling SchedulerAPI.create_volume attempt 2 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 15:42:40 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Volume a50e9659-3e2d-4f04-b0ac-ae78b8f61fad: re-scheduling SchedulerAPI.create_volume attempt 2 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 15:42:40 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Volume 57070f23-6b40-47b9-b9b4-e0d77240f544: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 15:42:40 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] deleating volume v_57070f23-6b40-47b9-b9b4-e0d77240f544 {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 15:42:40 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] get snapshots for volume v_57070f23-6b40-47b9-b9b4-e0d77240f544 {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 15:42:40 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_57070f23-6b40-47b9-b9b4-e0d77240f544/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:42:40 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_57070f23-6b40-47b9-b9b4-e0d77240f544/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:42:40 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Volume a50e9659-3e2d-4f04-b0ac-ae78b8f61fad: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 15:42:40 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] deleating volume v_a50e9659-3e2d-4f04-b0ac-ae78b8f61fad {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 15:42:40 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] get snapshots for volume v_a50e9659-3e2d-4f04-b0ac-ae78b8f61fad {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 15:42:40 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_a50e9659-3e2d-4f04-b0ac-ae78b8f61fad/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:42:40 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_a50e9659-3e2d-4f04-b0ac-ae78b8f61fad/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:42:40 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Volume reschedule parameters: True retry: {'num_attempts': 3, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 15:42:40 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Flow 'volume_create_manager' (862dc8f8-09bd-4559-a622-c1bc1cdfcb4b) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:42:40 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (21c2e379-aa21-4b1e-b467-1a7beae2f068) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:42:40 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (21c2e379-aa21-4b1e-b467-1a7beae2f068) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:38:02Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesActionsTest-Volume-1957591186',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=57070f23-6b40-47b9-b9b4-e0d77240f544,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='a8b97ff6ed6047f0a33e74684a1c162b',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:42:41Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:42:41Z,use_quota=True,user_id='c993ceeb4a5c49eea14057e2125518ad',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:42:40 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (64ee410e-815a-4b3a-b712-8b21a6c707d8) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:42:40 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (64ee410e-815a-4b3a-b712-8b21a6c707d8) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:42:40 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (a731bf96-2be9-43af-bf17-3e401bcf5093) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:42:40 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (a731bf96-2be9-43af-bf17-3e401bcf5093) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': '57070f23-6b40-47b9-b9b4-e0d77240f544', 'volume_name': 'volume-57070f23-6b40-47b9-b9b4-e0d77240f544', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:42:40 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (e15537ed-0296-43d3-872c-15aa19320938) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:42:40 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (e15537ed-0296-43d3-872c-15aa19320938) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:42:40 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (f7df48e2-21f0-477f-b45b-ee9c5fd966e8) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:42:40 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Volume 57070f23-6b40-47b9-b9b4-e0d77240f544: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-57070f23-6b40-47b9-b9b4-e0d77240f544', 'volume_size': 1} Apr 17 15:42:40 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] creating volume v_57070f23-6b40-47b9-b9b4-e0d77240f544. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 15:42:40 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] create volume {'name': 'v_57070f23-6b40-47b9-b9b4-e0d77240f544', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 15:42:40 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:42:40 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:42:40 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Volume reschedule parameters: True retry: {'num_attempts': 3, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 15:42:40 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Flow 'volume_create_manager' (c15a8b37-3406-4de3-92a0-bb523e3f9c64) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:42:40 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (73dac3cd-a0e1-4f9c-9609-de8d355deb71) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:42:40 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (73dac3cd-a0e1-4f9c-9609-de8d355deb71) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:38:02Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-CreateVolumesFromSourceVolumeTest-Volume-609446595',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=a50e9659-3e2d-4f04-b0ac-ae78b8f61fad,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='99747be727ae4592ae1b6176b488719a',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:42:41Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:42:41Z,use_quota=True,user_id='0696808ea0b14c0794e8664a37d9e2c3',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:42:40 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (dfa120d7-a8b0-4768-af04-6169c215bc02) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:42:40 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (dfa120d7-a8b0-4768-af04-6169c215bc02) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:42:40 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (54657ef1-92e1-4332-aa7f-06f8cdbe1b9e) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:42:40 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (54657ef1-92e1-4332-aa7f-06f8cdbe1b9e) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': 'a50e9659-3e2d-4f04-b0ac-ae78b8f61fad', 'volume_name': 'volume-a50e9659-3e2d-4f04-b0ac-ae78b8f61fad', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:42:40 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (4e5a97b9-0fff-4ea1-a88c-d5bc693ef21f) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:42:40 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (4e5a97b9-0fff-4ea1-a88c-d5bc693ef21f) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:42:40 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (ff49899e-7263-464c-9cc2-14c942422b2b) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:42:40 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Volume a50e9659-3e2d-4f04-b0ac-ae78b8f61fad: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-a50e9659-3e2d-4f04-b0ac-ae78b8f61fad', 'volume_size': 1} Apr 17 15:42:40 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] creating volume v_a50e9659-3e2d-4f04-b0ac-ae78b8f61fad. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 15:42:40 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] create volume {'name': 'v_a50e9659-3e2d-4f04-b0ac-ae78b8f61fad', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 15:42:40 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:42:40 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:42:43 user cinder-volume[84696]: DEBUG cinder.utils [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.840(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:42:43 user cinder-volume[84696]: DEBUG cinder.utils [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_57070f23-6b40-47b9-b9b4-e0d77240f544/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:42:43 user cinder-volume[84696]: DEBUG cinder.utils [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.832(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:42:43 user cinder-volume[84696]: DEBUG cinder.utils [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_a50e9659-3e2d-4f04-b0ac-ae78b8f61fad/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:42:43 user cinder-volume[84696]: DEBUG cinder.utils [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.494(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:42:43 user cinder-volume[84696]: DEBUG cinder.utils [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:42:43 user cinder-volume[84696]: DEBUG cinder.utils [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.601(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:42:43 user cinder-volume[84696]: DEBUG cinder.utils [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:42:46 user cinder-volume[84696]: DEBUG cinder.utils [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.908(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:42:46 user cinder-volume[84696]: DEBUG cinder.utils [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_57070f23-6b40-47b9-b9b4-e0d77240f544/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:42:46 user cinder-volume[84696]: DEBUG cinder.utils [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.560(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:42:46 user cinder-volume[84696]: DEBUG cinder.utils [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:42:46 user cinder-volume[84696]: DEBUG cinder.utils [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.668(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:42:46 user cinder-volume[84696]: DEBUG cinder.utils [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:42:46 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.287(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:42:46 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:42:46 user cinder-volume[84696]: DEBUG cinder.utils [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.903(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:42:46 user cinder-volume[84696]: DEBUG cinder.utils [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_a50e9659-3e2d-4f04-b0ac-ae78b8f61fad/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:42:49 user cinder-volume[84696]: DEBUG cinder.utils [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 74.506(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:42:49 user cinder-volume[84696]: DEBUG cinder.utils [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_29d9e55e-0b8b-4f70-b5c4-830d39f10fcd/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:42:49 user cinder-volume[84696]: DEBUG cinder.utils [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 139.087(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:42:49 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_8cc35ab1-363f-4ad6-9a37-239d2b1ea996/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:42:49 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (5bd01a1e-e653-4328-bb49-1a979b1d0544) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 15:42:49 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (bfbef947-a2ca-4dd5-83dc-38072220fdc8) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:42:49 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (bfbef947-a2ca-4dd5-83dc-38072220fdc8) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:42:49 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Flow 'volume_create_manager' (649c76c3-4271-423a-afd7-abe4e42d4197) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 15:42:49 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-822bc81b-4af7-4475-af70-370e1870fe50 tempest-ServersNegativeTestJSON-1985071109 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 15:42:49 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 15:42:49 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 15:42:49 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 15:42:49 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 15:42:49 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 15:42:49 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 15:42:49 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 15:42:49 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 15:42:49 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 15:42:49 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 15:42:49 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 15:42:49 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 15:42:49 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 15:42:49 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 15:42:49 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 15:42:49 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 15:42:49 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 15:42:49 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 15:42:49 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 15:42:49 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 15:42:49 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 15:42:49 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 15:42:49 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 15:42:49 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:42:49 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 15:42:49 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 15:42:49 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 15:42:49 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 15:42:49 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 15:42:49 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 15:42:49 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 15:42:49 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 15:42:49 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 15:42:49 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 15:42:49 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 15:42:49 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 15:42:49 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 15:42:49 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 15:42:49 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 15:42:49 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 15:42:49 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 15:42:52 user cinder-volume[84696]: DEBUG cinder.utils [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 11.808(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:42:52 user cinder-volume[84696]: DEBUG cinder.utils [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:42:52 user cinder-volume[84696]: DEBUG cinder.utils [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.043(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:42:52 user cinder-volume[84696]: DEBUG cinder.utils [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_a50e9659-3e2d-4f04-b0ac-ae78b8f61fad/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:42:52 user cinder-volume[84696]: DEBUG cinder.utils [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.054(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:42:52 user cinder-volume[84696]: DEBUG cinder.utils [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_57070f23-6b40-47b9-b9b4-e0d77240f544/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:42:52 user cinder-volume[84696]: DEBUG cinder.utils [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 139.101(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:42:52 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_7ce815ce-9603-4fd6-8cc8-9fd798b2f99d/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:42:52 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (6b63b69f-daf6-4306-9c39-dcb57b647423) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 15:42:52 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (66d5a1b6-9c8e-411d-a2ef-eafd6a5bb8b1) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:42:52 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (66d5a1b6-9c8e-411d-a2ef-eafd6a5bb8b1) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:42:52 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Flow 'volume_create_manager' (aa44070f-2660-491a-8952-dbba9d00738f) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 15:42:52 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 15:42:52 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 15:42:52 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 15:42:52 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 15:42:52 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 15:42:52 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 15:42:52 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 15:42:52 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 15:42:52 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 15:42:52 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 15:42:52 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 15:42:52 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 15:42:52 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 15:42:52 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 15:42:52 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 15:42:52 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 15:42:52 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 15:42:52 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 15:42:52 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 15:42:52 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 15:42:52 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 15:42:52 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 15:42:52 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 15:42:52 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 15:42:52 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:42:52 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 15:42:52 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 15:42:52 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 15:42:52 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 15:42:52 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 15:42:52 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 15:42:52 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 15:42:52 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 15:42:52 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 15:42:52 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 15:42:52 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 15:42:52 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 15:42:52 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 15:42:52 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 15:42:52 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 15:42:52 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 15:42:52 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 15:42:52 user cinder-volume[84696]: DEBUG cinder.utils [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 138.955(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:42:52 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:42:52 user cinder-volume[84696]: INFO cinder.message.api [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Creating message record for request_id = req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b Apr 17 15:42:52 user cinder-volume[84696]: DEBUG cinder.utils [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 11.729(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:42:52 user cinder-volume[84696]: DEBUG cinder.utils [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:42:52 user cinder-volume[84696]: DEBUG cinder.utils [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 74.545(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:42:52 user cinder-volume[84696]: DEBUG cinder.utils [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_44073502-102d-452a-8244-69a8233b9060/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:42:52 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 15:42:52 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (6f9fb967-ca8d-4eba-832e-081aac180111) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 15:42:52 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 15:42:52 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:38:17Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesSnapshotTestJSON-Volume-1408369126',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=7ce815ce-9603-4fd6-8cc8-9fd798b2f99d,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='aacddeb0a9ec4a0d8d832595382e1189',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:40:34Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:40:34Z,use_quota=True,user_id='fddc0a19102c4ee7920f657f195171c9',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 15:42:52 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:38:17Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesSnapshotTestJSON-Volume-1408369126',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=7ce815ce-9603-4fd6-8cc8-9fd798b2f99d,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='aacddeb0a9ec4a0d8d832595382e1189',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:40:34Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:40:34Z,use_quota=True,user_id='fddc0a19102c4ee7920f657f195171c9',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(7ce815ce-9603-4fd6-8cc8-9fd798b2f99d),volume_id=7ce815ce-9603-4fd6-8cc8-9fd798b2f99d,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': '7ce815ce-9603-4fd6-8cc8-9fd798b2f99d', 'volume_name': 'volume-7ce815ce-9603-4fd6-8cc8-9fd798b2f99d', 'volume_size': 1}} Apr 17 15:42:52 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': '7ce815ce-9603-4fd6-8cc8-9fd798b2f99d', 'volume': {'id': '7ce815ce-9603-4fd6-8cc8-9fd798b2f99d', '_name_id': None, 'ec2_id': None, 'user_id': 'fddc0a19102c4ee7920f657f195171c9', 'project_id': 'aacddeb0a9ec4a0d8d832595382e1189', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-VolumesSnapshotTestJSON-Volume-1408369126', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'admin_metadata': {}, 'glance_metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T15:38:16.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-7ce815ce-9603-4fd6-8cc8-9fd798b2f99d', 'name_id': '7ce815ce-9603-4fd6-8cc8-9fd798b2f99d', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumesSnapshotTestJSON-Volume-1408369126', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': 'aacddeb0a9ec4a0d8d832595382e1189', 'qos_specs': None, 'reservations': ['f313e2f7-2050-44c4-b79e-0ecee302728c', '287d06a9-6c9c-4bb1-ba13-85b7cf2d8b7e', '14fdea43-dc14-4800-b0fe-8defcfbc5c61', '5659b6c2-9b08-423d-bdcb-0a1bab01869d'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'fddc0a19102c4ee7920f657f195171c9', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumesSnapshotTestJSON-Volume-1408369126', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': 'aacddeb0a9ec4a0d8d832595382e1189', 'qos_specs': None, 'reservations': ['f313e2f7-2050-44c4-b79e-0ecee302728c', '287d06a9-6c9c-4bb1-ba13-85b7cf2d8b7e', '14fdea43-dc14-4800-b0fe-8defcfbc5c61', '5659b6c2-9b08-423d-bdcb-0a1bab01869d'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'fddc0a19102c4ee7920f657f195171c9', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': 'fddc0a19102c4ee7920f657f195171c9', 'metadata': {}, 'qos_specs': None, 'retry': {'num_attempts': 2, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']}}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(7ce815ce-9603-4fd6-8cc8-9fd798b2f99d),volume_id=7ce815ce-9603-4fd6-8cc8-9fd798b2f99d,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:38:17Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesSnapshotTestJSON-Volume-1408369126',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=7ce815ce-9603-4fd6-8cc8-9fd798b2f99d,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='aacddeb0a9ec4a0d8d832595382e1189',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:40:34Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:40:34Z,use_quota=True,user_id='fddc0a19102c4ee7920f657f195171c9',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 15:42:52 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:38:17Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesSnapshotTestJSON-Volume-1408369126',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=7ce815ce-9603-4fd6-8cc8-9fd798b2f99d,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='aacddeb0a9ec4a0d8d832595382e1189',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:40:34Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:40:34Z,use_quota=True,user_id='fddc0a19102c4ee7920f657f195171c9',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:38:17Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesSnapshotTestJSON-Volume-1408369126',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=7ce815ce-9603-4fd6-8cc8-9fd798b2f99d,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='aacddeb0a9ec4a0d8d832595382e1189',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:40:34Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:40:34Z,use_quota=True,user_id='fddc0a19102c4ee7920f657f195171c9',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 15:42:52 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 15:42:52 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 15:42:52 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:42:52 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 15:42:52 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 15:42:52 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 15:42:52 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 15:42:52 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 15:42:52 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 15:42:52 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 15:42:52 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 15:42:52 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 15:42:52 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 15:42:52 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 15:42:52 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 15:42:52 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 15:42:52 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 15:42:52 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 15:42:52 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 15:42:52 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 15:42:52 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (6f9fb967-ca8d-4eba-832e-081aac180111) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:42:52 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (6f9fb967-ca8d-4eba-832e-081aac180111) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:42:52 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (74005d33-c134-4cca-8c1a-d44af556cede) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:42:52 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (74005d33-c134-4cca-8c1a-d44af556cede) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:42:52 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (f95baaef-aff1-42f9-a9bb-b8e18cec8961) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:42:52 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (f95baaef-aff1-42f9-a9bb-b8e18cec8961) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:42:52 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (018eee5d-5087-4932-97d7-887e2a16ecd9) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:42:52 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Updating volume 7ce815ce-9603-4fd6-8cc8-9fd798b2f99d with {'scheduled_at': datetime.datetime(2023, 4, 17, 15, 42, 52, 727449), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 15:42:52 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Volume 7ce815ce-9603-4fd6-8cc8-9fd798b2f99d: re-scheduling SchedulerAPI.create_volume attempt 2 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 15:42:52 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Volume 7ce815ce-9603-4fd6-8cc8-9fd798b2f99d: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 15:42:52 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] deleating volume v_7ce815ce-9603-4fd6-8cc8-9fd798b2f99d {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 15:42:52 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] get snapshots for volume v_7ce815ce-9603-4fd6-8cc8-9fd798b2f99d {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 15:42:52 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_7ce815ce-9603-4fd6-8cc8-9fd798b2f99d/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:42:52 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_7ce815ce-9603-4fd6-8cc8-9fd798b2f99d/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:42:52 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Volume reschedule parameters: True retry: {'num_attempts': 3, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 15:42:52 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Flow 'volume_create_manager' (c4aa3ff4-61e1-444d-8ac9-70cbf23f00cf) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:42:52 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (af5be1b2-adf5-4c31-8f79-d06f505378fb) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:42:52 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (af5be1b2-adf5-4c31-8f79-d06f505378fb) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:38:17Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesSnapshotTestJSON-Volume-1408369126',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=7ce815ce-9603-4fd6-8cc8-9fd798b2f99d,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='aacddeb0a9ec4a0d8d832595382e1189',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:42:53Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:42:53Z,use_quota=True,user_id='fddc0a19102c4ee7920f657f195171c9',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:42:52 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (a6c87ba5-c4ef-454e-93ad-6732d94f5e50) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:42:52 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (a6c87ba5-c4ef-454e-93ad-6732d94f5e50) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:42:52 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (04cc913a-66b1-4868-aafb-08b4610c7827) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:42:52 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (04cc913a-66b1-4868-aafb-08b4610c7827) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': '7ce815ce-9603-4fd6-8cc8-9fd798b2f99d', 'volume_name': 'volume-7ce815ce-9603-4fd6-8cc8-9fd798b2f99d', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:42:52 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (2acfee05-b7aa-4c77-85f8-1e7fabed8148) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:42:52 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (2acfee05-b7aa-4c77-85f8-1e7fabed8148) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:42:52 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (bbd68265-417d-4306-9a9a-8a84172aa22c) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:42:52 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Volume 7ce815ce-9603-4fd6-8cc8-9fd798b2f99d: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-7ce815ce-9603-4fd6-8cc8-9fd798b2f99d', 'volume_size': 1} Apr 17 15:42:52 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] creating volume v_7ce815ce-9603-4fd6-8cc8-9fd798b2f99d. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 15:42:52 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] create volume {'name': 'v_7ce815ce-9603-4fd6-8cc8-9fd798b2f99d', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 15:42:52 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:42:52 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:42:55 user cinder-volume[84696]: DEBUG cinder.utils [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.952(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:42:55 user cinder-volume[84696]: DEBUG cinder.utils [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_7ce815ce-9603-4fd6-8cc8-9fd798b2f99d/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:42:55 user cinder-volume[84696]: DEBUG cinder.utils [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.777(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:42:55 user cinder-volume[84696]: DEBUG cinder.utils [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:42:55 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.504(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:42:55 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:42:58 user cinder-volume[84696]: DEBUG cinder.utils [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.023(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:42:58 user cinder-volume[84696]: DEBUG cinder.utils [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_7ce815ce-9603-4fd6-8cc8-9fd798b2f99d/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:42:58 user cinder-volume[84696]: DEBUG cinder.utils [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.846(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:42:58 user cinder-volume[84696]: DEBUG cinder.utils [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:43:01 user cinder-volume[84696]: DEBUG cinder.utils [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 71.442(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:43:01 user cinder-volume[84696]: DEBUG cinder.utils [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_153b4df8-1f37-4118-8aa6-1c155ea5700c/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:43:01 user cinder-volume[84696]: DEBUG cinder.utils [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.260(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:43:01 user cinder-volume[84696]: DEBUG cinder.utils [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_a50e9659-3e2d-4f04-b0ac-ae78b8f61fad/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:43:01 user cinder-volume[84696]: DEBUG cinder.utils [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.028(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:43:01 user cinder-volume[84696]: DEBUG cinder.utils [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:43:01 user cinder-volume[84696]: DEBUG cinder.utils [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.273(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:43:01 user cinder-volume[84696]: DEBUG cinder.utils [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_57070f23-6b40-47b9-b9b4-e0d77240f544/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:43:01 user cinder-volume[84696]: DEBUG cinder.utils [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 20.925(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:43:01 user cinder-volume[84696]: DEBUG cinder.utils [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:43:04 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=22,cinder:UPDATE=8,cinder:INSERT=3 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:43:04 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=18,cinder:UPDATE=5,cinder:INSERT=3 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:43:04 user cinder-volume[84696]: DEBUG cinder.utils [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.167(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:43:04 user cinder-volume[84696]: DEBUG cinder.utils [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_7ce815ce-9603-4fd6-8cc8-9fd798b2f99d/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:43:04 user cinder-volume[84696]: DEBUG cinder.utils [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 11.990(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:43:04 user cinder-volume[84696]: DEBUG cinder.utils [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:43:14 user cinder-volume[84696]: DEBUG cinder.utils [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.849(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:43:14 user cinder-volume[84696]: DEBUG cinder.utils [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:43:14 user cinder-volume[84696]: DEBUG cinder.utils [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 22.029(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:43:14 user cinder-volume[84696]: DEBUG cinder.utils [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_7ce815ce-9603-4fd6-8cc8-9fd798b2f99d/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:43:14 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 40.577(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:43:14 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:43:20 user cinder-volume[84696]: DEBUG cinder.utils [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.990(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:43:20 user cinder-volume[84696]: DEBUG cinder.utils [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:43:20 user cinder-volume[84696]: DEBUG cinder.utils [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 40.331(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:43:20 user cinder-volume[84696]: DEBUG cinder.utils [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_a50e9659-3e2d-4f04-b0ac-ae78b8f61fad/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:43:20 user cinder-volume[84696]: DEBUG cinder.utils [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 40.342(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:43:20 user cinder-volume[84696]: DEBUG cinder.utils [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_57070f23-6b40-47b9-b9b4-e0d77240f544/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:43:20 user cinder-volume[84696]: DEBUG cinder.utils [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 40.099(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:43:20 user cinder-volume[84696]: DEBUG cinder.utils [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:43:23 user cinder-volume[84696]: DEBUG cinder.utils [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 139.517(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:43:23 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:43:24 user cinder-volume[84696]: INFO cinder.message.api [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Creating message record for request_id = req-52df28ef-59c1-47f6-831b-cf97d2a31df8 Apr 17 15:43:24 user cinder-volume[84696]: DEBUG cinder.utils [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 139.754(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:43:24 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_a27d0ff3-5de9-4feb-82f7-d4571af88bb9/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:43:24 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (314a2907-5948-4ac0-85b7-c7ec1690c5b5) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 15:43:24 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (1bd8d5f5-00a3-49cb-804d-ac3208bc81f5) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:43:24 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (1bd8d5f5-00a3-49cb-804d-ac3208bc81f5) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:43:24 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Flow 'volume_create_manager' (ead7cc50-5ceb-43d4-9614-1ed0bfbb7450) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 15:43:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 15:43:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 15:43:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 15:43:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 15:43:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 15:43:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 15:43:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 15:43:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 15:43:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 15:43:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 15:43:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 15:43:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 15:43:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 15:43:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 15:43:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 15:43:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 15:43:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 15:43:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 15:43:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 15:43:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 15:43:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 15:43:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 15:43:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 15:43:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 15:43:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:43:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 15:43:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 15:43:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 15:43:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 15:43:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 15:43:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 15:43:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 15:43:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 15:43:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 15:43:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 15:43:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 15:43:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 15:43:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 15:43:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 15:43:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 15:43:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 15:43:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 15:43:24 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 15:43:24 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (f299f18a-17e0-4661-9805-d82f9894a488) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 15:43:24 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 15:43:24 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:38:47Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-ServersNegativeTestJSON-volume-2082564469',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=a27d0ff3-5de9-4feb-82f7-d4571af88bb9,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='d2dd48f279404859bebcc19bcc272915',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:41:04Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:41:04Z,use_quota=True,user_id='b0075b26d6044a50981d29df0d863959',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 15:43:24 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:38:47Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-ServersNegativeTestJSON-volume-2082564469',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=a27d0ff3-5de9-4feb-82f7-d4571af88bb9,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='d2dd48f279404859bebcc19bcc272915',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:41:04Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:41:04Z,use_quota=True,user_id='b0075b26d6044a50981d29df0d863959',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(a27d0ff3-5de9-4feb-82f7-d4571af88bb9),volume_id=a27d0ff3-5de9-4feb-82f7-d4571af88bb9,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': 'a27d0ff3-5de9-4feb-82f7-d4571af88bb9', 'volume_name': 'volume-a27d0ff3-5de9-4feb-82f7-d4571af88bb9', 'volume_size': 1}} Apr 17 15:43:24 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': 'a27d0ff3-5de9-4feb-82f7-d4571af88bb9', 'volume': {'id': 'a27d0ff3-5de9-4feb-82f7-d4571af88bb9', '_name_id': None, 'ec2_id': None, 'user_id': 'b0075b26d6044a50981d29df0d863959', 'project_id': 'd2dd48f279404859bebcc19bcc272915', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-ServersNegativeTestJSON-volume-2082564469', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'admin_metadata': {}, 'glance_metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T15:38:47.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-a27d0ff3-5de9-4feb-82f7-d4571af88bb9', 'name_id': 'a27d0ff3-5de9-4feb-82f7-d4571af88bb9', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-ServersNegativeTestJSON-volume-2082564469', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': 'd2dd48f279404859bebcc19bcc272915', 'qos_specs': None, 'reservations': ['5c815076-6ae9-452c-aa1e-9825025560c9', 'b96b1904-2fdf-4b5b-9c8d-61562d250d61', '569a260d-012a-4ff0-b9af-61caad629af7', 'c933f679-b335-4a11-b551-a5c307704f2e'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'b0075b26d6044a50981d29df0d863959', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-ServersNegativeTestJSON-volume-2082564469', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': 'd2dd48f279404859bebcc19bcc272915', 'qos_specs': None, 'reservations': ['5c815076-6ae9-452c-aa1e-9825025560c9', 'b96b1904-2fdf-4b5b-9c8d-61562d250d61', '569a260d-012a-4ff0-b9af-61caad629af7', 'c933f679-b335-4a11-b551-a5c307704f2e'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'b0075b26d6044a50981d29df0d863959', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': 'b0075b26d6044a50981d29df0d863959', 'metadata': {}, 'qos_specs': None, 'retry': {'num_attempts': 2, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']}}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(a27d0ff3-5de9-4feb-82f7-d4571af88bb9),volume_id=a27d0ff3-5de9-4feb-82f7-d4571af88bb9,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:38:47Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-ServersNegativeTestJSON-volume-2082564469',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=a27d0ff3-5de9-4feb-82f7-d4571af88bb9,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='d2dd48f279404859bebcc19bcc272915',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:41:04Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:41:04Z,use_quota=True,user_id='b0075b26d6044a50981d29df0d863959',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 15:43:24 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:38:47Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-ServersNegativeTestJSON-volume-2082564469',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=a27d0ff3-5de9-4feb-82f7-d4571af88bb9,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='d2dd48f279404859bebcc19bcc272915',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:41:04Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:41:04Z,use_quota=True,user_id='b0075b26d6044a50981d29df0d863959',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:38:47Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-ServersNegativeTestJSON-volume-2082564469',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=a27d0ff3-5de9-4feb-82f7-d4571af88bb9,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='d2dd48f279404859bebcc19bcc272915',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:41:04Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:41:04Z,use_quota=True,user_id='b0075b26d6044a50981d29df0d863959',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 15:43:24 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 15:43:24 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 15:43:24 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:43:24 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 15:43:24 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 15:43:24 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 15:43:24 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 15:43:24 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 15:43:24 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 15:43:24 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 15:43:24 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 15:43:24 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 15:43:24 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 15:43:24 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 15:43:24 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 15:43:24 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 15:43:24 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 15:43:24 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 15:43:24 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 15:43:24 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 15:43:24 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (f299f18a-17e0-4661-9805-d82f9894a488) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:43:24 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (f299f18a-17e0-4661-9805-d82f9894a488) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:43:24 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (91ad3650-32d3-4064-a30b-91e6b806aee8) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:43:24 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (91ad3650-32d3-4064-a30b-91e6b806aee8) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:43:24 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (2b8ca4fa-2bf5-40a7-af8c-19e1e1cabd0b) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:43:24 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (2b8ca4fa-2bf5-40a7-af8c-19e1e1cabd0b) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:43:24 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (88bc4781-d148-4199-87b5-ca7f3c1d508f) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:43:24 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Updating volume a27d0ff3-5de9-4feb-82f7-d4571af88bb9 with {'scheduled_at': datetime.datetime(2023, 4, 17, 15, 43, 24, 89603), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 15:43:24 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Volume a27d0ff3-5de9-4feb-82f7-d4571af88bb9: re-scheduling SchedulerAPI.create_volume attempt 2 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 15:43:24 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Volume a27d0ff3-5de9-4feb-82f7-d4571af88bb9: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 15:43:24 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] deleating volume v_a27d0ff3-5de9-4feb-82f7-d4571af88bb9 {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 15:43:24 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] get snapshots for volume v_a27d0ff3-5de9-4feb-82f7-d4571af88bb9 {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 15:43:24 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_a27d0ff3-5de9-4feb-82f7-d4571af88bb9/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:43:24 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_a27d0ff3-5de9-4feb-82f7-d4571af88bb9/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:43:24 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Volume reschedule parameters: True retry: {'num_attempts': 3, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 15:43:24 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Flow 'volume_create_manager' (a8d98d40-62ca-452b-87fd-500fc7aa8bbc) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:43:24 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (7fc90935-2625-4047-95ba-0dbd29cfe095) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:43:24 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (7fc90935-2625-4047-95ba-0dbd29cfe095) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:38:47Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-ServersNegativeTestJSON-volume-2082564469',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=a27d0ff3-5de9-4feb-82f7-d4571af88bb9,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='d2dd48f279404859bebcc19bcc272915',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:43:24Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:43:24Z,use_quota=True,user_id='b0075b26d6044a50981d29df0d863959',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:43:24 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (00be8551-29ba-4eb4-bc63-b825cf11ce67) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:43:24 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (00be8551-29ba-4eb4-bc63-b825cf11ce67) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:43:24 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (dd3d5025-a3fb-4838-be81-6c29d4ff7554) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:43:24 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (dd3d5025-a3fb-4838-be81-6c29d4ff7554) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': 'a27d0ff3-5de9-4feb-82f7-d4571af88bb9', 'volume_name': 'volume-a27d0ff3-5de9-4feb-82f7-d4571af88bb9', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:43:24 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (bc855868-3d54-4c61-a937-dff15179b2ef) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:43:24 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (bc855868-3d54-4c61-a937-dff15179b2ef) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:43:24 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (0113ba91-d3db-43cb-bfe3-b28bfa3a1afc) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:43:24 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Volume a27d0ff3-5de9-4feb-82f7-d4571af88bb9: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-a27d0ff3-5de9-4feb-82f7-d4571af88bb9', 'volume_size': 1} Apr 17 15:43:24 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] creating volume v_a27d0ff3-5de9-4feb-82f7-d4571af88bb9. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 15:43:24 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] create volume {'name': 'v_a27d0ff3-5de9-4feb-82f7-d4571af88bb9', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 15:43:24 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:43:24 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:43:27 user cinder-volume[84696]: DEBUG cinder.utils [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.726(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:43:27 user cinder-volume[84696]: DEBUG cinder.utils [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:43:27 user cinder-volume[84696]: DEBUG cinder.utils [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.951(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:43:27 user cinder-volume[84696]: DEBUG cinder.utils [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_a27d0ff3-5de9-4feb-82f7-d4571af88bb9/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:43:30 user cinder-volume[84696]: DEBUG cinder.utils [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.798(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:43:30 user cinder-volume[84696]: DEBUG cinder.utils [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:43:30 user cinder-volume[84696]: DEBUG cinder.utils [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.020(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:43:30 user cinder-volume[84696]: DEBUG cinder.utils [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_a27d0ff3-5de9-4feb-82f7-d4571af88bb9/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:43:33 user cinder-volume[84696]: DEBUG cinder.utils [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 40.277(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:43:33 user cinder-volume[84696]: DEBUG cinder.utils [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:43:33 user cinder-volume[84696]: DEBUG cinder.utils [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 40.457(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:43:33 user cinder-volume[84696]: DEBUG cinder.utils [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_7ce815ce-9603-4fd6-8cc8-9fd798b2f99d/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:43:33 user cinder-volume[84696]: DEBUG cinder.utils [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 139.720(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:43:33 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_e80c78b2-20c9-4f9b-8898-6ab131d1eb06/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:43:33 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (a9adefa1-d40c-47d2-91d7-eca949dad911) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 15:43:33 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (3e667095-326a-44c2-bf11-c7596ca0b673) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:43:33 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (3e667095-326a-44c2-bf11-c7596ca0b673) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:43:33 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Flow 'volume_create_manager' (5a27dc12-5c2b-4af0-b886-997e5825d26e) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 15:43:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-31dc7151-4287-4070-a9f9-0abbb6a67df4 tempest-TaggedBootDevicesTest-809025791 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 15:43:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 15:43:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 15:43:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 15:43:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 15:43:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 15:43:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 15:43:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 15:43:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 15:43:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 15:43:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 15:43:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 15:43:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 15:43:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 15:43:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 15:43:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 15:43:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 15:43:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 15:43:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 15:43:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 15:43:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 15:43:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 15:43:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 15:43:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 15:43:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:43:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 15:43:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1237, in execute Apr 17 15:43:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_from_image(context, Apr 17 15:43:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/utils.py", line 670, in _wrapper Apr 17 15:43:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return r(f, *args, **kwargs) Apr 17 15:43:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/tenacity/__init__.py", line 423, in __call__ Apr 17 15:43:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server do = self.iter(retry_state=retry_state) Apr 17 15:43:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/tenacity/__init__.py", line 360, in iter Apr 17 15:43:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return fut.result() Apr 17 15:43:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/lib/python3.10/concurrent/futures/_base.py", line 451, in result Apr 17 15:43:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self.__get_result() Apr 17 15:43:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result Apr 17 15:43:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self._exception Apr 17 15:43:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/tenacity/__init__.py", line 426, in __call__ Apr 17 15:43:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = fn(*args, **kwargs) Apr 17 15:43:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1125, in _create_from_image Apr 17 15:43:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_from_image_cache_or_download( Apr 17 15:43:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 944, in _create_from_image_cache_or_download Apr 17 15:43:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update, cloned = self._create_from_image_cache( Apr 17 15:43:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 843, in _create_from_image_cache Apr 17 15:43:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_from_source_volume( Apr 17 15:43:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 659, in _create_from_source_volume Apr 17 15:43:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self.driver.create_cloned_volume(volume, srcvol_ref) Apr 17 15:43:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 525, in create_cloned_volume Apr 17 15:43:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self._clone_object(vname, cvname) Apr 17 15:43:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 461, in _clone_object Apr 17 15:43:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_snapshot(oname, coname) Apr 17 15:43:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 639, in create_snapshot Apr 17 15:43:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201, 204): Apr 17 15:43:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 15:43:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 15:43:36 user cinder-volume[84696]: DEBUG cinder.utils [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 11.942(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:43:36 user cinder-volume[84696]: DEBUG cinder.utils [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:43:36 user cinder-volume[84696]: DEBUG cinder.utils [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.164(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:43:36 user cinder-volume[84696]: DEBUG cinder.utils [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_a27d0ff3-5de9-4feb-82f7-d4571af88bb9/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:43:36 user cinder-volume[84696]: DEBUG cinder.utils [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 137.507(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:43:36 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:43:36 user cinder-volume[84696]: INFO cinder.message.api [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Creating message record for request_id = req-51924a9e-5094-404b-8828-f76cf8bc761a Apr 17 15:43:36 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 15:43:36 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=11,cinder:INSERT=1,cinder:UPDATE=1 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:43:36 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=10,cinder:UPDATE=6,cinder:INSERT=2 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:43:36 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (dc7ba6cf-31c7-40fc-9aa4-64eb819b469b) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 15:43:36 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 15:43:36 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:41:18Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesActionsTest-Volume-1624790909',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=441910f8-d29c-4c9e-bc5b-780ad228c85d,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='a8b97ff6ed6047f0a33e74684a1c162b',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:41:19Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:41:19Z,use_quota=True,user_id='c993ceeb4a5c49eea14057e2125518ad',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 15:43:36 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:41:18Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesActionsTest-Volume-1624790909',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=441910f8-d29c-4c9e-bc5b-780ad228c85d,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='a8b97ff6ed6047f0a33e74684a1c162b',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:41:19Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:41:19Z,use_quota=True,user_id='c993ceeb4a5c49eea14057e2125518ad',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(441910f8-d29c-4c9e-bc5b-780ad228c85d),volume_id=441910f8-d29c-4c9e-bc5b-780ad228c85d,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': '441910f8-d29c-4c9e-bc5b-780ad228c85d', 'volume_name': 'volume-441910f8-d29c-4c9e-bc5b-780ad228c85d', 'volume_size': 1}} Apr 17 15:43:36 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'retry': {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']}, 'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': '441910f8-d29c-4c9e-bc5b-780ad228c85d', 'volume': {'id': '441910f8-d29c-4c9e-bc5b-780ad228c85d', '_name_id': None, 'ec2_id': None, 'user_id': 'c993ceeb4a5c49eea14057e2125518ad', 'project_id': 'a8b97ff6ed6047f0a33e74684a1c162b', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-VolumesActionsTest-Volume-1624790909', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T15:41:18.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-441910f8-d29c-4c9e-bc5b-780ad228c85d', 'name_id': '441910f8-d29c-4c9e-bc5b-780ad228c85d', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumesActionsTest-Volume-1624790909', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': 'a8b97ff6ed6047f0a33e74684a1c162b', 'qos_specs': None, 'reservations': ['78c7033e-ef76-463d-90b8-8624379b802e', '4c9ae8cb-21fa-4ffc-97d0-d16076593bb1', '7ea6c36f-c448-4bcd-9d91-ca9a7f03d807', '60f944ba-84eb-4ff9-b309-d6e9af0d6b61'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'c993ceeb4a5c49eea14057e2125518ad', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumesActionsTest-Volume-1624790909', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': 'a8b97ff6ed6047f0a33e74684a1c162b', 'qos_specs': None, 'reservations': ['78c7033e-ef76-463d-90b8-8624379b802e', '4c9ae8cb-21fa-4ffc-97d0-d16076593bb1', '7ea6c36f-c448-4bcd-9d91-ca9a7f03d807', '60f944ba-84eb-4ff9-b309-d6e9af0d6b61'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'c993ceeb4a5c49eea14057e2125518ad', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': 'c993ceeb4a5c49eea14057e2125518ad', 'metadata': {}, 'qos_specs': None}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(441910f8-d29c-4c9e-bc5b-780ad228c85d),volume_id=441910f8-d29c-4c9e-bc5b-780ad228c85d,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:41:18Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesActionsTest-Volume-1624790909',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=441910f8-d29c-4c9e-bc5b-780ad228c85d,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='a8b97ff6ed6047f0a33e74684a1c162b',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:41:19Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:41:19Z,use_quota=True,user_id='c993ceeb4a5c49eea14057e2125518ad',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 15:43:36 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:41:18Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesActionsTest-Volume-1624790909',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=441910f8-d29c-4c9e-bc5b-780ad228c85d,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='a8b97ff6ed6047f0a33e74684a1c162b',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:41:19Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:41:19Z,use_quota=True,user_id='c993ceeb4a5c49eea14057e2125518ad',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:41:18Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesActionsTest-Volume-1624790909',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=441910f8-d29c-4c9e-bc5b-780ad228c85d,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='a8b97ff6ed6047f0a33e74684a1c162b',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:41:19Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:41:19Z,use_quota=True,user_id='c993ceeb4a5c49eea14057e2125518ad',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 15:43:36 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 15:43:36 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 15:43:36 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:43:36 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 15:43:36 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 15:43:36 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 15:43:36 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 15:43:36 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 15:43:36 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 15:43:36 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 15:43:36 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 15:43:36 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 15:43:36 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 15:43:36 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 15:43:36 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 15:43:36 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 15:43:36 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 15:43:36 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 15:43:36 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 15:43:36 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 15:43:36 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (dc7ba6cf-31c7-40fc-9aa4-64eb819b469b) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:43:36 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (dc7ba6cf-31c7-40fc-9aa4-64eb819b469b) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:43:36 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (5937c4d5-cbe0-4096-9f56-e22a462541c1) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:43:36 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (5937c4d5-cbe0-4096-9f56-e22a462541c1) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:43:36 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (7e91bd90-72f9-4965-b566-ea790ae99d85) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:43:36 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (7e91bd90-72f9-4965-b566-ea790ae99d85) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:43:36 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (8f2b47e7-826d-42cd-b35a-0914962c14c7) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:43:36 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Updating volume 441910f8-d29c-4c9e-bc5b-780ad228c85d with {'scheduled_at': datetime.datetime(2023, 4, 17, 15, 43, 36, 348903), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 15:43:36 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Volume 441910f8-d29c-4c9e-bc5b-780ad228c85d: re-scheduling SchedulerAPI.create_volume attempt 1 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 15:43:36 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Volume 441910f8-d29c-4c9e-bc5b-780ad228c85d: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 15:43:36 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] deleating volume v_441910f8-d29c-4c9e-bc5b-780ad228c85d {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 15:43:36 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] get snapshots for volume v_441910f8-d29c-4c9e-bc5b-780ad228c85d {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 15:43:36 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_441910f8-d29c-4c9e-bc5b-780ad228c85d/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:43:36 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_441910f8-d29c-4c9e-bc5b-780ad228c85d/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:43:36 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Volume reschedule parameters: True retry: {'num_attempts': 2, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 15:43:36 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Flow 'volume_create_manager' (ec4c4cd2-651f-4aaf-b76f-a389f625dcdc) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:43:36 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (7b2b9368-64b3-4560-ba5b-59fd9f0b0863) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:43:36 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (7b2b9368-64b3-4560-ba5b-59fd9f0b0863) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:41:18Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesActionsTest-Volume-1624790909',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=441910f8-d29c-4c9e-bc5b-780ad228c85d,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='a8b97ff6ed6047f0a33e74684a1c162b',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:43:36Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:43:36Z,use_quota=True,user_id='c993ceeb4a5c49eea14057e2125518ad',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:43:36 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (4d0ba699-1018-4d1d-a6d5-05df49808c59) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:43:36 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (4d0ba699-1018-4d1d-a6d5-05df49808c59) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:43:36 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (a746d0ef-f626-4191-b33e-3af451243bd4) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:43:36 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (a746d0ef-f626-4191-b33e-3af451243bd4) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': '441910f8-d29c-4c9e-bc5b-780ad228c85d', 'volume_name': 'volume-441910f8-d29c-4c9e-bc5b-780ad228c85d', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:43:36 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (46017a24-ff2b-4f74-9087-847879a38773) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:43:36 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (46017a24-ff2b-4f74-9087-847879a38773) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:43:36 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (13dbd7c5-0d4f-4ac8-9785-2f46c4a60b54) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:43:36 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Volume 441910f8-d29c-4c9e-bc5b-780ad228c85d: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-441910f8-d29c-4c9e-bc5b-780ad228c85d', 'volume_size': 1} Apr 17 15:43:36 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] creating volume v_441910f8-d29c-4c9e-bc5b-780ad228c85d. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 15:43:36 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] create volume {'name': 'v_441910f8-d29c-4c9e-bc5b-780ad228c85d', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 15:43:36 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:43:36 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:43:39 user cinder-volume[84696]: DEBUG cinder.utils [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.977(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:43:39 user cinder-volume[84696]: DEBUG cinder.utils [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_441910f8-d29c-4c9e-bc5b-780ad228c85d/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:43:39 user cinder-volume[84696]: DEBUG cinder.utils [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.809(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:43:39 user cinder-volume[84696]: DEBUG cinder.utils [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:43:42 user cinder-volume[84696]: DEBUG cinder.utils [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.879(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:43:42 user cinder-volume[84696]: DEBUG cinder.utils [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:43:42 user cinder-volume[84696]: DEBUG cinder.utils [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.051(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:43:42 user cinder-volume[84696]: DEBUG cinder.utils [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_441910f8-d29c-4c9e-bc5b-780ad228c85d/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:43:43 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Volume reschedule parameters: True retry: {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 15:43:43 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Flow 'volume_create_manager' (52235598-0fc7-45e2-8428-7d140671d028) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:43:43 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (9e7517b4-e533-4710-8e8a-12463f40b91d) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:43:43 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (9e7517b4-e533-4710-8e8a-12463f40b91d) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:43:43Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TestMinimumBasicScenario-volume-1067752463',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=5c16bb82-43c5-4d91-9320-0e1f2bbec574,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='17d1dae0ad7149bbbd160525d64c0c20',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:43:43Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:43:43Z,use_quota=True,user_id='3c407d9d494d485c9c95dc3981284fba',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:43:43 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (ba07b316-c816-4732-abc1-488bfdbe265a) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:43:43 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (ba07b316-c816-4732-abc1-488bfdbe265a) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:43:43 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (1217957a-28bd-42a3-8ec2-82b3cc7890c8) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:43:43 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (1217957a-28bd-42a3-8ec2-82b3cc7890c8) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': '5c16bb82-43c5-4d91-9320-0e1f2bbec574', 'volume_name': 'volume-5c16bb82-43c5-4d91-9320-0e1f2bbec574', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:43:43 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (bd3deac7-067f-4638-802e-9e04e60d9396) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:43:43 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (bd3deac7-067f-4638-802e-9e04e60d9396) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:43:43 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (48562dfe-7934-46e1-bd66-7deffaf26fa9) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:43:43 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Volume 5c16bb82-43c5-4d91-9320-0e1f2bbec574: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-5c16bb82-43c5-4d91-9320-0e1f2bbec574', 'volume_size': 1} Apr 17 15:43:43 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] creating volume v_5c16bb82-43c5-4d91-9320-0e1f2bbec574. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 15:43:43 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] create volume {'name': 'v_5c16bb82-43c5-4d91-9320-0e1f2bbec574', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 15:43:43 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:43:43 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:43:45 user cinder-volume[84696]: DEBUG cinder.utils [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.379(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:43:45 user cinder-volume[84696]: DEBUG cinder.utils [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_a27d0ff3-5de9-4feb-82f7-d4571af88bb9/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:43:45 user cinder-volume[84696]: DEBUG cinder.utils [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.075(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:43:45 user cinder-volume[84696]: DEBUG cinder.utils [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:43:45 user cinder-volume[84696]: DEBUG cinder.utils [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.160(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:43:45 user cinder-volume[84696]: DEBUG cinder.utils [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:43:48 user cinder-volume[84696]: DEBUG cinder.utils [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.229(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:43:48 user cinder-volume[84696]: DEBUG cinder.utils [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_441910f8-d29c-4c9e-bc5b-780ad228c85d/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:43:48 user cinder-volume[84696]: DEBUG cinder.utils [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.061(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:43:48 user cinder-volume[84696]: DEBUG cinder.utils [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:43:48 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 74.402(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:43:48 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:43:48 user cinder-volume[84696]: DEBUG cinder.utils [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.186(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:43:48 user cinder-volume[84696]: DEBUG cinder.utils [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:43:54 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=9,cinder:UPDATE=4,cinder:INSERT=1 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:43:55 user cinder-volume[84696]: DEBUG cinder.utils [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 11.609(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:43:55 user cinder-volume[84696]: DEBUG cinder.utils [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:43:55 user cinder-volume[84696]: DEBUG cinder.utils [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 74.104(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:43:55 user cinder-volume[84696]: DEBUG cinder.utils [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:43:55 user cinder-volume[84696]: DEBUG cinder.utils [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 74.210(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:43:55 user cinder-volume[84696]: DEBUG cinder.utils [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:43:55 user cinder-volume[84696]: DEBUG cinder.utils [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 74.455(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:43:55 user cinder-volume[84696]: DEBUG cinder.utils [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_57070f23-6b40-47b9-b9b4-e0d77240f544/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:43:55 user cinder-volume[84696]: DEBUG cinder.utils [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 139.978(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:43:55 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_29d9e55e-0b8b-4f70-b5c4-830d39f10fcd/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:43:55 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (993cffe6-ba6b-4465-b578-d70fa9f61aa4) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 15:43:55 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (36fbe028-1cf1-477f-a456-9ddceaa032ac) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:43:55 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (36fbe028-1cf1-477f-a456-9ddceaa032ac) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:43:55 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Flow 'volume_create_manager' (849d3848-80f5-4a4d-8bd9-2bbb47a5c5c3) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 15:43:55 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-60e7b05f-e229-45c2-a0b3-cd53ab13bb63 tempest-VolumesSnapshotNegativeTestJSON-225666610 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 15:43:55 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 15:43:55 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 15:43:55 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 15:43:55 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 15:43:55 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 15:43:55 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 15:43:55 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 15:43:55 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 15:43:55 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 15:43:55 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 15:43:55 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 15:43:55 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 15:43:55 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 15:43:55 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 15:43:55 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 15:43:55 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 15:43:55 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 15:43:55 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 15:43:55 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 15:43:55 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 15:43:55 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 15:43:55 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 15:43:55 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 15:43:55 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:43:55 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 15:43:55 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 15:43:55 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 15:43:55 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 15:43:55 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 15:43:55 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 15:43:55 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 15:43:55 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 15:43:55 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 15:43:55 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 15:43:55 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 15:43:55 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 15:43:55 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 15:43:55 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 15:43:55 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 15:43:55 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 15:43:55 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 15:43:55 user cinder-volume[84696]: DEBUG cinder.utils [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 74.465(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:43:55 user cinder-volume[84696]: DEBUG cinder.utils [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_a50e9659-3e2d-4f04-b0ac-ae78b8f61fad/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:43:58 user cinder-volume[84696]: DEBUG cinder.utils [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.559(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:43:58 user cinder-volume[84696]: DEBUG cinder.utils [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:43:58 user cinder-volume[84696]: DEBUG cinder.utils [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.730(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:43:58 user cinder-volume[84696]: DEBUG cinder.utils [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_441910f8-d29c-4c9e-bc5b-780ad228c85d/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:43:58 user cinder-volume[84696]: DEBUG cinder.utils [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 139.990(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:43:58 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_44073502-102d-452a-8244-69a8233b9060/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:43:58 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (3ce1e426-0c16-497b-adfe-a54970dc7680) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 15:43:58 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (b8ed0dc1-929c-461f-b9e4-2dad2e96b158) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:43:58 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (b8ed0dc1-929c-461f-b9e4-2dad2e96b158) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:43:58 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Flow 'volume_create_manager' (e8170c0b-3484-48ff-9fce-b5176a90fd27) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 15:43:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-ec7f2ef2-a4da-41de-8aa1-8396b20368a6 tempest-VolumesActionsTest-707453245 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 15:43:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 15:43:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 15:43:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 15:43:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 15:43:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 15:43:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 15:43:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 15:43:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 15:43:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 15:43:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 15:43:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 15:43:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 15:43:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 15:43:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 15:43:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 15:43:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 15:43:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 15:43:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 15:43:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 15:43:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 15:43:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 15:43:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 15:43:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 15:43:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:43:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 15:43:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 15:43:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 15:43:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 15:43:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 15:43:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 15:43:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 15:43:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 15:43:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 15:43:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 15:43:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 15:43:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 15:43:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 15:43:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 15:43:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 15:43:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 15:43:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 15:44:02 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Volume reschedule parameters: True retry: {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 15:44:02 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Flow 'volume_create_manager' (398ad45c-928a-4370-97f4-8e626bb3991b) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:44:02 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (997e1140-d27b-408f-85ff-51beb6aecd35) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:44:02 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (997e1140-d27b-408f-85ff-51beb6aecd35) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:44:03Z,deleted=False,deleted_at=None,display_description='',display_name='05e08eea-cc2c-4332-9e97-07cb9b466af6-blank-vol',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=2c9714de-f8ea-4d1d-99f9-beb85126ad91,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='540826be4baf46dba9617b2fd47821a3',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:44:03Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:44:03Z,use_quota=True,user_id='d819e3f51368413c99691d1b45bbcfb1',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:44:02 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (91eb764b-f46d-44d6-b36e-ef5e6fb68451) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:44:02 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (91eb764b-f46d-44d6-b36e-ef5e6fb68451) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:44:02 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (1d0c2f5a-3d4a-4ec8-be80-f06f6886d0c3) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:44:02 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (1d0c2f5a-3d4a-4ec8-be80-f06f6886d0c3) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': '2c9714de-f8ea-4d1d-99f9-beb85126ad91', 'volume_name': 'volume-2c9714de-f8ea-4d1d-99f9-beb85126ad91', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:44:02 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (de8925ee-bbf5-4cf6-a515-9dca7f4b11cd) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:44:02 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (de8925ee-bbf5-4cf6-a515-9dca7f4b11cd) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:44:02 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (e2425f44-15f4-4105-a07c-08bc6bfa8c8e) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:44:02 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Volume 2c9714de-f8ea-4d1d-99f9-beb85126ad91: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-2c9714de-f8ea-4d1d-99f9-beb85126ad91', 'volume_size': 1} Apr 17 15:44:02 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] creating volume v_2c9714de-f8ea-4d1d-99f9-beb85126ad91. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 15:44:02 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] create volume {'name': 'v_2c9714de-f8ea-4d1d-99f9-beb85126ad91', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 15:44:02 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:44:02 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:44:04 user cinder-volume[84696]: DEBUG cinder.utils [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.910(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:44:04 user cinder-volume[84696]: DEBUG cinder.utils [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:44:04 user cinder-volume[84696]: DEBUG cinder.utils [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 20.827(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:44:04 user cinder-volume[84696]: DEBUG cinder.utils [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:44:04 user cinder-volume[84696]: DEBUG cinder.utils [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 40.133(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:44:04 user cinder-volume[84696]: DEBUG cinder.utils [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_a27d0ff3-5de9-4feb-82f7-d4571af88bb9/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:44:04 user cinder-volume[84696]: DEBUG cinder.utils [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 1.292(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:44:04 user cinder-volume[84696]: DEBUG cinder.utils [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:44:07 user cinder-volume[84696]: DEBUG cinder.utils [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 74.389(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:44:07 user cinder-volume[84696]: DEBUG cinder.utils [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:44:07 user cinder-volume[84696]: DEBUG cinder.utils [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 4.362(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:44:07 user cinder-volume[84696]: DEBUG cinder.utils [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:44:07 user cinder-volume[84696]: DEBUG cinder.utils [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 136.917(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:44:07 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_153b4df8-1f37-4118-8aa6-1c155ea5700c/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:44:07 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (7c086dcf-f653-40fe-aa4b-b1e6bb7df085) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 15:44:07 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (d479dfe8-d2ee-4a0b-9f0e-2b183635c771) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:44:07 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (d479dfe8-d2ee-4a0b-9f0e-2b183635c771) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:44:07 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Flow 'volume_create_manager' (e5ab2144-842b-4aa0-821b-f3352dc9c371) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 15:44:07 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-de22aca9-c3f8-4015-be3e-9d70b59359d2 tempest-VolumesCloneTest-1847825601 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 15:44:07 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 15:44:07 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 15:44:07 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 15:44:07 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 15:44:07 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 15:44:07 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 15:44:07 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 15:44:07 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 15:44:07 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 15:44:07 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 15:44:07 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 15:44:07 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 15:44:07 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 15:44:07 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 15:44:07 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 15:44:07 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 15:44:07 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 15:44:07 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 15:44:07 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 15:44:07 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 15:44:07 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 15:44:07 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 15:44:07 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 15:44:07 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:44:07 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 15:44:07 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 15:44:07 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 15:44:07 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 15:44:07 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 15:44:07 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 15:44:07 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 15:44:07 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 15:44:07 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 15:44:07 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 15:44:07 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 15:44:07 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 15:44:07 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 15:44:07 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 15:44:07 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 15:44:07 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 15:44:07 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 15:44:07 user cinder-volume[84696]: DEBUG cinder.utils [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 74.592(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:44:07 user cinder-volume[84696]: DEBUG cinder.utils [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_7ce815ce-9603-4fd6-8cc8-9fd798b2f99d/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:44:14 user cinder-volume[84696]: DEBUG cinder.utils [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 11.080(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:44:14 user cinder-volume[84696]: DEBUG cinder.utils [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:44:14 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=14,cinder:UPDATE=3 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:44:17 user cinder-volume[84696]: DEBUG cinder.utils [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 40.737(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:44:17 user cinder-volume[84696]: DEBUG cinder.utils [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_441910f8-d29c-4c9e-bc5b-780ad228c85d/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:44:17 user cinder-volume[84696]: DEBUG cinder.utils [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 40.569(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:44:17 user cinder-volume[84696]: DEBUG cinder.utils [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:44:23 user cinder-volume[84696]: DEBUG cinder.utils [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.833(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:44:23 user cinder-volume[84696]: DEBUG cinder.utils [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:44:23 user cinder-volume[84696]: DEBUG cinder.utils [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 20.297(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:44:23 user cinder-volume[84696]: DEBUG cinder.utils [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:44:24 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=9,cinder:UPDATE=3 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:44:38 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Volume reschedule parameters: True retry: {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 15:44:38 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Flow 'volume_create_manager' (1a9a0e5d-c7b0-4ded-a0f8-8dd4d09ecaf4) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:44:38 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (cc5db28e-1f6b-4789-b35a-420b13e30f80) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:44:38 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (cc5db28e-1f6b-4789-b35a-420b13e30f80) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:44:38Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-CreateVolumesFromImageTest-Volume-1463431787',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=dfd745ed-b0c4-4e35-b77c-a911c90ce33e,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='75babbbca67f4035a48ee8ee816b18db',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:44:39Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:44:39Z,use_quota=True,user_id='d5299006144f4ee6817349348a489017',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:44:38 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (b63ce73b-bcf8-4cfb-87d9-296580b1da26) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:44:38 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (b63ce73b-bcf8-4cfb-87d9-296580b1da26) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:44:38 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (c861ba13-463a-4282-8a81-0a0f685ac80a) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:44:38 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (c861ba13-463a-4282-8a81-0a0f685ac80a) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': 'dfd745ed-b0c4-4e35-b77c-a911c90ce33e', 'volume_name': 'volume-dfd745ed-b0c4-4e35-b77c-a911c90ce33e', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:44:38 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (c50e358a-199d-4c99-b7fa-60390bab6fb1) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:44:38 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (c50e358a-199d-4c99-b7fa-60390bab6fb1) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:44:38 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (e66d2ac4-ac3f-4a64-832b-f75a72f684a5) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:44:38 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Volume dfd745ed-b0c4-4e35-b77c-a911c90ce33e: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-dfd745ed-b0c4-4e35-b77c-a911c90ce33e', 'volume_size': 1} Apr 17 15:44:38 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] creating volume v_dfd745ed-b0c4-4e35-b77c-a911c90ce33e. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 15:44:38 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] create volume {'name': 'v_dfd745ed-b0c4-4e35-b77c-a911c90ce33e', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 15:44:38 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:44:38 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:44:38 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Volume reschedule parameters: True retry: {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 15:44:38 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Flow 'volume_create_manager' (c41a7b6c-c6fe-4d33-a8a4-0f6ddd345296) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:44:38 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (824dee1d-7348-4235-a659-2415ca567d67) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:44:38 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (824dee1d-7348-4235-a659-2415ca567d67) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:44:39Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-CreateVolumesFromImageTest-Volume-800604710',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=aea45c70-7c05-4146-847e-cbb777d3dfbe,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='75babbbca67f4035a48ee8ee816b18db',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:44:39Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:44:39Z,use_quota=True,user_id='d5299006144f4ee6817349348a489017',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:44:38 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (947d63b2-51cb-4ef3-83c3-96e17bebdb4c) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:44:38 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (947d63b2-51cb-4ef3-83c3-96e17bebdb4c) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:44:38 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (7b117907-bafc-4b55-9754-be5fb969a68b) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:44:38 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (7b117907-bafc-4b55-9754-be5fb969a68b) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': 'aea45c70-7c05-4146-847e-cbb777d3dfbe', 'volume_name': 'volume-aea45c70-7c05-4146-847e-cbb777d3dfbe', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:44:38 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (a8042c3a-402f-47d4-b497-1b5428ab1082) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:44:38 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (a8042c3a-402f-47d4-b497-1b5428ab1082) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:44:38 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (b50442f2-2fa6-4ae8-825e-4ae1ed629db3) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:44:38 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Volume aea45c70-7c05-4146-847e-cbb777d3dfbe: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-aea45c70-7c05-4146-847e-cbb777d3dfbe', 'volume_size': 1} Apr 17 15:44:38 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] creating volume v_aea45c70-7c05-4146-847e-cbb777d3dfbe. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 15:44:38 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] create volume {'name': 'v_aea45c70-7c05-4146-847e-cbb777d3dfbe', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 15:44:38 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:44:38 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:44:39 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Volume reschedule parameters: True retry: {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 15:44:39 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Flow 'volume_create_manager' (150566c6-b9f6-438e-9f32-f77b92c1a70e) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:44:39 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (dfed3372-6304-4ab9-89dd-8b11f2799c62) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:44:39 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (dfed3372-6304-4ab9-89dd-8b11f2799c62) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:44:39Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-CreateVolumesFromImageTest-Volume-2146530532',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=fbbadf16-651a-47d5-b308-a37e56aa07f2,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='75babbbca67f4035a48ee8ee816b18db',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:44:39Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:44:39Z,use_quota=True,user_id='d5299006144f4ee6817349348a489017',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:44:39 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (07ade4d8-0421-45fb-9322-2eed895f7d00) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:44:39 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (07ade4d8-0421-45fb-9322-2eed895f7d00) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:44:39 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (a3ce4a58-f2fc-40ce-951e-9808b120591b) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:44:39 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (a3ce4a58-f2fc-40ce-951e-9808b120591b) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': 'fbbadf16-651a-47d5-b308-a37e56aa07f2', 'volume_name': 'volume-fbbadf16-651a-47d5-b308-a37e56aa07f2', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:44:39 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (e329f716-c4db-4244-bd9f-1c4f59575094) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:44:39 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (e329f716-c4db-4244-bd9f-1c4f59575094) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:44:39 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (d5773f69-e8c3-40bd-9484-70f6d2778aa5) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:44:39 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Volume fbbadf16-651a-47d5-b308-a37e56aa07f2: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-fbbadf16-651a-47d5-b308-a37e56aa07f2', 'volume_size': 1} Apr 17 15:44:39 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] creating volume v_fbbadf16-651a-47d5-b308-a37e56aa07f2. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 15:44:39 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] create volume {'name': 'v_fbbadf16-651a-47d5-b308-a37e56aa07f2', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 15:44:39 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:44:39 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:44:39 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Volume reschedule parameters: True retry: {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 15:44:39 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Flow 'volume_create_manager' (70451a17-1a5a-4c30-9d62-5035d2806cfe) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:44:39 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (f29bb2ab-3b18-4e88-81f3-09c13e394081) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:44:39 user cinder-volume[84696]: DEBUG cinder.utils [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 75.208(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:44:39 user cinder-volume[84696]: DEBUG cinder.utils [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_a27d0ff3-5de9-4feb-82f7-d4571af88bb9/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:44:39 user cinder-volume[84696]: DEBUG cinder.utils [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 36.367(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:44:39 user cinder-volume[84696]: DEBUG cinder.utils [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:44:39 user cinder-volume[84696]: DEBUG cinder.utils [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 74.990(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:44:39 user cinder-volume[84696]: DEBUG cinder.utils [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:44:39 user cinder-volume[84696]: DEBUG cinder.utils [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 0.389(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:44:39 user cinder-volume[84696]: DEBUG cinder.utils [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:44:39 user cinder-volume[84696]: DEBUG cinder.utils [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 0.604(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:44:39 user cinder-volume[84696]: DEBUG cinder.utils [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:44:39 user cinder-volume[84696]: DEBUG cinder.utils [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 0.165(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:44:39 user cinder-volume[84696]: DEBUG cinder.utils [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:44:39 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (f29bb2ab-3b18-4e88-81f3-09c13e394081) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:44:39Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-CreateVolumesFromImageTest-Volume-1776307206',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=13cc73df-7780-4934-8cbd-7f372b1cf57a,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='75babbbca67f4035a48ee8ee816b18db',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:44:39Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:44:39Z,use_quota=True,user_id='d5299006144f4ee6817349348a489017',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:44:39 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (182b3545-4d79-41e7-8994-9e2a802d1769) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:44:39 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (182b3545-4d79-41e7-8994-9e2a802d1769) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:44:39 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (7060b283-2751-438b-acca-fdd61100417f) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:44:39 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (7060b283-2751-438b-acca-fdd61100417f) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': '13cc73df-7780-4934-8cbd-7f372b1cf57a', 'volume_name': 'volume-13cc73df-7780-4934-8cbd-7f372b1cf57a', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:44:39 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (9c125e0a-3123-49ab-b2f3-c2cf229225b5) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:44:39 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (9c125e0a-3123-49ab-b2f3-c2cf229225b5) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:44:39 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (6003413c-9a6d-4d1d-bc80-b0476dbdfa23) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:44:39 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Volume 13cc73df-7780-4934-8cbd-7f372b1cf57a: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-13cc73df-7780-4934-8cbd-7f372b1cf57a', 'volume_size': 1} Apr 17 15:44:39 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] creating volume v_13cc73df-7780-4934-8cbd-7f372b1cf57a. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 15:44:39 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] create volume {'name': 'v_13cc73df-7780-4934-8cbd-7f372b1cf57a', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 15:44:39 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:44:39 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:44:39 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Volume reschedule parameters: True retry: {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 15:44:39 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Flow 'volume_create_manager' (b73b5d1b-f361-4c7c-b68e-e1a4dfc5a76a) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:44:39 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (3086b358-7bdb-42d8-83a9-53a9139ba127) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:44:39 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (3086b358-7bdb-42d8-83a9-53a9139ba127) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:44:39Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-CreateVolumesFromImageTest-Volume-1478171265',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=ab5889f8-b6c7-4673-a666-f4b4e5ff5fd5,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='75babbbca67f4035a48ee8ee816b18db',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:44:39Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:44:39Z,use_quota=True,user_id='d5299006144f4ee6817349348a489017',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:44:39 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (34f4f4ff-2793-4469-b09b-4747e98a8a60) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:44:39 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (34f4f4ff-2793-4469-b09b-4747e98a8a60) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:44:39 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (aa4c31be-b709-4965-91f8-2f89acf992d8) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:44:39 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (aa4c31be-b709-4965-91f8-2f89acf992d8) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': 'ab5889f8-b6c7-4673-a666-f4b4e5ff5fd5', 'volume_name': 'volume-ab5889f8-b6c7-4673-a666-f4b4e5ff5fd5', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:44:39 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (a2033970-30aa-483e-908d-181c3c80d670) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:44:39 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (a2033970-30aa-483e-908d-181c3c80d670) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:44:39 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (33b88de9-f6af-4460-b2a0-b6ca9b4da0ac) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:44:39 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Volume ab5889f8-b6c7-4673-a666-f4b4e5ff5fd5: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-ab5889f8-b6c7-4673-a666-f4b4e5ff5fd5', 'volume_size': 1} Apr 17 15:44:39 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] creating volume v_ab5889f8-b6c7-4673-a666-f4b4e5ff5fd5. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 15:44:39 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] create volume {'name': 'v_ab5889f8-b6c7-4673-a666-f4b4e5ff5fd5', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 15:44:39 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:44:39 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:44:42 user cinder-volume[84696]: DEBUG cinder.utils [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 3.025(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:44:42 user cinder-volume[84696]: DEBUG cinder.utils [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:44:42 user cinder-volume[84696]: DEBUG cinder.utils [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.814(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:44:42 user cinder-volume[84696]: DEBUG cinder.utils [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:44:42 user cinder-volume[84696]: DEBUG cinder.utils [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 3.230(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:44:42 user cinder-volume[84696]: DEBUG cinder.utils [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:44:42 user cinder-volume[84696]: DEBUG cinder.utils [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 3.457(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:44:42 user cinder-volume[84696]: DEBUG cinder.utils [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:44:42 user cinder-volume[84696]: DEBUG cinder.utils [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 3.672(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:44:42 user cinder-volume[84696]: DEBUG cinder.utils [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:44:42 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Volume reschedule parameters: True retry: {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 15:44:42 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Flow 'volume_create_manager' (c06604dc-2077-4b09-9327-56c070b16b86) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:44:42 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (ab78d32b-45bb-465a-9c4d-e05cc2e00585) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:44:42 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (ab78d32b-45bb-465a-9c4d-e05cc2e00585) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:44:42Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-ServerRescueNegativeTestJSON-volume-1878278176',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=68f814b0-1291-4a12-ae65-dc0892203d36,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='9324c99bd4d040bdb5a7f1db782b2783',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:44:43Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:44:43Z,use_quota=True,user_id='8e3487d6b32d46beac556617050fd664',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:44:42 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (78cd626e-1d4d-4838-8d0e-2d6b218f83b8) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:44:42 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (78cd626e-1d4d-4838-8d0e-2d6b218f83b8) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:44:42 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (86c42ee2-e424-4a87-b832-e2cad1fe942d) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:44:42 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (86c42ee2-e424-4a87-b832-e2cad1fe942d) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': '68f814b0-1291-4a12-ae65-dc0892203d36', 'volume_name': 'volume-68f814b0-1291-4a12-ae65-dc0892203d36', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:44:42 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (1ba33adb-82eb-4638-9fdf-8a225a8b9494) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:44:42 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (1ba33adb-82eb-4638-9fdf-8a225a8b9494) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:44:42 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (f33783fd-fc7d-43a3-9b4e-f4003948152d) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:44:42 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Volume 68f814b0-1291-4a12-ae65-dc0892203d36: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-68f814b0-1291-4a12-ae65-dc0892203d36', 'volume_size': 1} Apr 17 15:44:42 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] creating volume v_68f814b0-1291-4a12-ae65-dc0892203d36. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 15:44:42 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] create volume {'name': 'v_68f814b0-1291-4a12-ae65-dc0892203d36', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 15:44:42 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:44:42 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:44:45 user cinder-volume[84696]: DEBUG cinder.utils [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.783(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:44:45 user cinder-volume[84696]: DEBUG cinder.utils [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:44:45 user cinder-volume[84696]: DEBUG cinder.utils [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.886(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:44:45 user cinder-volume[84696]: DEBUG cinder.utils [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:44:45 user cinder-volume[84696]: DEBUG cinder.utils [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.100(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:44:45 user cinder-volume[84696]: DEBUG cinder.utils [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:44:48 user cinder-volume[84696]: DEBUG cinder.utils [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 9.811(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:44:48 user cinder-volume[84696]: DEBUG cinder.utils [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:44:48 user cinder-volume[84696]: DEBUG cinder.utils [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.856(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:44:48 user cinder-volume[84696]: DEBUG cinder.utils [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:44:48 user cinder-volume[84696]: DEBUG cinder.utils [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 9.373(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:44:48 user cinder-volume[84696]: DEBUG cinder.utils [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:44:48 user cinder-volume[84696]: DEBUG cinder.utils [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 9.600(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:44:48 user cinder-volume[84696]: DEBUG cinder.utils [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:44:51 user cinder-volume[84696]: DEBUG cinder.utils [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 75.236(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:44:51 user cinder-volume[84696]: DEBUG cinder.utils [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_441910f8-d29c-4c9e-bc5b-780ad228c85d/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:44:51 user cinder-volume[84696]: DEBUG cinder.utils [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.245(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:44:51 user cinder-volume[84696]: DEBUG cinder.utils [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:44:51 user cinder-volume[84696]: DEBUG cinder.utils [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.033(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:44:51 user cinder-volume[84696]: DEBUG cinder.utils [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:44:51 user cinder-volume[84696]: DEBUG cinder.utils [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 75.069(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:44:51 user cinder-volume[84696]: DEBUG cinder.utils [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:44:52 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=16,cinder:UPDATE=2 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:44:54 user cinder-volume[84696]: DEBUG cinder.utils [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 11.999(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:44:54 user cinder-volume[84696]: DEBUG cinder.utils [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:44:54 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 140.476(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:44:54 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:44:54 user cinder-volume[84696]: ERROR oslo_service.periodic_task [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Error during VolumeManager.publish_service_capabilities: TypeError: 'NoneType' object is not subscriptable Apr 17 15:44:54 user cinder-volume[84696]: ERROR oslo_service.periodic_task Traceback (most recent call last): Apr 17 15:44:54 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/usr/local/lib/python3.10/dist-packages/oslo_service/periodic_task.py", line 216, in run_periodic_tasks Apr 17 15:44:54 user cinder-volume[84696]: ERROR oslo_service.periodic_task task(self, context) Apr 17 15:44:54 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/manager.py", line 2876, in publish_service_capabilities Apr 17 15:44:54 user cinder-volume[84696]: ERROR oslo_service.periodic_task self._report_driver_status(context) Apr 17 15:44:54 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/manager.py", line 2755, in _report_driver_status Apr 17 15:44:54 user cinder-volume[84696]: ERROR oslo_service.periodic_task volume_stats = get_stats() Apr 17 15:44:54 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/usr/local/lib/python3.10/dist-packages/oslo_utils/timeutils.py", line 319, in wrapper Apr 17 15:44:54 user cinder-volume[84696]: ERROR oslo_service.periodic_task result = func(*args, **kwargs) Apr 17 15:44:54 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/manager.py", line 2753, in get_stats Apr 17 15:44:54 user cinder-volume[84696]: ERROR oslo_service.periodic_task return self.driver.get_volume_stats(refresh=True) Apr 17 15:44:54 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/driver.py", line 666, in get_volume_stats Apr 17 15:44:54 user cinder-volume[84696]: ERROR oslo_service.periodic_task self._update_volume_stats() Apr 17 15:44:54 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 726, in _update_volume_stats Apr 17 15:44:54 user cinder-volume[84696]: ERROR oslo_service.periodic_task pool_stats = self.ra.get_pool_stats() Apr 17 15:44:54 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 897, in get_pool_stats Apr 17 15:44:54 user cinder-volume[84696]: ERROR oslo_service.periodic_task if not resp["error"] and resp["code"] == 200: Apr 17 15:44:54 user cinder-volume[84696]: ERROR oslo_service.periodic_task TypeError: 'NoneType' object is not subscriptable Apr 17 15:44:54 user cinder-volume[84696]: ERROR oslo_service.periodic_task Apr 17 15:44:54 user cinder-volume[84696]: WARNING oslo.service.loopingcall [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Function 'cinder.service.Service.periodic_tasks' run outlasted interval by 80.48 sec Apr 17 15:44:54 user cinder-volume[84696]: DEBUG oslo_service.periodic_task [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Running periodic task VolumeManager.publish_service_capabilities {{(pid=84696) run_periodic_tasks /usr/local/lib/python3.10/dist-packages/oslo_service/periodic_task.py:210}} Apr 17 15:44:54 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Updating volume stats {{(pid=84696) _update_volume_stats /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:724}} Apr 17 15:44:54 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Get pool Pool-0 fsprops {{(pid=84696) get_pool_stats /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:894}} Apr 17 15:44:54 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:44:54 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:44:57 user cinder-volume[84696]: DEBUG cinder.utils [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 74.329(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:44:57 user cinder-volume[84696]: DEBUG cinder.utils [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:44:57 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 3.068(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:44:57 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:44:57 user cinder-volume[84696]: DEBUG cinder.utils [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 19.030(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:44:57 user cinder-volume[84696]: DEBUG cinder.utils [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:44:57 user cinder-volume[84696]: DEBUG cinder.utils [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 18.590(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:44:57 user cinder-volume[84696]: DEBUG cinder.utils [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:44:57 user cinder-volume[84696]: DEBUG cinder.utils [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 18.818(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:44:57 user cinder-volume[84696]: DEBUG cinder.utils [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:45:00 user cinder-volume[84696]: DEBUG cinder.utils [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 139.914(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:45:00 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:45:00 user cinder-volume[84696]: INFO cinder.message.api [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Creating message record for request_id = req-e0039512-b662-485b-95d9-48de5642a75d Apr 17 15:45:00 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.163(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:45:00 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:45:00 user cinder-volume[84696]: DEBUG cinder.utils [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 140.026(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:45:00 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:45:00 user cinder-volume[84696]: INFO cinder.message.api [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Creating message record for request_id = req-e55035f7-bed9-4dad-9cbc-b8b208b454bf Apr 17 15:45:00 user cinder-volume[84696]: DEBUG cinder.utils [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.485(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:45:00 user cinder-volume[84696]: DEBUG cinder.utils [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:45:00 user cinder-volume[84696]: DEBUG cinder.utils [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 140.273(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:45:00 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_57070f23-6b40-47b9-b9b4-e0d77240f544/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:45:00 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (f52a7c8f-8888-422c-a059-bcd722bb11a9) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 15:45:00 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (c2ded1ae-f9cb-46ae-8906-a1b14db7dbf4) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:45:00 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (c2ded1ae-f9cb-46ae-8906-a1b14db7dbf4) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:45:00 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Flow 'volume_create_manager' (9b1b0d98-6674-45cb-9465-f6cbc1d7939e) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 15:45:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 15:45:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 15:45:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 15:45:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 15:45:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 15:45:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 15:45:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 15:45:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 15:45:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 15:45:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 15:45:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 15:45:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 15:45:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 15:45:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 15:45:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 15:45:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 15:45:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 15:45:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 15:45:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 15:45:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 15:45:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 15:45:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 15:45:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 15:45:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 15:45:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:45:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 15:45:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 15:45:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 15:45:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 15:45:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 15:45:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 15:45:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 15:45:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 15:45:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 15:45:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 15:45:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 15:45:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 15:45:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 15:45:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 15:45:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 15:45:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 15:45:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 15:45:00 user cinder-volume[84696]: DEBUG cinder.utils [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.292(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:45:00 user cinder-volume[84696]: DEBUG cinder.utils [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:45:00 user cinder-volume[84696]: DEBUG cinder.utils [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 140.283(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:45:00 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_a50e9659-3e2d-4f04-b0ac-ae78b8f61fad/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:45:00 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (a0c6b43b-6bc3-4e94-973e-6e446ec021f2) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 15:45:00 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (a76e2bc9-d5d6-4c71-9ace-6a787a70e9e2) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:45:00 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (a76e2bc9-d5d6-4c71-9ace-6a787a70e9e2) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:45:00 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Flow 'volume_create_manager' (6ce3f926-71f3-4cc6-86ce-7b3b8d601821) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 15:45:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 15:45:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 15:45:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 15:45:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 15:45:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 15:45:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 15:45:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 15:45:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 15:45:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 15:45:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 15:45:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 15:45:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 15:45:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 15:45:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 15:45:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 15:45:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 15:45:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 15:45:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 15:45:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 15:45:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 15:45:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 15:45:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 15:45:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 15:45:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 15:45:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:45:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 15:45:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 15:45:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 15:45:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 15:45:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 15:45:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 15:45:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 15:45:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 15:45:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 15:45:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 15:45:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 15:45:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 15:45:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 15:45:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 15:45:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 15:45:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 15:45:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 15:45:00 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 15:45:00 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=24,cinder:UPDATE=6 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:45:00 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 15:45:00 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (ff49899e-7263-464c-9cc2-14c942422b2b) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 15:45:00 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 15:45:00 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:38:02Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-CreateVolumesFromSourceVolumeTest-Volume-609446595',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=a50e9659-3e2d-4f04-b0ac-ae78b8f61fad,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='99747be727ae4592ae1b6176b488719a',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:42:41Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:42:41Z,use_quota=True,user_id='0696808ea0b14c0794e8664a37d9e2c3',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 15:45:00 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:38:02Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-CreateVolumesFromSourceVolumeTest-Volume-609446595',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=a50e9659-3e2d-4f04-b0ac-ae78b8f61fad,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='99747be727ae4592ae1b6176b488719a',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:42:41Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:42:41Z,use_quota=True,user_id='0696808ea0b14c0794e8664a37d9e2c3',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(a50e9659-3e2d-4f04-b0ac-ae78b8f61fad),volume_id=a50e9659-3e2d-4f04-b0ac-ae78b8f61fad,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': 'a50e9659-3e2d-4f04-b0ac-ae78b8f61fad', 'volume_name': 'volume-a50e9659-3e2d-4f04-b0ac-ae78b8f61fad', 'volume_size': 1}} Apr 17 15:45:00 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': 'a50e9659-3e2d-4f04-b0ac-ae78b8f61fad', 'volume': {'id': 'a50e9659-3e2d-4f04-b0ac-ae78b8f61fad', '_name_id': None, 'ec2_id': None, 'user_id': '0696808ea0b14c0794e8664a37d9e2c3', 'project_id': '99747be727ae4592ae1b6176b488719a', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-CreateVolumesFromSourceVolumeTest-Volume-609446595', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'admin_metadata': {}, 'glance_metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T15:38:02.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-a50e9659-3e2d-4f04-b0ac-ae78b8f61fad', 'name_id': 'a50e9659-3e2d-4f04-b0ac-ae78b8f61fad', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-CreateVolumesFromSourceVolumeTest-Volume-609446595', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '99747be727ae4592ae1b6176b488719a', 'qos_specs': None, 'reservations': ['e9df84e9-0002-4c5e-9db7-eae2489ad0e9', 'd438171a-8bd2-47cf-81e3-e4af8b2d355c', 'cf24dd26-1a0b-48b2-95ca-4eceaffb6e37', 'e9528d9a-1681-4b42-a96b-ae681a21177b'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': '0696808ea0b14c0794e8664a37d9e2c3', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-CreateVolumesFromSourceVolumeTest-Volume-609446595', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '99747be727ae4592ae1b6176b488719a', 'qos_specs': None, 'reservations': ['e9df84e9-0002-4c5e-9db7-eae2489ad0e9', 'd438171a-8bd2-47cf-81e3-e4af8b2d355c', 'cf24dd26-1a0b-48b2-95ca-4eceaffb6e37', 'e9528d9a-1681-4b42-a96b-ae681a21177b'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': '0696808ea0b14c0794e8664a37d9e2c3', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': '0696808ea0b14c0794e8664a37d9e2c3', 'metadata': {}, 'qos_specs': None, 'retry': {'num_attempts': 3, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']}}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(a50e9659-3e2d-4f04-b0ac-ae78b8f61fad),volume_id=a50e9659-3e2d-4f04-b0ac-ae78b8f61fad,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:38:02Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-CreateVolumesFromSourceVolumeTest-Volume-609446595',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=a50e9659-3e2d-4f04-b0ac-ae78b8f61fad,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='99747be727ae4592ae1b6176b488719a',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:42:41Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:42:41Z,use_quota=True,user_id='0696808ea0b14c0794e8664a37d9e2c3',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 15:45:00 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:38:02Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-CreateVolumesFromSourceVolumeTest-Volume-609446595',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=a50e9659-3e2d-4f04-b0ac-ae78b8f61fad,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='99747be727ae4592ae1b6176b488719a',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:42:41Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:42:41Z,use_quota=True,user_id='0696808ea0b14c0794e8664a37d9e2c3',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:38:02Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-CreateVolumesFromSourceVolumeTest-Volume-609446595',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=a50e9659-3e2d-4f04-b0ac-ae78b8f61fad,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='99747be727ae4592ae1b6176b488719a',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:42:41Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:42:41Z,use_quota=True,user_id='0696808ea0b14c0794e8664a37d9e2c3',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 15:45:00 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 15:45:00 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 15:45:00 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:45:00 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 15:45:00 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 15:45:00 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 15:45:00 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 15:45:00 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 15:45:00 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 15:45:00 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 15:45:00 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 15:45:00 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 15:45:00 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 15:45:00 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 15:45:00 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 15:45:00 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 15:45:00 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 15:45:00 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 15:45:00 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 15:45:00 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 15:45:00 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (ff49899e-7263-464c-9cc2-14c942422b2b) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:45:00 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (ff49899e-7263-464c-9cc2-14c942422b2b) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:45:00 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (4e5a97b9-0fff-4ea1-a88c-d5bc693ef21f) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:45:00 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (4e5a97b9-0fff-4ea1-a88c-d5bc693ef21f) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:45:00 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (54657ef1-92e1-4332-aa7f-06f8cdbe1b9e) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:45:00 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (54657ef1-92e1-4332-aa7f-06f8cdbe1b9e) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:45:00 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (dfa120d7-a8b0-4768-af04-6169c215bc02) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:45:00 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Updating volume a50e9659-3e2d-4f04-b0ac-ae78b8f61fad with {'scheduled_at': datetime.datetime(2023, 4, 17, 15, 45, 0, 963415), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 15:45:00 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (f7df48e2-21f0-477f-b45b-ee9c5fd966e8) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 15:45:00 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 15:45:00 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:38:02Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesActionsTest-Volume-1957591186',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=57070f23-6b40-47b9-b9b4-e0d77240f544,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='a8b97ff6ed6047f0a33e74684a1c162b',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:42:41Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:42:41Z,use_quota=True,user_id='c993ceeb4a5c49eea14057e2125518ad',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 15:45:00 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:38:02Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesActionsTest-Volume-1957591186',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=57070f23-6b40-47b9-b9b4-e0d77240f544,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='a8b97ff6ed6047f0a33e74684a1c162b',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:42:41Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:42:41Z,use_quota=True,user_id='c993ceeb4a5c49eea14057e2125518ad',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(57070f23-6b40-47b9-b9b4-e0d77240f544),volume_id=57070f23-6b40-47b9-b9b4-e0d77240f544,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': '57070f23-6b40-47b9-b9b4-e0d77240f544', 'volume_name': 'volume-57070f23-6b40-47b9-b9b4-e0d77240f544', 'volume_size': 1}} Apr 17 15:45:00 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': '57070f23-6b40-47b9-b9b4-e0d77240f544', 'volume': {'id': '57070f23-6b40-47b9-b9b4-e0d77240f544', '_name_id': None, 'ec2_id': None, 'user_id': 'c993ceeb4a5c49eea14057e2125518ad', 'project_id': 'a8b97ff6ed6047f0a33e74684a1c162b', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-VolumesActionsTest-Volume-1957591186', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'admin_metadata': {}, 'glance_metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T15:38:02.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-57070f23-6b40-47b9-b9b4-e0d77240f544', 'name_id': '57070f23-6b40-47b9-b9b4-e0d77240f544', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumesActionsTest-Volume-1957591186', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': 'a8b97ff6ed6047f0a33e74684a1c162b', 'qos_specs': None, 'reservations': ['24ba47c4-9401-42c6-8d85-7f78767497d2', '1d6429be-3a45-4be3-a3f2-22a585792470', 'd0b8f941-f03c-4f32-b89a-dd9df5910de8', '4cdcf710-02d8-4b32-903f-875263491ca7'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'c993ceeb4a5c49eea14057e2125518ad', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumesActionsTest-Volume-1957591186', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': 'a8b97ff6ed6047f0a33e74684a1c162b', 'qos_specs': None, 'reservations': ['24ba47c4-9401-42c6-8d85-7f78767497d2', '1d6429be-3a45-4be3-a3f2-22a585792470', 'd0b8f941-f03c-4f32-b89a-dd9df5910de8', '4cdcf710-02d8-4b32-903f-875263491ca7'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'c993ceeb4a5c49eea14057e2125518ad', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': 'c993ceeb4a5c49eea14057e2125518ad', 'metadata': {}, 'qos_specs': None, 'retry': {'num_attempts': 3, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']}}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(57070f23-6b40-47b9-b9b4-e0d77240f544),volume_id=57070f23-6b40-47b9-b9b4-e0d77240f544,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:38:02Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesActionsTest-Volume-1957591186',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=57070f23-6b40-47b9-b9b4-e0d77240f544,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='a8b97ff6ed6047f0a33e74684a1c162b',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:42:41Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:42:41Z,use_quota=True,user_id='c993ceeb4a5c49eea14057e2125518ad',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 15:45:00 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:38:02Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesActionsTest-Volume-1957591186',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=57070f23-6b40-47b9-b9b4-e0d77240f544,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='a8b97ff6ed6047f0a33e74684a1c162b',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:42:41Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:42:41Z,use_quota=True,user_id='c993ceeb4a5c49eea14057e2125518ad',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:38:02Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesActionsTest-Volume-1957591186',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=57070f23-6b40-47b9-b9b4-e0d77240f544,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='a8b97ff6ed6047f0a33e74684a1c162b',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:42:41Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:42:41Z,use_quota=True,user_id='c993ceeb4a5c49eea14057e2125518ad',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 15:45:00 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 15:45:00 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 15:45:00 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:45:00 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 15:45:00 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 15:45:00 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 15:45:00 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 15:45:00 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 15:45:00 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 15:45:00 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 15:45:00 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 15:45:00 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 15:45:00 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 15:45:00 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 15:45:00 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 15:45:00 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 15:45:00 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 15:45:00 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 15:45:00 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 15:45:00 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 15:45:00 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (f7df48e2-21f0-477f-b45b-ee9c5fd966e8) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:45:00 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (f7df48e2-21f0-477f-b45b-ee9c5fd966e8) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:45:00 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (e15537ed-0296-43d3-872c-15aa19320938) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:45:00 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (e15537ed-0296-43d3-872c-15aa19320938) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:45:01 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (a731bf96-2be9-43af-bf17-3e401bcf5093) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:45:01 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (a731bf96-2be9-43af-bf17-3e401bcf5093) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:45:01 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (64ee410e-815a-4b3a-b712-8b21a6c707d8) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:45:01 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Updating volume 57070f23-6b40-47b9-b9b4-e0d77240f544 with {'scheduled_at': datetime.datetime(2023, 4, 17, 15, 45, 1, 8575), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 15:45:01 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Volume 57070f23-6b40-47b9-b9b4-e0d77240f544: re-scheduling SchedulerAPI.create_volume attempt 3 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 15:45:01 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Volume a50e9659-3e2d-4f04-b0ac-ae78b8f61fad: re-scheduling SchedulerAPI.create_volume attempt 3 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 15:45:01 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Volume 57070f23-6b40-47b9-b9b4-e0d77240f544: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 15:45:01 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] deleating volume v_57070f23-6b40-47b9-b9b4-e0d77240f544 {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 15:45:01 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] get snapshots for volume v_57070f23-6b40-47b9-b9b4-e0d77240f544 {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 15:45:01 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_57070f23-6b40-47b9-b9b4-e0d77240f544/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:45:01 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_57070f23-6b40-47b9-b9b4-e0d77240f544/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:45:01 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Volume a50e9659-3e2d-4f04-b0ac-ae78b8f61fad: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 15:45:01 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] deleating volume v_a50e9659-3e2d-4f04-b0ac-ae78b8f61fad {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 15:45:01 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] get snapshots for volume v_a50e9659-3e2d-4f04-b0ac-ae78b8f61fad {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 15:45:01 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_a50e9659-3e2d-4f04-b0ac-ae78b8f61fad/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:45:01 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_a50e9659-3e2d-4f04-b0ac-ae78b8f61fad/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:45:03 user cinder-volume[84696]: DEBUG cinder.utils [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.215(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:45:03 user cinder-volume[84696]: DEBUG cinder.utils [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:45:03 user cinder-volume[84696]: DEBUG cinder.utils [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.853(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:45:03 user cinder-volume[84696]: DEBUG cinder.utils [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_a50e9659-3e2d-4f04-b0ac-ae78b8f61fad/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:45:03 user cinder-volume[84696]: DEBUG cinder.utils [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.858(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:45:03 user cinder-volume[84696]: DEBUG cinder.utils [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_57070f23-6b40-47b9-b9b4-e0d77240f544/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:45:06 user cinder-volume[84696]: DEBUG cinder.utils [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.923(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:45:06 user cinder-volume[84696]: DEBUG cinder.utils [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_a50e9659-3e2d-4f04-b0ac-ae78b8f61fad/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:45:06 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.285(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:45:06 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:45:06 user cinder-volume[84696]: DEBUG cinder.utils [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.930(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:45:06 user cinder-volume[84696]: DEBUG cinder.utils [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_57070f23-6b40-47b9-b9b4-e0d77240f544/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:45:13 user cinder-volume[84696]: DEBUG cinder.utils [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.067(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:45:13 user cinder-volume[84696]: DEBUG cinder.utils [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_a50e9659-3e2d-4f04-b0ac-ae78b8f61fad/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:45:13 user cinder-volume[84696]: DEBUG cinder.utils [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.072(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:45:13 user cinder-volume[84696]: DEBUG cinder.utils [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_57070f23-6b40-47b9-b9b4-e0d77240f544/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:45:13 user cinder-volume[84696]: DEBUG cinder.utils [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 140.183(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:45:13 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:45:13 user cinder-volume[84696]: INFO cinder.message.api [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Creating message record for request_id = req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b Apr 17 15:45:13 user cinder-volume[84696]: DEBUG cinder.utils [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 70.157(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:45:13 user cinder-volume[84696]: DEBUG cinder.utils [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:45:13 user cinder-volume[84696]: DEBUG cinder.utils [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 140.366(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:45:13 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_7ce815ce-9603-4fd6-8cc8-9fd798b2f99d/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:45:13 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (018eee5d-5087-4932-97d7-887e2a16ecd9) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 15:45:13 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (5a9b725b-64e1-4f2b-9db9-01fb473d3018) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:45:13 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (5a9b725b-64e1-4f2b-9db9-01fb473d3018) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:45:13 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Flow 'volume_create_manager' (0a67e980-27da-43b1-b2f3-13d324bdb9a5) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 15:45:13 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 15:45:13 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 15:45:13 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 15:45:13 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 15:45:13 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 15:45:13 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 15:45:13 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 15:45:13 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 15:45:13 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 15:45:13 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 15:45:13 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 15:45:13 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 15:45:13 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 15:45:13 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 15:45:13 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 15:45:13 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 15:45:13 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 15:45:13 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 15:45:13 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 15:45:13 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 15:45:13 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 15:45:13 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 15:45:13 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 15:45:13 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 15:45:13 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:45:13 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 15:45:13 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 15:45:13 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 15:45:13 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 15:45:13 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 15:45:13 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 15:45:13 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 15:45:13 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 15:45:13 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 15:45:13 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 15:45:13 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 15:45:13 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 15:45:13 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 15:45:13 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 15:45:13 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 15:45:13 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 15:45:13 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 15:45:13 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 15:45:13 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (bbd68265-417d-4306-9a9a-8a84172aa22c) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 15:45:13 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 15:45:13 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:38:17Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesSnapshotTestJSON-Volume-1408369126',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=7ce815ce-9603-4fd6-8cc8-9fd798b2f99d,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='aacddeb0a9ec4a0d8d832595382e1189',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:42:53Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:42:53Z,use_quota=True,user_id='fddc0a19102c4ee7920f657f195171c9',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 15:45:13 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:38:17Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesSnapshotTestJSON-Volume-1408369126',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=7ce815ce-9603-4fd6-8cc8-9fd798b2f99d,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='aacddeb0a9ec4a0d8d832595382e1189',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:42:53Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:42:53Z,use_quota=True,user_id='fddc0a19102c4ee7920f657f195171c9',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(7ce815ce-9603-4fd6-8cc8-9fd798b2f99d),volume_id=7ce815ce-9603-4fd6-8cc8-9fd798b2f99d,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': '7ce815ce-9603-4fd6-8cc8-9fd798b2f99d', 'volume_name': 'volume-7ce815ce-9603-4fd6-8cc8-9fd798b2f99d', 'volume_size': 1}} Apr 17 15:45:13 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': '7ce815ce-9603-4fd6-8cc8-9fd798b2f99d', 'volume': {'id': '7ce815ce-9603-4fd6-8cc8-9fd798b2f99d', '_name_id': None, 'ec2_id': None, 'user_id': 'fddc0a19102c4ee7920f657f195171c9', 'project_id': 'aacddeb0a9ec4a0d8d832595382e1189', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-VolumesSnapshotTestJSON-Volume-1408369126', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'admin_metadata': {}, 'glance_metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T15:38:16.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-7ce815ce-9603-4fd6-8cc8-9fd798b2f99d', 'name_id': '7ce815ce-9603-4fd6-8cc8-9fd798b2f99d', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumesSnapshotTestJSON-Volume-1408369126', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': 'aacddeb0a9ec4a0d8d832595382e1189', 'qos_specs': None, 'reservations': ['f313e2f7-2050-44c4-b79e-0ecee302728c', '287d06a9-6c9c-4bb1-ba13-85b7cf2d8b7e', '14fdea43-dc14-4800-b0fe-8defcfbc5c61', '5659b6c2-9b08-423d-bdcb-0a1bab01869d'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'fddc0a19102c4ee7920f657f195171c9', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumesSnapshotTestJSON-Volume-1408369126', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': 'aacddeb0a9ec4a0d8d832595382e1189', 'qos_specs': None, 'reservations': ['f313e2f7-2050-44c4-b79e-0ecee302728c', '287d06a9-6c9c-4bb1-ba13-85b7cf2d8b7e', '14fdea43-dc14-4800-b0fe-8defcfbc5c61', '5659b6c2-9b08-423d-bdcb-0a1bab01869d'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'fddc0a19102c4ee7920f657f195171c9', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': 'fddc0a19102c4ee7920f657f195171c9', 'metadata': {}, 'qos_specs': None, 'retry': {'num_attempts': 3, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']}}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(7ce815ce-9603-4fd6-8cc8-9fd798b2f99d),volume_id=7ce815ce-9603-4fd6-8cc8-9fd798b2f99d,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:38:17Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesSnapshotTestJSON-Volume-1408369126',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=7ce815ce-9603-4fd6-8cc8-9fd798b2f99d,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='aacddeb0a9ec4a0d8d832595382e1189',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:42:53Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:42:53Z,use_quota=True,user_id='fddc0a19102c4ee7920f657f195171c9',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 15:45:13 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:38:17Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesSnapshotTestJSON-Volume-1408369126',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=7ce815ce-9603-4fd6-8cc8-9fd798b2f99d,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='aacddeb0a9ec4a0d8d832595382e1189',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:42:53Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:42:53Z,use_quota=True,user_id='fddc0a19102c4ee7920f657f195171c9',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:38:17Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesSnapshotTestJSON-Volume-1408369126',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=7ce815ce-9603-4fd6-8cc8-9fd798b2f99d,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='aacddeb0a9ec4a0d8d832595382e1189',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:42:53Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:42:53Z,use_quota=True,user_id='fddc0a19102c4ee7920f657f195171c9',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 15:45:13 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 15:45:13 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 15:45:13 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:45:13 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 15:45:13 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 15:45:13 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 15:45:13 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 15:45:13 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 15:45:13 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 15:45:13 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 15:45:13 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 15:45:13 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 15:45:13 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 15:45:13 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 15:45:13 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 15:45:13 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 15:45:13 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 15:45:13 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 15:45:13 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 15:45:13 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 15:45:13 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (bbd68265-417d-4306-9a9a-8a84172aa22c) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:45:13 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (bbd68265-417d-4306-9a9a-8a84172aa22c) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:45:13 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (2acfee05-b7aa-4c77-85f8-1e7fabed8148) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:45:13 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (2acfee05-b7aa-4c77-85f8-1e7fabed8148) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:45:13 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (04cc913a-66b1-4868-aafb-08b4610c7827) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:45:13 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (04cc913a-66b1-4868-aafb-08b4610c7827) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:45:13 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (a6c87ba5-c4ef-454e-93ad-6732d94f5e50) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:45:13 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Updating volume 7ce815ce-9603-4fd6-8cc8-9fd798b2f99d with {'scheduled_at': datetime.datetime(2023, 4, 17, 15, 45, 13, 193205), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 15:45:13 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Volume 7ce815ce-9603-4fd6-8cc8-9fd798b2f99d: re-scheduling SchedulerAPI.create_volume attempt 3 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 15:45:13 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Volume 7ce815ce-9603-4fd6-8cc8-9fd798b2f99d: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 15:45:13 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] deleating volume v_7ce815ce-9603-4fd6-8cc8-9fd798b2f99d {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 15:45:13 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] get snapshots for volume v_7ce815ce-9603-4fd6-8cc8-9fd798b2f99d {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 15:45:13 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_7ce815ce-9603-4fd6-8cc8-9fd798b2f99d/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:45:13 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_7ce815ce-9603-4fd6-8cc8-9fd798b2f99d/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:45:16 user cinder-volume[84696]: DEBUG cinder.utils [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.971(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:45:16 user cinder-volume[84696]: DEBUG cinder.utils [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_7ce815ce-9603-4fd6-8cc8-9fd798b2f99d/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:45:16 user cinder-volume[84696]: DEBUG cinder.utils [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 37.020(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:45:16 user cinder-volume[84696]: DEBUG cinder.utils [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:45:16 user cinder-volume[84696]: DEBUG cinder.utils [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 37.247(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:45:16 user cinder-volume[84696]: DEBUG cinder.utils [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:45:16 user cinder-volume[84696]: DEBUG cinder.utils [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 37.462(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:45:16 user cinder-volume[84696]: DEBUG cinder.utils [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:45:16 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.503(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:45:16 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:45:19 user cinder-volume[84696]: DEBUG cinder.utils [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.889(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:45:19 user cinder-volume[84696]: DEBUG cinder.utils [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:45:19 user cinder-volume[84696]: DEBUG cinder.utils [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.044(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:45:19 user cinder-volume[84696]: DEBUG cinder.utils [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_7ce815ce-9603-4fd6-8cc8-9fd798b2f99d/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:45:19 user cinder-volume[84696]: DEBUG cinder.utils [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.678(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:45:19 user cinder-volume[84696]: DEBUG cinder.utils [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:45:22 user cinder-volume[84696]: DEBUG cinder.utils [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.647(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:45:22 user cinder-volume[84696]: DEBUG cinder.utils [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:45:22 user cinder-volume[84696]: DEBUG cinder.utils [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.284(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:45:22 user cinder-volume[84696]: DEBUG cinder.utils [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_a50e9659-3e2d-4f04-b0ac-ae78b8f61fad/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:45:22 user cinder-volume[84696]: DEBUG cinder.utils [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.289(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:45:22 user cinder-volume[84696]: DEBUG cinder.utils [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_57070f23-6b40-47b9-b9b4-e0d77240f544/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:45:24 user cinder-volume[84696]: DEBUG cinder.coordination [req-6ee9fe55-7b3a-42cf-906e-c25ad8813b64 req-9a73de69-ad8a-45b1-8113-a76d843c2607 tempest-TestShelveInstance-1683612675 None] Acquiring lock "cinder-d895d11d-6880-4418-a9ad-feca813ed5d6-delete_volume" by "delete_volume" {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:192}} Apr 17 15:45:24 user cinder-volume[84696]: DEBUG cinder.coordination [req-6ee9fe55-7b3a-42cf-906e-c25ad8813b64 req-9a73de69-ad8a-45b1-8113-a76d843c2607 tempest-TestShelveInstance-1683612675 None] Lock "cinder-d895d11d-6880-4418-a9ad-feca813ed5d6-delete_volume" acquired by "delete_volume" :: waited 0.011s {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:196}} Apr 17 15:45:24 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=11,cinder:UPDATE=4,cinder:INSERT=4 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:45:24 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-6ee9fe55-7b3a-42cf-906e-c25ad8813b64 req-9a73de69-ad8a-45b1-8113-a76d843c2607 tempest-TestShelveInstance-1683612675 None] remove_export for volume: d895d11d-6880-4418-a9ad-feca813ed5d6. {{(pid=84696) remove_export /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:718}} Apr 17 15:45:24 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-6ee9fe55-7b3a-42cf-906e-c25ad8813b64 req-9a73de69-ad8a-45b1-8113-a76d843c2607 tempest-TestShelveInstance-1683612675 None] remove export {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:912}} Apr 17 15:45:24 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-6ee9fe55-7b3a-42cf-906e-c25ad8813b64 req-9a73de69-ad8a-45b1-8113-a76d843c2607 tempest-TestShelveInstance-1683612675 None] detach volume:Volume(_name_id=None,admin_metadata={},attach_status='detached',availability_zone='nova',bootable=True,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:25:25Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-tempest.common.compute-volume-1444682483',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=d895d11d-6880-4418-a9ad-feca813ed5d6,launched_at=2023-04-17T15:25:29Z,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='a2df343eb0d64207a5f9f91aa0810999',provider_auth='CHAP FVRqMWdJ 6d3UKemu7AN8VG',provider_geometry=None,provider_id=None,provider_location='172.16.0.220:3260,1 iqn.2010-10.org.openstack:d895d11d-6880-4418-a9ad-feca813ed5d6 0',replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:25:25Z,service_uuid='e628352e-c4cf-40eb-8389-cce40663026a',shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='deleting',terminated_at=2023-04-17T15:45:24Z,updated_at=2023-04-17T15:45:24Z,use_quota=True,user_id='909c09f2587f42f895352916d3f8636a',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c) from target:iqn.2010-10.org.openstack:d895d11d-6880-4418-a9ad-feca813ed5d6. {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:913}} Apr 17 15:45:24 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [req-6ee9fe55-7b3a-42cf-906e-c25ad8813b64 req-9a73de69-ad8a-45b1-8113-a76d843c2607 tempest-TestShelveInstance-1683612675 None] detach volume v_d895d11d-6880-4418-a9ad-feca813ed5d6 from target iqn.2010-10.org.openstack:d895d11d-6880-4418-a9ad-feca813ed5d6 {{(pid=84696) detach_target_vol /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:606}} Apr 17 15:45:24 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-6ee9fe55-7b3a-42cf-906e-c25ad8813b64 req-9a73de69-ad8a-45b1-8113-a76d843c2607 tempest-TestShelveInstance-1683612675 None] Sending pool request DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:d895d11d-6880-4418-a9ad-feca813ed5d6/luns/v_d895d11d-6880-4418-a9ad-feca813ed5d6 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:45:24 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-6ee9fe55-7b3a-42cf-906e-c25ad8813b64 req-9a73de69-ad8a-45b1-8113-a76d843c2607 tempest-TestShelveInstance-1683612675 None] Sending DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:d895d11d-6880-4418-a9ad-feca813ed5d6/luns/v_d895d11d-6880-4418-a9ad-feca813ed5d6 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:45:25 user cinder-volume[84696]: DEBUG cinder.utils [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.187(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:45:25 user cinder-volume[84696]: DEBUG cinder.utils [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_7ce815ce-9603-4fd6-8cc8-9fd798b2f99d/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:45:25 user cinder-volume[84696]: DEBUG cinder.utils [req-6ee9fe55-7b3a-42cf-906e-c25ad8813b64 req-9a73de69-ad8a-45b1-8113-a76d843c2607 tempest-TestShelveInstance-1683612675 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 1.348(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:45:25 user cinder-volume[84696]: DEBUG cinder.utils [req-6ee9fe55-7b3a-42cf-906e-c25ad8813b64 req-9a73de69-ad8a-45b1-8113-a76d843c2607 tempest-TestShelveInstance-1683612675 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:d895d11d-6880-4418-a9ad-feca813ed5d6/luns/v_d895d11d-6880-4418-a9ad-feca813ed5d6 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:45:28 user cinder-volume[84696]: DEBUG cinder.utils [req-6ee9fe55-7b3a-42cf-906e-c25ad8813b64 req-9a73de69-ad8a-45b1-8113-a76d843c2607 tempest-TestShelveInstance-1683612675 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 4.423(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:45:28 user cinder-volume[84696]: DEBUG cinder.utils [req-6ee9fe55-7b3a-42cf-906e-c25ad8813b64 req-9a73de69-ad8a-45b1-8113-a76d843c2607 tempest-TestShelveInstance-1683612675 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:d895d11d-6880-4418-a9ad-feca813ed5d6/luns/v_d895d11d-6880-4418-a9ad-feca813ed5d6 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:45:34 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:UPDATE=6,cinder:SELECT=14,cinder:INSERT=2 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:45:35 user cinder-volume[84696]: DEBUG cinder.utils [req-6ee9fe55-7b3a-42cf-906e-c25ad8813b64 req-9a73de69-ad8a-45b1-8113-a76d843c2607 tempest-TestShelveInstance-1683612675 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 11.043(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:45:35 user cinder-volume[84696]: DEBUG cinder.utils [req-6ee9fe55-7b3a-42cf-906e-c25ad8813b64 req-9a73de69-ad8a-45b1-8113-a76d843c2607 tempest-TestShelveInstance-1683612675 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:d895d11d-6880-4418-a9ad-feca813ed5d6/luns/v_d895d11d-6880-4418-a9ad-feca813ed5d6 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:45:35 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 40.411(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:45:35 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:45:35 user cinder-volume[84696]: DEBUG cinder.utils [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.885(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:45:35 user cinder-volume[84696]: DEBUG cinder.utils [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_7ce815ce-9603-4fd6-8cc8-9fd798b2f99d/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:45:41 user cinder-volume[84696]: DEBUG cinder.utils [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 40.362(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:45:41 user cinder-volume[84696]: DEBUG cinder.utils [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_57070f23-6b40-47b9-b9b4-e0d77240f544/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:45:41 user cinder-volume[84696]: DEBUG cinder.utils [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 40.360(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:45:41 user cinder-volume[84696]: DEBUG cinder.utils [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_a50e9659-3e2d-4f04-b0ac-ae78b8f61fad/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:45:44 user cinder-volume[84696]: DEBUG cinder.utils [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 140.134(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:45:44 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:45:44 user cinder-volume[84696]: INFO cinder.message.api [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Creating message record for request_id = req-52df28ef-59c1-47f6-831b-cf97d2a31df8 Apr 17 15:45:44 user cinder-volume[84696]: DEBUG cinder.utils [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 140.358(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:45:44 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_a27d0ff3-5de9-4feb-82f7-d4571af88bb9/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:45:44 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (88bc4781-d148-4199-87b5-ca7f3c1d508f) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 15:45:44 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (50a78af3-ba39-4893-a01a-240aa1cf974f) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:45:44 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (50a78af3-ba39-4893-a01a-240aa1cf974f) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:45:44 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Flow 'volume_create_manager' (a9a0f28b-5a79-46cf-a810-f69068e92f0a) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 15:45:44 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 15:45:44 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 15:45:44 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 15:45:44 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 15:45:44 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 15:45:44 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 15:45:44 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 15:45:44 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 15:45:44 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 15:45:44 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 15:45:44 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 15:45:44 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 15:45:44 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 15:45:44 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 15:45:44 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 15:45:44 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 15:45:44 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 15:45:44 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 15:45:44 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 15:45:44 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 15:45:44 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 15:45:44 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 15:45:44 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 15:45:44 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 15:45:44 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:45:44 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 15:45:44 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 15:45:44 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 15:45:44 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 15:45:44 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 15:45:44 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 15:45:44 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 15:45:44 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 15:45:44 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 15:45:44 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 15:45:44 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 15:45:44 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 15:45:44 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 15:45:44 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 15:45:44 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 15:45:44 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 15:45:44 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 15:45:44 user cinder-volume[84696]: DEBUG cinder.utils [req-6ee9fe55-7b3a-42cf-906e-c25ad8813b64 req-9a73de69-ad8a-45b1-8113-a76d843c2607 tempest-TestShelveInstance-1683612675 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 20.455(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:45:44 user cinder-volume[84696]: DEBUG cinder.utils [req-6ee9fe55-7b3a-42cf-906e-c25ad8813b64 req-9a73de69-ad8a-45b1-8113-a76d843c2607 tempest-TestShelveInstance-1683612675 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:d895d11d-6880-4418-a9ad-feca813ed5d6/luns/v_d895d11d-6880-4418-a9ad-feca813ed5d6 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:45:44 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 15:45:44 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (0113ba91-d3db-43cb-bfe3-b28bfa3a1afc) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 15:45:44 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 15:45:44 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:38:47Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-ServersNegativeTestJSON-volume-2082564469',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=a27d0ff3-5de9-4feb-82f7-d4571af88bb9,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='d2dd48f279404859bebcc19bcc272915',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:43:24Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:43:24Z,use_quota=True,user_id='b0075b26d6044a50981d29df0d863959',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 15:45:44 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:38:47Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-ServersNegativeTestJSON-volume-2082564469',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=a27d0ff3-5de9-4feb-82f7-d4571af88bb9,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='d2dd48f279404859bebcc19bcc272915',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:43:24Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:43:24Z,use_quota=True,user_id='b0075b26d6044a50981d29df0d863959',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(a27d0ff3-5de9-4feb-82f7-d4571af88bb9),volume_id=a27d0ff3-5de9-4feb-82f7-d4571af88bb9,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': 'a27d0ff3-5de9-4feb-82f7-d4571af88bb9', 'volume_name': 'volume-a27d0ff3-5de9-4feb-82f7-d4571af88bb9', 'volume_size': 1}} Apr 17 15:45:44 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': 'a27d0ff3-5de9-4feb-82f7-d4571af88bb9', 'volume': {'id': 'a27d0ff3-5de9-4feb-82f7-d4571af88bb9', '_name_id': None, 'ec2_id': None, 'user_id': 'b0075b26d6044a50981d29df0d863959', 'project_id': 'd2dd48f279404859bebcc19bcc272915', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-ServersNegativeTestJSON-volume-2082564469', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'admin_metadata': {}, 'glance_metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T15:38:47.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-a27d0ff3-5de9-4feb-82f7-d4571af88bb9', 'name_id': 'a27d0ff3-5de9-4feb-82f7-d4571af88bb9', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-ServersNegativeTestJSON-volume-2082564469', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': 'd2dd48f279404859bebcc19bcc272915', 'qos_specs': None, 'reservations': ['5c815076-6ae9-452c-aa1e-9825025560c9', 'b96b1904-2fdf-4b5b-9c8d-61562d250d61', '569a260d-012a-4ff0-b9af-61caad629af7', 'c933f679-b335-4a11-b551-a5c307704f2e'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'b0075b26d6044a50981d29df0d863959', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-ServersNegativeTestJSON-volume-2082564469', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': 'd2dd48f279404859bebcc19bcc272915', 'qos_specs': None, 'reservations': ['5c815076-6ae9-452c-aa1e-9825025560c9', 'b96b1904-2fdf-4b5b-9c8d-61562d250d61', '569a260d-012a-4ff0-b9af-61caad629af7', 'c933f679-b335-4a11-b551-a5c307704f2e'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'b0075b26d6044a50981d29df0d863959', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': 'b0075b26d6044a50981d29df0d863959', 'metadata': {}, 'qos_specs': None, 'retry': {'num_attempts': 3, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']}}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(a27d0ff3-5de9-4feb-82f7-d4571af88bb9),volume_id=a27d0ff3-5de9-4feb-82f7-d4571af88bb9,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:38:47Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-ServersNegativeTestJSON-volume-2082564469',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=a27d0ff3-5de9-4feb-82f7-d4571af88bb9,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='d2dd48f279404859bebcc19bcc272915',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:43:24Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:43:24Z,use_quota=True,user_id='b0075b26d6044a50981d29df0d863959',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 15:45:44 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:38:47Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-ServersNegativeTestJSON-volume-2082564469',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=a27d0ff3-5de9-4feb-82f7-d4571af88bb9,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='d2dd48f279404859bebcc19bcc272915',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:43:24Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:43:24Z,use_quota=True,user_id='b0075b26d6044a50981d29df0d863959',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:38:47Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-ServersNegativeTestJSON-volume-2082564469',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=a27d0ff3-5de9-4feb-82f7-d4571af88bb9,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='d2dd48f279404859bebcc19bcc272915',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:43:24Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:43:24Z,use_quota=True,user_id='b0075b26d6044a50981d29df0d863959',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 15:45:44 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 15:45:44 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 15:45:44 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:45:44 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 15:45:44 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 15:45:44 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 15:45:44 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 15:45:44 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 15:45:44 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 15:45:44 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 15:45:44 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 15:45:44 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 15:45:44 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 15:45:44 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 15:45:44 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 15:45:44 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 15:45:44 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 15:45:44 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 15:45:44 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 15:45:44 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 15:45:44 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (0113ba91-d3db-43cb-bfe3-b28bfa3a1afc) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:45:44 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (0113ba91-d3db-43cb-bfe3-b28bfa3a1afc) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:45:44 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (bc855868-3d54-4c61-a937-dff15179b2ef) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:45:44 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (bc855868-3d54-4c61-a937-dff15179b2ef) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:45:44 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (dd3d5025-a3fb-4838-be81-6c29d4ff7554) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:45:44 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (dd3d5025-a3fb-4838-be81-6c29d4ff7554) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:45:44 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (00be8551-29ba-4eb4-bc63-b825cf11ce67) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:45:44 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Updating volume a27d0ff3-5de9-4feb-82f7-d4571af88bb9 with {'scheduled_at': datetime.datetime(2023, 4, 17, 15, 45, 44, 580226), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 15:45:44 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Volume a27d0ff3-5de9-4feb-82f7-d4571af88bb9: re-scheduling SchedulerAPI.create_volume attempt 3 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 15:45:44 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Volume a27d0ff3-5de9-4feb-82f7-d4571af88bb9: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 15:45:44 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] deleating volume v_a27d0ff3-5de9-4feb-82f7-d4571af88bb9 {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 15:45:44 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] get snapshots for volume v_a27d0ff3-5de9-4feb-82f7-d4571af88bb9 {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 15:45:44 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_a27d0ff3-5de9-4feb-82f7-d4571af88bb9/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:45:44 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_a27d0ff3-5de9-4feb-82f7-d4571af88bb9/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:45:47 user cinder-volume[84696]: DEBUG cinder.utils [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.928(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:45:47 user cinder-volume[84696]: DEBUG cinder.utils [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_a27d0ff3-5de9-4feb-82f7-d4571af88bb9/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:45:47 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Volume reschedule parameters: True retry: {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 15:45:47 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Flow 'volume_create_manager' (09111813-7745-4322-aaca-b836df46177b) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:45:47 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (53e578f2-e99b-4739-a718-da9597addaf0) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:45:47 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (53e578f2-e99b-4739-a718-da9597addaf0) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:45:47Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TestVolumeBootPattern-volume-origin-14805124',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=93a32a7c-4c6a-472a-8f91-9d34558ba232,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='c8a836b47d954cb9a7c1cd7c0f790426',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:45:48Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:45:48Z,use_quota=True,user_id='a80e753322224ec1a897598b5b9ed6ea',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:45:47 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (fd6372ff-955f-45b3-a7c4-551f96bdd404) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:45:47 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (fd6372ff-955f-45b3-a7c4-551f96bdd404) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:45:47 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (a85a3b7a-915c-43f4-8b75-f1fb81ad62ee) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:45:47 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (a85a3b7a-915c-43f4-8b75-f1fb81ad62ee) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'image', 'volume_id': '93a32a7c-4c6a-472a-8f91-9d34558ba232', 'volume_name': 'volume-93a32a7c-4c6a-472a-8f91-9d34558ba232', 'volume_size': 1, 'image_id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'image_location': (None, None), 'image_meta': {'name': 'cirros-0.5.2-x86_64-disk', 'disk_format': 'qcow2', 'container_format': 'bare', 'visibility': 'public', 'size': 16300544, 'virtual_size': 117440512, 'status': 'active', 'checksum': 'b874c39491a2377b8490f5f1e89761a4', 'protected': False, 'min_ram': 0, 'min_disk': 0, 'owner': 'a43c211b4ed542538799349489b1d884', 'os_hidden': False, 'os_hash_algo': 'sha512', 'os_hash_value': '6b813aa46bb90b4da216a4d19376593fa3f4fc7e617f03a92b7fe11e9a3981cbe8f0959dbebe36225e5f53dc4492341a4863cac4ed1ee0909f3fc78ef9c3e869', 'id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'created_at': datetime.datetime(2023, 4, 17, 15, 19, 53, tzinfo=datetime.timezone.utc), 'updated_at': datetime.datetime(2023, 4, 17, 15, 19, 55, tzinfo=datetime.timezone.utc), 'tags': [], 'file': '/v2/images/b3f0dec1-7758-44a8-82e0-76ada8837428/file', 'properties': {'hw_rng_model': 'virtio', 'owner_specified.openstack.md5': '', 'owner_specified.openstack.object': 'images/cirros-0.5.2-x86_64-disk', 'owner_specified.openstack.sha256': ''}}, 'image_service': }' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:45:47 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (a07076e3-7672-44e5-9253-90d6448b71bc) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:45:47 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (a07076e3-7672-44e5-9253-90d6448b71bc) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:45:47 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (f5db5269-5ad7-4f1b-be93-2e00babdad94) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:45:47 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Volume 93a32a7c-4c6a-472a-8f91-9d34558ba232: being created as image with specification: {'status': 'creating', 'volume_name': 'volume-93a32a7c-4c6a-472a-8f91-9d34558ba232', 'volume_size': 1, 'image_id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'image_location': (None, None), 'image_meta': {'name': 'cirros-0.5.2-x86_64-disk', 'disk_format': 'qcow2', 'container_format': 'bare', 'visibility': 'public', 'size': 16300544, 'virtual_size': 117440512, 'status': 'active', 'checksum': 'b874c39491a2377b8490f5f1e89761a4', 'protected': False, 'min_ram': 0, 'min_disk': 0, 'owner': 'a43c211b4ed542538799349489b1d884', 'os_hidden': False, 'os_hash_algo': 'sha512', 'os_hash_value': '6b813aa46bb90b4da216a4d19376593fa3f4fc7e617f03a92b7fe11e9a3981cbe8f0959dbebe36225e5f53dc4492341a4863cac4ed1ee0909f3fc78ef9c3e869', 'id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'created_at': datetime.datetime(2023, 4, 17, 15, 19, 53, tzinfo=datetime.timezone.utc), 'updated_at': datetime.datetime(2023, 4, 17, 15, 19, 55, tzinfo=datetime.timezone.utc), 'tags': [], 'file': '/v2/images/b3f0dec1-7758-44a8-82e0-76ada8837428/file', 'properties': {'hw_rng_model': 'virtio', 'owner_specified.openstack.md5': '', 'owner_specified.openstack.object': 'images/cirros-0.5.2-x86_64-disk', 'owner_specified.openstack.sha256': ''}}, 'image_service': } Apr 17 15:45:47 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Cloning 93a32a7c-4c6a-472a-8f91-9d34558ba232 from image b3f0dec1-7758-44a8-82e0-76ada8837428 at location (None, None). {{(pid=84696) _create_from_image /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1071}} Apr 17 15:45:47 user cinder-volume[84696]: DEBUG cinder.coordination [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Acquiring lock "cinder-b3f0dec1-7758-44a8-82e0-76ada8837428" by "_prepare_image_cache_entry" {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:192}} Apr 17 15:45:47 user cinder-volume[84696]: DEBUG cinder.coordination [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Lock "cinder-b3f0dec1-7758-44a8-82e0-76ada8837428" acquired by "_prepare_image_cache_entry" :: waited 0.009s {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:196}} Apr 17 15:45:47 user cinder-volume[84696]: DEBUG cinder.image.cache [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Found image-volume cache entry: {'id': 1, 'image_id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'volume_id': '006f20ae-5306-44fe-afdf-e433f79284e8', 'host': 'user@joviandss-2#joviandss-2', 'size': 1, 'image_updated_at': datetime.datetime(2023, 4, 17, 15, 19, 55), 'last_used': datetime.datetime(2023, 4, 17, 15, 45, 47, 745545)}. {{(pid=84696) get_entry /opt/stack/cinder/cinder/image/cache.py:76}} Apr 17 15:45:47 user cinder-volume[84696]: DEBUG cinder.image.cache [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Image-volume cache entry image_update_at = 2023-04-17 15:19:55+00:00, requested image updated_at = 2023-04-17 15:19:55+00:00. {{(pid=84696) _should_update_entry /opt/stack/cinder/cinder/image/cache.py:259}} Apr 17 15:45:47 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Found cache entry for image = b3f0dec1-7758-44a8-82e0-76ada8837428 on host user@joviandss-2#joviandss-2. {{(pid=84696) _prepare_image_cache_entry /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:886}} Apr 17 15:45:47 user cinder-volume[84696]: DEBUG cinder.coordination [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Lock "cinder-b3f0dec1-7758-44a8-82e0-76ada8837428" released by "_prepare_image_cache_entry" :: held 0.016s {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:207}} Apr 17 15:45:47 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Attempting to retrieve cache entry for image = b3f0dec1-7758-44a8-82e0-76ada8837428 on host user@joviandss-2#joviandss-2. {{(pid=84696) _create_from_image_cache /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:827}} Apr 17 15:45:47 user cinder-volume[84696]: DEBUG cinder.image.cache [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Found image-volume cache entry: {'id': 1, 'image_id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'volume_id': '006f20ae-5306-44fe-afdf-e433f79284e8', 'host': 'user@joviandss-2#joviandss-2', 'size': 1, 'image_updated_at': datetime.datetime(2023, 4, 17, 15, 19, 55), 'last_used': datetime.datetime(2023, 4, 17, 15, 45, 47, 760289)}. {{(pid=84696) get_entry /opt/stack/cinder/cinder/image/cache.py:76}} Apr 17 15:45:47 user cinder-volume[84696]: DEBUG cinder.image.cache [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Image-volume cache entry image_update_at = 2023-04-17 15:19:55+00:00, requested image updated_at = 2023-04-17 15:19:55+00:00. {{(pid=84696) _should_update_entry /opt/stack/cinder/cinder/image/cache.py:259}} Apr 17 15:45:47 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Creating from source image-volume 006f20ae-5306-44fe-afdf-e433f79284e8 {{(pid=84696) _create_from_image_cache /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:841}} Apr 17 15:45:47 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] cloned volume 006f20ae-5306-44fe-afdf-e433f79284e8 to 93a32a7c-4c6a-472a-8f91-9d34558ba232 {{(pid=84696) create_cloned_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:521}} Apr 17 15:45:47 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] cloning v_006f20ae-5306-44fe-afdf-e433f79284e8 to v_93a32a7c-4c6a-472a-8f91-9d34558ba232 {{(pid=84696) _clone_object /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:456}} Apr 17 15:45:47 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] create snapshot v_93a32a7c-4c6a-472a-8f91-9d34558ba232 {{(pid=84696) create_snapshot /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:635}} Apr 17 15:45:47 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:45:47 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:45:50 user cinder-volume[84696]: DEBUG cinder.utils [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.844(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:45:50 user cinder-volume[84696]: DEBUG cinder.utils [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:45:50 user cinder-volume[84696]: DEBUG cinder.utils [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 71.892(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:45:50 user cinder-volume[84696]: DEBUG cinder.utils [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:45:50 user cinder-volume[84696]: DEBUG cinder.utils [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 71.680(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:45:50 user cinder-volume[84696]: DEBUG cinder.utils [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:45:50 user cinder-volume[84696]: DEBUG cinder.utils [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.003(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:45:50 user cinder-volume[84696]: DEBUG cinder.utils [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_a27d0ff3-5de9-4feb-82f7-d4571af88bb9/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:45:50 user cinder-volume[84696]: DEBUG cinder.utils [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 71.456(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:45:50 user cinder-volume[84696]: DEBUG cinder.utils [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:45:53 user cinder-volume[84696]: DEBUG cinder.utils [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 74.321(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:45:53 user cinder-volume[84696]: DEBUG cinder.utils [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:45:53 user cinder-volume[84696]: DEBUG cinder.utils [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 74.110(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:45:53 user cinder-volume[84696]: DEBUG cinder.utils [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:45:53 user cinder-volume[84696]: DEBUG cinder.utils [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.920(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:45:53 user cinder-volume[84696]: DEBUG cinder.utils [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:45:53 user cinder-volume[84696]: DEBUG cinder.utils [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 40.480(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:45:53 user cinder-volume[84696]: DEBUG cinder.utils [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_7ce815ce-9603-4fd6-8cc8-9fd798b2f99d/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:45:54 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=17,cinder:UPDATE=4,cinder:INSERT=1 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:45:56 user cinder-volume[84696]: DEBUG cinder.utils [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 74.079(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:45:56 user cinder-volume[84696]: DEBUG cinder.utils [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:45:56 user cinder-volume[84696]: DEBUG cinder.utils [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 140.386(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:45:56 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_441910f8-d29c-4c9e-bc5b-780ad228c85d/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:45:56 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (8f2b47e7-826d-42cd-b35a-0914962c14c7) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 15:45:56 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (dde3ea60-4edc-4d71-bbb6-e3308a9eae12) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:45:56 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (dde3ea60-4edc-4d71-bbb6-e3308a9eae12) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:45:56 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Flow 'volume_create_manager' (29988406-5462-4c36-9fe1-ce18873db3ae) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 15:45:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 15:45:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 15:45:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 15:45:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 15:45:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 15:45:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 15:45:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 15:45:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 15:45:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 15:45:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 15:45:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 15:45:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 15:45:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 15:45:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 15:45:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 15:45:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 15:45:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 15:45:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 15:45:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 15:45:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 15:45:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 15:45:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 15:45:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 15:45:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 15:45:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:45:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 15:45:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 15:45:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 15:45:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 15:45:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 15:45:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 15:45:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 15:45:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 15:45:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 15:45:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 15:45:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 15:45:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 15:45:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 15:45:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 15:45:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 15:45:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 15:45:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 15:45:56 user cinder-volume[84696]: DEBUG cinder.utils [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.162(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:45:56 user cinder-volume[84696]: DEBUG cinder.utils [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_a27d0ff3-5de9-4feb-82f7-d4571af88bb9/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:45:56 user cinder-volume[84696]: DEBUG cinder.utils [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 140.237(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:45:56 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:45:56 user cinder-volume[84696]: INFO cinder.message.api [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Creating message record for request_id = req-51924a9e-5094-404b-8828-f76cf8bc761a Apr 17 15:45:56 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 15:45:56 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (13dbd7c5-0d4f-4ac8-9785-2f46c4a60b54) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 15:45:56 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 15:45:56 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:41:18Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesActionsTest-Volume-1624790909',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=441910f8-d29c-4c9e-bc5b-780ad228c85d,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='a8b97ff6ed6047f0a33e74684a1c162b',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:43:36Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:43:36Z,use_quota=True,user_id='c993ceeb4a5c49eea14057e2125518ad',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 15:45:56 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:41:18Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesActionsTest-Volume-1624790909',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=441910f8-d29c-4c9e-bc5b-780ad228c85d,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='a8b97ff6ed6047f0a33e74684a1c162b',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:43:36Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:43:36Z,use_quota=True,user_id='c993ceeb4a5c49eea14057e2125518ad',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(441910f8-d29c-4c9e-bc5b-780ad228c85d),volume_id=441910f8-d29c-4c9e-bc5b-780ad228c85d,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': '441910f8-d29c-4c9e-bc5b-780ad228c85d', 'volume_name': 'volume-441910f8-d29c-4c9e-bc5b-780ad228c85d', 'volume_size': 1}} Apr 17 15:45:56 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': '441910f8-d29c-4c9e-bc5b-780ad228c85d', 'volume': {'id': '441910f8-d29c-4c9e-bc5b-780ad228c85d', '_name_id': None, 'ec2_id': None, 'user_id': 'c993ceeb4a5c49eea14057e2125518ad', 'project_id': 'a8b97ff6ed6047f0a33e74684a1c162b', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-VolumesActionsTest-Volume-1624790909', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'admin_metadata': {}, 'glance_metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T15:41:18.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-441910f8-d29c-4c9e-bc5b-780ad228c85d', 'name_id': '441910f8-d29c-4c9e-bc5b-780ad228c85d', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumesActionsTest-Volume-1624790909', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': 'a8b97ff6ed6047f0a33e74684a1c162b', 'qos_specs': None, 'reservations': ['78c7033e-ef76-463d-90b8-8624379b802e', '4c9ae8cb-21fa-4ffc-97d0-d16076593bb1', '7ea6c36f-c448-4bcd-9d91-ca9a7f03d807', '60f944ba-84eb-4ff9-b309-d6e9af0d6b61'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'c993ceeb4a5c49eea14057e2125518ad', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumesActionsTest-Volume-1624790909', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': 'a8b97ff6ed6047f0a33e74684a1c162b', 'qos_specs': None, 'reservations': ['78c7033e-ef76-463d-90b8-8624379b802e', '4c9ae8cb-21fa-4ffc-97d0-d16076593bb1', '7ea6c36f-c448-4bcd-9d91-ca9a7f03d807', '60f944ba-84eb-4ff9-b309-d6e9af0d6b61'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'c993ceeb4a5c49eea14057e2125518ad', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': 'c993ceeb4a5c49eea14057e2125518ad', 'metadata': {}, 'qos_specs': None, 'retry': {'num_attempts': 2, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']}}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(441910f8-d29c-4c9e-bc5b-780ad228c85d),volume_id=441910f8-d29c-4c9e-bc5b-780ad228c85d,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:41:18Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesActionsTest-Volume-1624790909',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=441910f8-d29c-4c9e-bc5b-780ad228c85d,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='a8b97ff6ed6047f0a33e74684a1c162b',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:43:36Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:43:36Z,use_quota=True,user_id='c993ceeb4a5c49eea14057e2125518ad',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 15:45:56 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:41:18Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesActionsTest-Volume-1624790909',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=441910f8-d29c-4c9e-bc5b-780ad228c85d,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='a8b97ff6ed6047f0a33e74684a1c162b',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:43:36Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:43:36Z,use_quota=True,user_id='c993ceeb4a5c49eea14057e2125518ad',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:41:18Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesActionsTest-Volume-1624790909',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=441910f8-d29c-4c9e-bc5b-780ad228c85d,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='a8b97ff6ed6047f0a33e74684a1c162b',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:43:36Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:43:36Z,use_quota=True,user_id='c993ceeb4a5c49eea14057e2125518ad',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 15:45:56 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 15:45:56 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 15:45:56 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:45:56 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 15:45:56 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 15:45:56 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 15:45:56 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 15:45:56 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 15:45:56 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 15:45:56 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 15:45:56 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 15:45:56 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 15:45:56 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 15:45:56 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 15:45:56 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 15:45:56 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 15:45:56 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 15:45:56 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 15:45:56 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 15:45:56 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 15:45:56 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (13dbd7c5-0d4f-4ac8-9785-2f46c4a60b54) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:45:56 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (13dbd7c5-0d4f-4ac8-9785-2f46c4a60b54) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:45:56 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (46017a24-ff2b-4f74-9087-847879a38773) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:45:56 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (46017a24-ff2b-4f74-9087-847879a38773) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:45:56 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (a746d0ef-f626-4191-b33e-3af451243bd4) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:45:56 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (a746d0ef-f626-4191-b33e-3af451243bd4) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:45:56 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (4d0ba699-1018-4d1d-a6d5-05df49808c59) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:45:56 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Updating volume 441910f8-d29c-4c9e-bc5b-780ad228c85d with {'scheduled_at': datetime.datetime(2023, 4, 17, 15, 45, 56, 843241), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 15:45:56 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Volume 441910f8-d29c-4c9e-bc5b-780ad228c85d: re-scheduling SchedulerAPI.create_volume attempt 2 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 15:45:56 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Volume 441910f8-d29c-4c9e-bc5b-780ad228c85d: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 15:45:56 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] deleating volume v_441910f8-d29c-4c9e-bc5b-780ad228c85d {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 15:45:56 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] get snapshots for volume v_441910f8-d29c-4c9e-bc5b-780ad228c85d {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 15:45:56 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_441910f8-d29c-4c9e-bc5b-780ad228c85d/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:45:56 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_441910f8-d29c-4c9e-bc5b-780ad228c85d/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:45:56 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Volume reschedule parameters: True retry: {'num_attempts': 3, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 15:45:56 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Flow 'volume_create_manager' (9fa28ced-2781-4f50-a11f-28ca658d5dc7) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:45:57 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (f5a689b7-5e75-4896-96de-bb13400ad29d) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:45:57 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (f5a689b7-5e75-4896-96de-bb13400ad29d) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:41:18Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesActionsTest-Volume-1624790909',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=441910f8-d29c-4c9e-bc5b-780ad228c85d,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='a8b97ff6ed6047f0a33e74684a1c162b',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:45:57Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:45:57Z,use_quota=True,user_id='c993ceeb4a5c49eea14057e2125518ad',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:45:57 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (282c69f7-04a6-489b-b737-2ce95b429beb) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:45:57 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (282c69f7-04a6-489b-b737-2ce95b429beb) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:45:57 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (70f464cd-d9ff-4fa4-aa19-dccc1bdaefa2) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:45:57 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (70f464cd-d9ff-4fa4-aa19-dccc1bdaefa2) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': '441910f8-d29c-4c9e-bc5b-780ad228c85d', 'volume_name': 'volume-441910f8-d29c-4c9e-bc5b-780ad228c85d', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:45:57 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (7ff4de90-0cac-49ae-8514-c1d398eed82e) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:45:57 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (7ff4de90-0cac-49ae-8514-c1d398eed82e) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:45:57 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (82bda95f-132e-4edd-b2c1-b8ce90491f71) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:45:57 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Volume 441910f8-d29c-4c9e-bc5b-780ad228c85d: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-441910f8-d29c-4c9e-bc5b-780ad228c85d', 'volume_size': 1} Apr 17 15:45:57 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] creating volume v_441910f8-d29c-4c9e-bc5b-780ad228c85d. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 15:45:57 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] create volume {'name': 'v_441910f8-d29c-4c9e-bc5b-780ad228c85d', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 15:45:57 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:45:57 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:45:59 user cinder-volume[84696]: DEBUG cinder.utils [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.061(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:45:59 user cinder-volume[84696]: DEBUG cinder.utils [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:45:59 user cinder-volume[84696]: DEBUG cinder.utils [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.791(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:45:59 user cinder-volume[84696]: DEBUG cinder.utils [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:45:59 user cinder-volume[84696]: DEBUG cinder.utils [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.965(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:45:59 user cinder-volume[84696]: DEBUG cinder.utils [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_441910f8-d29c-4c9e-bc5b-780ad228c85d/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:46:02 user cinder-volume[84696]: DEBUG cinder.utils [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 139.485(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:46:02 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:46:02 user cinder-volume[84696]: INFO cinder.message.api [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Creating message record for request_id = req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b Apr 17 15:46:02 user cinder-volume[84696]: DEBUG cinder.utils [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.041(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:46:02 user cinder-volume[84696]: DEBUG cinder.utils [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_441910f8-d29c-4c9e-bc5b-780ad228c85d/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:46:02 user cinder-volume[84696]: DEBUG cinder.utils [req-6ee9fe55-7b3a-42cf-906e-c25ad8813b64 req-9a73de69-ad8a-45b1-8113-a76d843c2607 tempest-TestShelveInstance-1683612675 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 38.861(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:46:02 user cinder-volume[84696]: DEBUG cinder.utils [req-6ee9fe55-7b3a-42cf-906e-c25ad8813b64 req-9a73de69-ad8a-45b1-8113-a76d843c2607 tempest-TestShelveInstance-1683612675 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:d895d11d-6880-4418-a9ad-feca813ed5d6/luns/v_d895d11d-6880-4418-a9ad-feca813ed5d6 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:46:02 user cinder-volume[84696]: DEBUG cinder.utils [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.872(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:46:02 user cinder-volume[84696]: DEBUG cinder.utils [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:46:02 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 15:46:02 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (48562dfe-7934-46e1-bd66-7deffaf26fa9) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 15:46:02 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 15:46:02 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:43:43Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TestMinimumBasicScenario-volume-1067752463',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=5c16bb82-43c5-4d91-9320-0e1f2bbec574,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='17d1dae0ad7149bbbd160525d64c0c20',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:43:43Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:43:43Z,use_quota=True,user_id='3c407d9d494d485c9c95dc3981284fba',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 15:46:02 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:43:43Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TestMinimumBasicScenario-volume-1067752463',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=5c16bb82-43c5-4d91-9320-0e1f2bbec574,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='17d1dae0ad7149bbbd160525d64c0c20',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:43:43Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:43:43Z,use_quota=True,user_id='3c407d9d494d485c9c95dc3981284fba',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(5c16bb82-43c5-4d91-9320-0e1f2bbec574),volume_id=5c16bb82-43c5-4d91-9320-0e1f2bbec574,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': '5c16bb82-43c5-4d91-9320-0e1f2bbec574', 'volume_name': 'volume-5c16bb82-43c5-4d91-9320-0e1f2bbec574', 'volume_size': 1}} Apr 17 15:46:02 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'retry': {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']}, 'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': '5c16bb82-43c5-4d91-9320-0e1f2bbec574', 'volume': {'id': '5c16bb82-43c5-4d91-9320-0e1f2bbec574', '_name_id': None, 'ec2_id': None, 'user_id': '3c407d9d494d485c9c95dc3981284fba', 'project_id': '17d1dae0ad7149bbbd160525d64c0c20', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-TestMinimumBasicScenario-volume-1067752463', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T15:43:43.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-5c16bb82-43c5-4d91-9320-0e1f2bbec574', 'name_id': '5c16bb82-43c5-4d91-9320-0e1f2bbec574', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-TestMinimumBasicScenario-volume-1067752463', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '17d1dae0ad7149bbbd160525d64c0c20', 'qos_specs': None, 'reservations': ['dc14bf14-c68a-4eb7-b82a-f5cc6f1cd0d8', 'bc18e142-51f5-4393-aeb8-9697eb8e95e4', '3bbb7ba4-5edd-410f-beff-181109d36e4a', 'd2dbd02b-21f5-42b4-a139-2855ceadc14e'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': '3c407d9d494d485c9c95dc3981284fba', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-TestMinimumBasicScenario-volume-1067752463', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '17d1dae0ad7149bbbd160525d64c0c20', 'qos_specs': None, 'reservations': ['dc14bf14-c68a-4eb7-b82a-f5cc6f1cd0d8', 'bc18e142-51f5-4393-aeb8-9697eb8e95e4', '3bbb7ba4-5edd-410f-beff-181109d36e4a', 'd2dbd02b-21f5-42b4-a139-2855ceadc14e'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': '3c407d9d494d485c9c95dc3981284fba', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': '3c407d9d494d485c9c95dc3981284fba', 'metadata': {}, 'qos_specs': None}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(5c16bb82-43c5-4d91-9320-0e1f2bbec574),volume_id=5c16bb82-43c5-4d91-9320-0e1f2bbec574,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:43:43Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TestMinimumBasicScenario-volume-1067752463',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=5c16bb82-43c5-4d91-9320-0e1f2bbec574,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='17d1dae0ad7149bbbd160525d64c0c20',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:43:43Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:43:43Z,use_quota=True,user_id='3c407d9d494d485c9c95dc3981284fba',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 15:46:02 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:43:43Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TestMinimumBasicScenario-volume-1067752463',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=5c16bb82-43c5-4d91-9320-0e1f2bbec574,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='17d1dae0ad7149bbbd160525d64c0c20',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:43:43Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:43:43Z,use_quota=True,user_id='3c407d9d494d485c9c95dc3981284fba',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:43:43Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TestMinimumBasicScenario-volume-1067752463',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=5c16bb82-43c5-4d91-9320-0e1f2bbec574,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='17d1dae0ad7149bbbd160525d64c0c20',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:43:43Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:43:43Z,use_quota=True,user_id='3c407d9d494d485c9c95dc3981284fba',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 15:46:02 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 15:46:02 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 15:46:02 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:46:02 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 15:46:02 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 15:46:02 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 15:46:02 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 15:46:02 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 15:46:02 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 15:46:02 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 15:46:02 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 15:46:02 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 15:46:02 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 15:46:02 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 15:46:02 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 15:46:02 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 15:46:02 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 15:46:02 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 15:46:02 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 15:46:02 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 15:46:02 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (48562dfe-7934-46e1-bd66-7deffaf26fa9) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:46:02 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (48562dfe-7934-46e1-bd66-7deffaf26fa9) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:46:02 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (bd3deac7-067f-4638-802e-9e04e60d9396) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:46:02 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (bd3deac7-067f-4638-802e-9e04e60d9396) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:46:02 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (1217957a-28bd-42a3-8ec2-82b3cc7890c8) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:46:02 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (1217957a-28bd-42a3-8ec2-82b3cc7890c8) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:46:02 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (ba07b316-c816-4732-abc1-488bfdbe265a) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:46:02 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Updating volume 5c16bb82-43c5-4d91-9320-0e1f2bbec574 with {'scheduled_at': datetime.datetime(2023, 4, 17, 15, 46, 2, 981315), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 15:46:02 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Volume 5c16bb82-43c5-4d91-9320-0e1f2bbec574: re-scheduling SchedulerAPI.create_volume attempt 1 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 15:46:03 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Volume 5c16bb82-43c5-4d91-9320-0e1f2bbec574: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 15:46:03 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] deleating volume v_5c16bb82-43c5-4d91-9320-0e1f2bbec574 {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 15:46:03 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] get snapshots for volume v_5c16bb82-43c5-4d91-9320-0e1f2bbec574 {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 15:46:03 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_5c16bb82-43c5-4d91-9320-0e1f2bbec574/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:46:03 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_5c16bb82-43c5-4d91-9320-0e1f2bbec574/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:46:03 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Volume reschedule parameters: True retry: {'num_attempts': 2, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 15:46:03 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Flow 'volume_create_manager' (feb9fcc8-fc74-4994-8e14-fb8d6e4a7172) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:46:03 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (edd27c77-19b6-4c8d-a123-8b0c447218e5) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:46:03 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (edd27c77-19b6-4c8d-a123-8b0c447218e5) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:43:43Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TestMinimumBasicScenario-volume-1067752463',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=5c16bb82-43c5-4d91-9320-0e1f2bbec574,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='17d1dae0ad7149bbbd160525d64c0c20',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:46:03Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:46:03Z,use_quota=True,user_id='3c407d9d494d485c9c95dc3981284fba',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:46:03 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (23135497-50b1-444f-9668-b80977e69ae3) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:46:03 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (23135497-50b1-444f-9668-b80977e69ae3) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:46:03 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (56e57616-c597-47ee-a18a-2db3962f40ee) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:46:03 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (56e57616-c597-47ee-a18a-2db3962f40ee) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': '5c16bb82-43c5-4d91-9320-0e1f2bbec574', 'volume_name': 'volume-5c16bb82-43c5-4d91-9320-0e1f2bbec574', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:46:03 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (2622fe9a-32d4-49a4-b140-79612e7d3cb9) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:46:03 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (2622fe9a-32d4-49a4-b140-79612e7d3cb9) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:46:03 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (313e3521-7297-4c03-bcef-83a05d743c7b) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:46:03 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Volume 5c16bb82-43c5-4d91-9320-0e1f2bbec574: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-5c16bb82-43c5-4d91-9320-0e1f2bbec574', 'volume_size': 1} Apr 17 15:46:03 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] creating volume v_5c16bb82-43c5-4d91-9320-0e1f2bbec574. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 15:46:03 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] create volume {'name': 'v_5c16bb82-43c5-4d91-9320-0e1f2bbec574', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 15:46:03 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:46:03 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:46:05 user cinder-volume[84696]: DEBUG cinder.utils [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.976(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:46:05 user cinder-volume[84696]: DEBUG cinder.utils [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_5c16bb82-43c5-4d91-9320-0e1f2bbec574/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:46:05 user cinder-volume[84696]: DEBUG cinder.utils [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.363(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:46:05 user cinder-volume[84696]: DEBUG cinder.utils [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_a27d0ff3-5de9-4feb-82f7-d4571af88bb9/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:46:05 user cinder-volume[84696]: DEBUG cinder.utils [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.738(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:46:05 user cinder-volume[84696]: DEBUG cinder.utils [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:46:09 user cinder-volume[84696]: DEBUG cinder.utils [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.808(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:46:09 user cinder-volume[84696]: DEBUG cinder.utils [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:46:09 user cinder-volume[84696]: DEBUG cinder.utils [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.281(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:46:09 user cinder-volume[84696]: DEBUG cinder.utils [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:46:09 user cinder-volume[84696]: DEBUG cinder.utils [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.010(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:46:09 user cinder-volume[84696]: DEBUG cinder.utils [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:46:09 user cinder-volume[84696]: DEBUG cinder.utils [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.052(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:46:09 user cinder-volume[84696]: DEBUG cinder.utils [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_5c16bb82-43c5-4d91-9320-0e1f2bbec574/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:46:09 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 74.370(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:46:09 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:46:09 user cinder-volume[84696]: DEBUG cinder.utils [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.185(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:46:09 user cinder-volume[84696]: DEBUG cinder.utils [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_441910f8-d29c-4c9e-bc5b-780ad228c85d/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:46:14 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=20,cinder:INSERT=5,cinder:UPDATE=10 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:46:15 user cinder-volume[84696]: DEBUG cinder.utils [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 11.948(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:46:15 user cinder-volume[84696]: DEBUG cinder.utils [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:46:15 user cinder-volume[84696]: DEBUG cinder.utils [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.190(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:46:15 user cinder-volume[84696]: DEBUG cinder.utils [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_5c16bb82-43c5-4d91-9320-0e1f2bbec574/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:46:15 user cinder-volume[84696]: DEBUG cinder.utils [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 74.149(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:46:15 user cinder-volume[84696]: DEBUG cinder.utils [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_a50e9659-3e2d-4f04-b0ac-ae78b8f61fad/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:46:15 user cinder-volume[84696]: DEBUG cinder.utils [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 74.153(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:46:15 user cinder-volume[84696]: DEBUG cinder.utils [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_57070f23-6b40-47b9-b9b4-e0d77240f544/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:46:18 user cinder-volume[84696]: DEBUG cinder.utils [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.220(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:46:18 user cinder-volume[84696]: DEBUG cinder.utils [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:46:18 user cinder-volume[84696]: DEBUG cinder.utils [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 135.305(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:46:18 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:46:18 user cinder-volume[84696]: INFO cinder.message.api [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Creating message record for request_id = req-636827e5-ca50-4d75-a738-9b54469bf4fe Apr 17 15:46:18 user cinder-volume[84696]: DEBUG cinder.utils [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.396(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:46:18 user cinder-volume[84696]: DEBUG cinder.utils [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_441910f8-d29c-4c9e-bc5b-780ad228c85d/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:46:18 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 15:46:18 user cinder-volume[84696]: WARNING cinder.volume.manager [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (e2425f44-15f4-4105-a07c-08bc6bfa8c8e) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 15:46:18 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 15:46:18 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:44:03Z,deleted=False,deleted_at=None,display_description='',display_name='05e08eea-cc2c-4332-9e97-07cb9b466af6-blank-vol',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=2c9714de-f8ea-4d1d-99f9-beb85126ad91,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='540826be4baf46dba9617b2fd47821a3',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:44:03Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:44:03Z,use_quota=True,user_id='d819e3f51368413c99691d1b45bbcfb1',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 15:46:18 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:44:03Z,deleted=False,deleted_at=None,display_description='',display_name='05e08eea-cc2c-4332-9e97-07cb9b466af6-blank-vol',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=2c9714de-f8ea-4d1d-99f9-beb85126ad91,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='540826be4baf46dba9617b2fd47821a3',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:44:03Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:44:03Z,use_quota=True,user_id='d819e3f51368413c99691d1b45bbcfb1',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(2c9714de-f8ea-4d1d-99f9-beb85126ad91),volume_id=2c9714de-f8ea-4d1d-99f9-beb85126ad91,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': '2c9714de-f8ea-4d1d-99f9-beb85126ad91', 'volume_name': 'volume-2c9714de-f8ea-4d1d-99f9-beb85126ad91', 'volume_size': 1}} Apr 17 15:46:18 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'retry': {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']}, 'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': '2c9714de-f8ea-4d1d-99f9-beb85126ad91', 'volume': {'id': '2c9714de-f8ea-4d1d-99f9-beb85126ad91', '_name_id': None, 'ec2_id': None, 'user_id': 'd819e3f51368413c99691d1b45bbcfb1', 'project_id': '540826be4baf46dba9617b2fd47821a3', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': '05e08eea-cc2c-4332-9e97-07cb9b466af6-blank-vol', 'display_description': '', 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T15:44:02.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-2c9714de-f8ea-4d1d-99f9-beb85126ad91', 'name_id': '2c9714de-f8ea-4d1d-99f9-beb85126ad91', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': '', 'display_name': '05e08eea-cc2c-4332-9e97-07cb9b466af6-blank-vol', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '540826be4baf46dba9617b2fd47821a3', 'qos_specs': None, 'reservations': ['e20dc1ca-0a9e-4413-be1e-c7f19faae9e1', '81c416d5-515b-4265-888a-740c186d6f45', 'da7244e3-def8-4c3b-b96c-0088e02b1914', 'afef167b-9548-4f1a-a499-b206d13c826a'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'd819e3f51368413c99691d1b45bbcfb1', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': '', 'display_name': '05e08eea-cc2c-4332-9e97-07cb9b466af6-blank-vol', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '540826be4baf46dba9617b2fd47821a3', 'qos_specs': None, 'reservations': ['e20dc1ca-0a9e-4413-be1e-c7f19faae9e1', '81c416d5-515b-4265-888a-740c186d6f45', 'da7244e3-def8-4c3b-b96c-0088e02b1914', 'afef167b-9548-4f1a-a499-b206d13c826a'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'd819e3f51368413c99691d1b45bbcfb1', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': 'd819e3f51368413c99691d1b45bbcfb1', 'metadata': {}, 'qos_specs': None}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(2c9714de-f8ea-4d1d-99f9-beb85126ad91),volume_id=2c9714de-f8ea-4d1d-99f9-beb85126ad91,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:44:03Z,deleted=False,deleted_at=None,display_description='',display_name='05e08eea-cc2c-4332-9e97-07cb9b466af6-blank-vol',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=2c9714de-f8ea-4d1d-99f9-beb85126ad91,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='540826be4baf46dba9617b2fd47821a3',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:44:03Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:44:03Z,use_quota=True,user_id='d819e3f51368413c99691d1b45bbcfb1',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 15:46:18 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:44:03Z,deleted=False,deleted_at=None,display_description='',display_name='05e08eea-cc2c-4332-9e97-07cb9b466af6-blank-vol',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=2c9714de-f8ea-4d1d-99f9-beb85126ad91,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='540826be4baf46dba9617b2fd47821a3',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:44:03Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:44:03Z,use_quota=True,user_id='d819e3f51368413c99691d1b45bbcfb1',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:44:03Z,deleted=False,deleted_at=None,display_description='',display_name='05e08eea-cc2c-4332-9e97-07cb9b466af6-blank-vol',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=2c9714de-f8ea-4d1d-99f9-beb85126ad91,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='540826be4baf46dba9617b2fd47821a3',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:44:03Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:44:03Z,use_quota=True,user_id='d819e3f51368413c99691d1b45bbcfb1',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 15:46:18 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 15:46:18 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 15:46:18 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:46:18 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 15:46:18 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 15:46:18 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 15:46:18 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 15:46:18 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 15:46:18 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 15:46:18 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 15:46:18 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 15:46:18 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 15:46:18 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 15:46:18 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 15:46:18 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 15:46:18 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 15:46:18 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 15:46:18 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 15:46:18 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 15:46:18 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 15:46:18 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (e2425f44-15f4-4105-a07c-08bc6bfa8c8e) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:46:18 user cinder-volume[84696]: WARNING cinder.volume.manager [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (e2425f44-15f4-4105-a07c-08bc6bfa8c8e) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:46:18 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (de8925ee-bbf5-4cf6-a515-9dca7f4b11cd) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:46:18 user cinder-volume[84696]: WARNING cinder.volume.manager [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (de8925ee-bbf5-4cf6-a515-9dca7f4b11cd) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:46:18 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (1d0c2f5a-3d4a-4ec8-be80-f06f6886d0c3) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:46:18 user cinder-volume[84696]: WARNING cinder.volume.manager [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (1d0c2f5a-3d4a-4ec8-be80-f06f6886d0c3) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:46:18 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (91eb764b-f46d-44d6-b36e-ef5e6fb68451) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:46:18 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Updating volume 2c9714de-f8ea-4d1d-99f9-beb85126ad91 with {'scheduled_at': datetime.datetime(2023, 4, 17, 15, 46, 18, 329210), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 15:46:18 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Volume 2c9714de-f8ea-4d1d-99f9-beb85126ad91: re-scheduling SchedulerAPI.create_volume attempt 1 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 15:46:18 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Volume 2c9714de-f8ea-4d1d-99f9-beb85126ad91: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 15:46:18 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] deleating volume v_2c9714de-f8ea-4d1d-99f9-beb85126ad91 {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 15:46:18 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] get snapshots for volume v_2c9714de-f8ea-4d1d-99f9-beb85126ad91 {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 15:46:18 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_2c9714de-f8ea-4d1d-99f9-beb85126ad91/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:46:18 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_2c9714de-f8ea-4d1d-99f9-beb85126ad91/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:46:18 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Volume reschedule parameters: True retry: {'num_attempts': 2, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 15:46:18 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Flow 'volume_create_manager' (26920333-f21d-422c-8a95-071d6214fa61) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:46:18 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (ce5b4d58-ae31-4588-b654-7c512790cdca) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:46:18 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (ce5b4d58-ae31-4588-b654-7c512790cdca) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:44:03Z,deleted=False,deleted_at=None,display_description='',display_name='05e08eea-cc2c-4332-9e97-07cb9b466af6-blank-vol',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=2c9714de-f8ea-4d1d-99f9-beb85126ad91,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='540826be4baf46dba9617b2fd47821a3',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:46:18Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:46:18Z,use_quota=True,user_id='d819e3f51368413c99691d1b45bbcfb1',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:46:18 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (6d6245bf-af9c-4f2e-b7ff-5d36fbf9d72d) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:46:18 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (6d6245bf-af9c-4f2e-b7ff-5d36fbf9d72d) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:46:18 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (709c2d10-0581-4c42-964c-79370796a7ff) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:46:18 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (709c2d10-0581-4c42-964c-79370796a7ff) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': '2c9714de-f8ea-4d1d-99f9-beb85126ad91', 'volume_name': 'volume-2c9714de-f8ea-4d1d-99f9-beb85126ad91', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:46:18 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (e0532b43-faf5-4c9a-bef5-86d349478076) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:46:18 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (e0532b43-faf5-4c9a-bef5-86d349478076) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:46:18 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (2eecdd60-c8ce-49c2-8382-d7abd644f212) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:46:18 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Volume 2c9714de-f8ea-4d1d-99f9-beb85126ad91: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-2c9714de-f8ea-4d1d-99f9-beb85126ad91', 'volume_size': 1} Apr 17 15:46:18 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] creating volume v_2c9714de-f8ea-4d1d-99f9-beb85126ad91. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 15:46:18 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] create volume {'name': 'v_2c9714de-f8ea-4d1d-99f9-beb85126ad91', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 15:46:18 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:46:18 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:46:21 user cinder-volume[84696]: DEBUG cinder.utils [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.987(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:46:21 user cinder-volume[84696]: DEBUG cinder.utils [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_2c9714de-f8ea-4d1d-99f9-beb85126ad91/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:46:21 user cinder-volume[84696]: DEBUG cinder.utils [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.802(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:46:21 user cinder-volume[84696]: DEBUG cinder.utils [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:46:24 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=21,cinder:UPDATE=3,cinder:INSERT=2 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:46:24 user cinder-volume[84696]: DEBUG cinder.utils [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.055(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:46:24 user cinder-volume[84696]: DEBUG cinder.utils [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_2c9714de-f8ea-4d1d-99f9-beb85126ad91/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:46:24 user cinder-volume[84696]: DEBUG cinder.utils [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.870(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:46:24 user cinder-volume[84696]: DEBUG cinder.utils [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:46:24 user cinder-volume[84696]: DEBUG cinder.utils [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.166(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:46:24 user cinder-volume[84696]: DEBUG cinder.utils [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:46:24 user cinder-volume[84696]: DEBUG cinder.utils [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.408(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:46:24 user cinder-volume[84696]: DEBUG cinder.utils [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_5c16bb82-43c5-4d91-9320-0e1f2bbec574/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:46:24 user cinder-volume[84696]: DEBUG cinder.utils [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.794(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:46:24 user cinder-volume[84696]: DEBUG cinder.utils [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_a27d0ff3-5de9-4feb-82f7-d4571af88bb9/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:46:27 user cinder-volume[84696]: DEBUG cinder.utils [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.708(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:46:27 user cinder-volume[84696]: DEBUG cinder.utils [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:46:27 user cinder-volume[84696]: DEBUG cinder.utils [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 74.268(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:46:27 user cinder-volume[84696]: DEBUG cinder.utils [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_7ce815ce-9603-4fd6-8cc8-9fd798b2f99d/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:46:30 user cinder-volume[84696]: DEBUG cinder.utils [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.199(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:46:30 user cinder-volume[84696]: DEBUG cinder.utils [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_2c9714de-f8ea-4d1d-99f9-beb85126ad91/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:46:30 user cinder-volume[84696]: DEBUG cinder.utils [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.013(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:46:30 user cinder-volume[84696]: DEBUG cinder.utils [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:46:37 user cinder-volume[84696]: DEBUG cinder.utils [req-6ee9fe55-7b3a-42cf-906e-c25ad8813b64 req-9a73de69-ad8a-45b1-8113-a76d843c2607 tempest-TestShelveInstance-1683612675 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 73.091(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:46:37 user cinder-volume[84696]: DEBUG cinder.utils [req-6ee9fe55-7b3a-42cf-906e-c25ad8813b64 req-9a73de69-ad8a-45b1-8113-a76d843c2607 tempest-TestShelveInstance-1683612675 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:d895d11d-6880-4418-a9ad-feca813ed5d6/luns/v_d895d11d-6880-4418-a9ad-feca813ed5d6 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:46:37 user cinder-volume[84696]: DEBUG cinder.utils [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 40.102(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:46:37 user cinder-volume[84696]: DEBUG cinder.utils [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:46:37 user cinder-volume[84696]: DEBUG cinder.utils [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 40.275(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:46:37 user cinder-volume[84696]: DEBUG cinder.utils [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_441910f8-d29c-4c9e-bc5b-780ad228c85d/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:46:40 user cinder-volume[84696]: DEBUG cinder.utils [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.676(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:46:40 user cinder-volume[84696]: DEBUG cinder.utils [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:46:40 user cinder-volume[84696]: DEBUG cinder.utils [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.864(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:46:40 user cinder-volume[84696]: DEBUG cinder.utils [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_2c9714de-f8ea-4d1d-99f9-beb85126ad91/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:46:43 user cinder-volume[84696]: DEBUG cinder.utils [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 40.044(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:46:43 user cinder-volume[84696]: DEBUG cinder.utils [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:46:43 user cinder-volume[84696]: DEBUG cinder.utils [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 40.285(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:46:43 user cinder-volume[84696]: DEBUG cinder.utils [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_5c16bb82-43c5-4d91-9320-0e1f2bbec574/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:46:54 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=11,cinder:UPDATE=4 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:46:57 user cinder-volume[84696]: DEBUG cinder.utils [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 138.010(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:46:57 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:46:57 user cinder-volume[84696]: INFO cinder.message.api [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Creating message record for request_id = req-be98cecd-d344-4ab9-b5f6-c268a7041adb Apr 17 15:46:57 user cinder-volume[84696]: DEBUG cinder.utils [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 38.830(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:46:57 user cinder-volume[84696]: DEBUG cinder.utils [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_2c9714de-f8ea-4d1d-99f9-beb85126ad91/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:46:57 user cinder-volume[84696]: DEBUG cinder.utils [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 138.244(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:46:57 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:46:57 user cinder-volume[84696]: INFO cinder.message.api [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Creating message record for request_id = req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f Apr 17 15:46:57 user cinder-volume[84696]: DEBUG cinder.utils [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 138.460(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:46:57 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:46:57 user cinder-volume[84696]: INFO cinder.message.api [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Creating message record for request_id = req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 Apr 17 15:46:57 user cinder-volume[84696]: DEBUG cinder.utils [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 72.571(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:46:57 user cinder-volume[84696]: DEBUG cinder.utils [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_a27d0ff3-5de9-4feb-82f7-d4571af88bb9/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:46:57 user cinder-volume[84696]: DEBUG cinder.utils [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 38.650(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:46:57 user cinder-volume[84696]: DEBUG cinder.utils [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:46:57 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 15:46:57 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=5,cinder:UPDATE=3 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:46:57 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 15:46:57 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 15:46:57 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (d5773f69-e8c3-40bd-9484-70f6d2778aa5) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 15:46:57 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 15:46:57 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:44:39Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-CreateVolumesFromImageTest-Volume-2146530532',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=fbbadf16-651a-47d5-b308-a37e56aa07f2,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='75babbbca67f4035a48ee8ee816b18db',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:44:39Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:44:39Z,use_quota=True,user_id='d5299006144f4ee6817349348a489017',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 15:46:57 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:44:39Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-CreateVolumesFromImageTest-Volume-2146530532',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=fbbadf16-651a-47d5-b308-a37e56aa07f2,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='75babbbca67f4035a48ee8ee816b18db',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:44:39Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:44:39Z,use_quota=True,user_id='d5299006144f4ee6817349348a489017',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(fbbadf16-651a-47d5-b308-a37e56aa07f2),volume_id=fbbadf16-651a-47d5-b308-a37e56aa07f2,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': 'fbbadf16-651a-47d5-b308-a37e56aa07f2', 'volume_name': 'volume-fbbadf16-651a-47d5-b308-a37e56aa07f2', 'volume_size': 1}} Apr 17 15:46:57 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'retry': {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']}, 'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': 'fbbadf16-651a-47d5-b308-a37e56aa07f2', 'volume': {'id': 'fbbadf16-651a-47d5-b308-a37e56aa07f2', '_name_id': None, 'ec2_id': None, 'user_id': 'd5299006144f4ee6817349348a489017', 'project_id': '75babbbca67f4035a48ee8ee816b18db', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-CreateVolumesFromImageTest-Volume-2146530532', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T15:44:38.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-fbbadf16-651a-47d5-b308-a37e56aa07f2', 'name_id': 'fbbadf16-651a-47d5-b308-a37e56aa07f2', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-CreateVolumesFromImageTest-Volume-2146530532', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '75babbbca67f4035a48ee8ee816b18db', 'qos_specs': None, 'reservations': ['ebe2c0e3-e7c3-4d72-b0ff-d7918e651f9b', '11cb75cb-f9b7-46b2-bc19-f76f314b9643', '9657a0be-fa65-4315-b170-50972324b109', 'b1d14f83-c9a6-448d-acae-a1bf415819e5'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'd5299006144f4ee6817349348a489017', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-CreateVolumesFromImageTest-Volume-2146530532', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '75babbbca67f4035a48ee8ee816b18db', 'qos_specs': None, 'reservations': ['ebe2c0e3-e7c3-4d72-b0ff-d7918e651f9b', '11cb75cb-f9b7-46b2-bc19-f76f314b9643', '9657a0be-fa65-4315-b170-50972324b109', 'b1d14f83-c9a6-448d-acae-a1bf415819e5'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'd5299006144f4ee6817349348a489017', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': 'd5299006144f4ee6817349348a489017', 'metadata': {}, 'qos_specs': None}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(fbbadf16-651a-47d5-b308-a37e56aa07f2),volume_id=fbbadf16-651a-47d5-b308-a37e56aa07f2,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:44:39Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-CreateVolumesFromImageTest-Volume-2146530532',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=fbbadf16-651a-47d5-b308-a37e56aa07f2,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='75babbbca67f4035a48ee8ee816b18db',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:44:39Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:44:39Z,use_quota=True,user_id='d5299006144f4ee6817349348a489017',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 15:46:57 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:44:39Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-CreateVolumesFromImageTest-Volume-2146530532',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=fbbadf16-651a-47d5-b308-a37e56aa07f2,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='75babbbca67f4035a48ee8ee816b18db',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:44:39Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:44:39Z,use_quota=True,user_id='d5299006144f4ee6817349348a489017',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:44:39Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-CreateVolumesFromImageTest-Volume-2146530532',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=fbbadf16-651a-47d5-b308-a37e56aa07f2,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='75babbbca67f4035a48ee8ee816b18db',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:44:39Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:44:39Z,use_quota=True,user_id='d5299006144f4ee6817349348a489017',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 15:46:57 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 15:46:57 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 15:46:57 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:46:57 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 15:46:57 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 15:46:57 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 15:46:57 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 15:46:57 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 15:46:57 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 15:46:57 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 15:46:57 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 15:46:57 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 15:46:57 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 15:46:57 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 15:46:57 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 15:46:57 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 15:46:57 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 15:46:57 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 15:46:57 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 15:46:57 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 15:46:57 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (d5773f69-e8c3-40bd-9484-70f6d2778aa5) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:46:57 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (d5773f69-e8c3-40bd-9484-70f6d2778aa5) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:46:57 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (e329f716-c4db-4244-bd9f-1c4f59575094) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:46:57 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (e329f716-c4db-4244-bd9f-1c4f59575094) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:46:57 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (a3ce4a58-f2fc-40ce-951e-9808b120591b) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:46:57 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (a3ce4a58-f2fc-40ce-951e-9808b120591b) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:46:57 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (07ade4d8-0421-45fb-9322-2eed895f7d00) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:46:57 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Updating volume fbbadf16-651a-47d5-b308-a37e56aa07f2 with {'scheduled_at': datetime.datetime(2023, 4, 17, 15, 46, 57, 260221), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 15:46:57 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (e66d2ac4-ac3f-4a64-832b-f75a72f684a5) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 15:46:57 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 15:46:57 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:44:38Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-CreateVolumesFromImageTest-Volume-1463431787',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=dfd745ed-b0c4-4e35-b77c-a911c90ce33e,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='75babbbca67f4035a48ee8ee816b18db',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:44:39Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:44:39Z,use_quota=True,user_id='d5299006144f4ee6817349348a489017',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 15:46:57 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:44:38Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-CreateVolumesFromImageTest-Volume-1463431787',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=dfd745ed-b0c4-4e35-b77c-a911c90ce33e,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='75babbbca67f4035a48ee8ee816b18db',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:44:39Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:44:39Z,use_quota=True,user_id='d5299006144f4ee6817349348a489017',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(dfd745ed-b0c4-4e35-b77c-a911c90ce33e),volume_id=dfd745ed-b0c4-4e35-b77c-a911c90ce33e,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': 'dfd745ed-b0c4-4e35-b77c-a911c90ce33e', 'volume_name': 'volume-dfd745ed-b0c4-4e35-b77c-a911c90ce33e', 'volume_size': 1}} Apr 17 15:46:57 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'retry': {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']}, 'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': 'dfd745ed-b0c4-4e35-b77c-a911c90ce33e', 'volume': {'id': 'dfd745ed-b0c4-4e35-b77c-a911c90ce33e', '_name_id': None, 'ec2_id': None, 'user_id': 'd5299006144f4ee6817349348a489017', 'project_id': '75babbbca67f4035a48ee8ee816b18db', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-CreateVolumesFromImageTest-Volume-1463431787', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T15:44:38.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-dfd745ed-b0c4-4e35-b77c-a911c90ce33e', 'name_id': 'dfd745ed-b0c4-4e35-b77c-a911c90ce33e', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-CreateVolumesFromImageTest-Volume-1463431787', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '75babbbca67f4035a48ee8ee816b18db', 'qos_specs': None, 'reservations': ['24de6078-218c-412e-8c05-3e1cd2d51a18', '25d60ea2-888f-44b7-a809-be58574b0515', '8d3c3175-5e14-45c1-aac7-a222d3122bbb', '7c2bd4f1-736b-4589-98c3-db4b057ca99b'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'd5299006144f4ee6817349348a489017', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-CreateVolumesFromImageTest-Volume-1463431787', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '75babbbca67f4035a48ee8ee816b18db', 'qos_specs': None, 'reservations': ['24de6078-218c-412e-8c05-3e1cd2d51a18', '25d60ea2-888f-44b7-a809-be58574b0515', '8d3c3175-5e14-45c1-aac7-a222d3122bbb', '7c2bd4f1-736b-4589-98c3-db4b057ca99b'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'd5299006144f4ee6817349348a489017', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': 'd5299006144f4ee6817349348a489017', 'metadata': {}, 'qos_specs': None}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(dfd745ed-b0c4-4e35-b77c-a911c90ce33e),volume_id=dfd745ed-b0c4-4e35-b77c-a911c90ce33e,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:44:38Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-CreateVolumesFromImageTest-Volume-1463431787',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=dfd745ed-b0c4-4e35-b77c-a911c90ce33e,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='75babbbca67f4035a48ee8ee816b18db',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:44:39Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:44:39Z,use_quota=True,user_id='d5299006144f4ee6817349348a489017',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 15:46:57 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:44:38Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-CreateVolumesFromImageTest-Volume-1463431787',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=dfd745ed-b0c4-4e35-b77c-a911c90ce33e,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='75babbbca67f4035a48ee8ee816b18db',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:44:39Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:44:39Z,use_quota=True,user_id='d5299006144f4ee6817349348a489017',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:44:38Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-CreateVolumesFromImageTest-Volume-1463431787',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=dfd745ed-b0c4-4e35-b77c-a911c90ce33e,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='75babbbca67f4035a48ee8ee816b18db',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:44:39Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:44:39Z,use_quota=True,user_id='d5299006144f4ee6817349348a489017',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 15:46:57 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 15:46:57 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 15:46:57 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:46:57 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 15:46:57 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 15:46:57 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 15:46:57 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 15:46:57 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 15:46:57 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 15:46:57 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 15:46:57 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 15:46:57 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 15:46:57 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 15:46:57 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 15:46:57 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 15:46:57 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 15:46:57 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 15:46:57 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 15:46:57 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 15:46:57 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 15:46:57 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (e66d2ac4-ac3f-4a64-832b-f75a72f684a5) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:46:57 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (e66d2ac4-ac3f-4a64-832b-f75a72f684a5) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:46:57 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (c50e358a-199d-4c99-b7fa-60390bab6fb1) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:46:57 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (c50e358a-199d-4c99-b7fa-60390bab6fb1) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:46:57 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (c861ba13-463a-4282-8a81-0a0f685ac80a) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:46:57 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (c861ba13-463a-4282-8a81-0a0f685ac80a) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:46:57 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (b63ce73b-bcf8-4cfb-87d9-296580b1da26) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:46:57 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Updating volume dfd745ed-b0c4-4e35-b77c-a911c90ce33e with {'scheduled_at': datetime.datetime(2023, 4, 17, 15, 46, 57, 315557), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 15:46:57 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (b50442f2-2fa6-4ae8-825e-4ae1ed629db3) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 15:46:57 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 15:46:57 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:44:39Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-CreateVolumesFromImageTest-Volume-800604710',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=aea45c70-7c05-4146-847e-cbb777d3dfbe,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='75babbbca67f4035a48ee8ee816b18db',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:44:39Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:44:39Z,use_quota=True,user_id='d5299006144f4ee6817349348a489017',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 15:46:57 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:44:39Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-CreateVolumesFromImageTest-Volume-800604710',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=aea45c70-7c05-4146-847e-cbb777d3dfbe,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='75babbbca67f4035a48ee8ee816b18db',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:44:39Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:44:39Z,use_quota=True,user_id='d5299006144f4ee6817349348a489017',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(aea45c70-7c05-4146-847e-cbb777d3dfbe),volume_id=aea45c70-7c05-4146-847e-cbb777d3dfbe,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': 'aea45c70-7c05-4146-847e-cbb777d3dfbe', 'volume_name': 'volume-aea45c70-7c05-4146-847e-cbb777d3dfbe', 'volume_size': 1}} Apr 17 15:46:57 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'retry': {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']}, 'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': 'aea45c70-7c05-4146-847e-cbb777d3dfbe', 'volume': {'id': 'aea45c70-7c05-4146-847e-cbb777d3dfbe', '_name_id': None, 'ec2_id': None, 'user_id': 'd5299006144f4ee6817349348a489017', 'project_id': '75babbbca67f4035a48ee8ee816b18db', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-CreateVolumesFromImageTest-Volume-800604710', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T15:44:38.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-aea45c70-7c05-4146-847e-cbb777d3dfbe', 'name_id': 'aea45c70-7c05-4146-847e-cbb777d3dfbe', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-CreateVolumesFromImageTest-Volume-800604710', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '75babbbca67f4035a48ee8ee816b18db', 'qos_specs': None, 'reservations': ['0a40a0f0-8514-4f1f-ac7e-cbc2d47cbe5f', 'b703b541-7376-4c99-8896-74ebf0e7fb9f', '4c0d25ea-51e7-41c5-8f64-d45e42a842b1', '32031e87-40eb-4f2d-aa17-2ea553f6a406'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'd5299006144f4ee6817349348a489017', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-CreateVolumesFromImageTest-Volume-800604710', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '75babbbca67f4035a48ee8ee816b18db', 'qos_specs': None, 'reservations': ['0a40a0f0-8514-4f1f-ac7e-cbc2d47cbe5f', 'b703b541-7376-4c99-8896-74ebf0e7fb9f', '4c0d25ea-51e7-41c5-8f64-d45e42a842b1', '32031e87-40eb-4f2d-aa17-2ea553f6a406'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'd5299006144f4ee6817349348a489017', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': 'd5299006144f4ee6817349348a489017', 'metadata': {}, 'qos_specs': None}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(aea45c70-7c05-4146-847e-cbb777d3dfbe),volume_id=aea45c70-7c05-4146-847e-cbb777d3dfbe,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:44:39Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-CreateVolumesFromImageTest-Volume-800604710',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=aea45c70-7c05-4146-847e-cbb777d3dfbe,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='75babbbca67f4035a48ee8ee816b18db',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:44:39Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:44:39Z,use_quota=True,user_id='d5299006144f4ee6817349348a489017',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 15:46:57 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:44:39Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-CreateVolumesFromImageTest-Volume-800604710',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=aea45c70-7c05-4146-847e-cbb777d3dfbe,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='75babbbca67f4035a48ee8ee816b18db',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:44:39Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:44:39Z,use_quota=True,user_id='d5299006144f4ee6817349348a489017',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:44:39Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-CreateVolumesFromImageTest-Volume-800604710',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=aea45c70-7c05-4146-847e-cbb777d3dfbe,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='75babbbca67f4035a48ee8ee816b18db',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:44:39Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:44:39Z,use_quota=True,user_id='d5299006144f4ee6817349348a489017',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 15:46:57 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 15:46:57 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 15:46:57 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:46:57 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 15:46:57 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 15:46:57 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 15:46:57 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 15:46:57 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 15:46:57 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 15:46:57 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 15:46:57 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 15:46:57 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 15:46:57 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 15:46:57 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 15:46:57 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 15:46:57 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 15:46:57 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 15:46:57 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 15:46:57 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 15:46:57 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 15:46:57 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (b50442f2-2fa6-4ae8-825e-4ae1ed629db3) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:46:57 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (b50442f2-2fa6-4ae8-825e-4ae1ed629db3) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:46:57 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (a8042c3a-402f-47d4-b497-1b5428ab1082) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:46:57 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (a8042c3a-402f-47d4-b497-1b5428ab1082) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:46:57 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (7b117907-bafc-4b55-9754-be5fb969a68b) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:46:57 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (7b117907-bafc-4b55-9754-be5fb969a68b) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:46:57 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (947d63b2-51cb-4ef3-83c3-96e17bebdb4c) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:46:57 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Updating volume aea45c70-7c05-4146-847e-cbb777d3dfbe with {'scheduled_at': datetime.datetime(2023, 4, 17, 15, 46, 57, 359405), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 15:46:57 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Volume fbbadf16-651a-47d5-b308-a37e56aa07f2: re-scheduling SchedulerAPI.create_volume attempt 1 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 15:46:57 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Volume dfd745ed-b0c4-4e35-b77c-a911c90ce33e: re-scheduling SchedulerAPI.create_volume attempt 1 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 15:46:57 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Volume fbbadf16-651a-47d5-b308-a37e56aa07f2: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 15:46:57 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] deleating volume v_fbbadf16-651a-47d5-b308-a37e56aa07f2 {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 15:46:57 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] get snapshots for volume v_fbbadf16-651a-47d5-b308-a37e56aa07f2 {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 15:46:57 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_fbbadf16-651a-47d5-b308-a37e56aa07f2/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:46:57 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_fbbadf16-651a-47d5-b308-a37e56aa07f2/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:46:57 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Volume aea45c70-7c05-4146-847e-cbb777d3dfbe: re-scheduling SchedulerAPI.create_volume attempt 1 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 15:46:57 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Volume dfd745ed-b0c4-4e35-b77c-a911c90ce33e: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 15:46:57 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] deleating volume v_dfd745ed-b0c4-4e35-b77c-a911c90ce33e {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 15:46:57 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] get snapshots for volume v_dfd745ed-b0c4-4e35-b77c-a911c90ce33e {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 15:46:57 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_dfd745ed-b0c4-4e35-b77c-a911c90ce33e/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:46:57 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_dfd745ed-b0c4-4e35-b77c-a911c90ce33e/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:46:57 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Volume aea45c70-7c05-4146-847e-cbb777d3dfbe: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 15:46:57 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] deleating volume v_aea45c70-7c05-4146-847e-cbb777d3dfbe {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 15:46:57 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] get snapshots for volume v_aea45c70-7c05-4146-847e-cbb777d3dfbe {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 15:46:57 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_aea45c70-7c05-4146-847e-cbb777d3dfbe/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:46:57 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_aea45c70-7c05-4146-847e-cbb777d3dfbe/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:46:57 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Volume reschedule parameters: True retry: {'num_attempts': 2, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 15:46:57 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Flow 'volume_create_manager' (15d67a7b-0986-468a-b44b-c6ffaf68d3fa) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:46:57 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (c0efca2b-d4b5-442d-a5ed-4f1fef3b0734) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:46:57 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (c0efca2b-d4b5-442d-a5ed-4f1fef3b0734) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:44:39Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-CreateVolumesFromImageTest-Volume-2146530532',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=fbbadf16-651a-47d5-b308-a37e56aa07f2,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='75babbbca67f4035a48ee8ee816b18db',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:46:57Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:46:57Z,use_quota=True,user_id='d5299006144f4ee6817349348a489017',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:46:57 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (6af14be6-309b-47d8-9e72-4d46f866188d) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:46:57 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (6af14be6-309b-47d8-9e72-4d46f866188d) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:46:57 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (a1ae4143-8324-4ec4-92b5-17e7ab558456) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:46:57 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (a1ae4143-8324-4ec4-92b5-17e7ab558456) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': 'fbbadf16-651a-47d5-b308-a37e56aa07f2', 'volume_name': 'volume-fbbadf16-651a-47d5-b308-a37e56aa07f2', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:46:57 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (9cc722d5-3458-4059-b7c9-a8df875a18a8) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:46:57 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (9cc722d5-3458-4059-b7c9-a8df875a18a8) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:46:57 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (a5831bc7-05e8-4397-ac61-3fc78593eda3) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:46:57 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Volume fbbadf16-651a-47d5-b308-a37e56aa07f2: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-fbbadf16-651a-47d5-b308-a37e56aa07f2', 'volume_size': 1} Apr 17 15:46:57 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] creating volume v_fbbadf16-651a-47d5-b308-a37e56aa07f2. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 15:46:57 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] create volume {'name': 'v_fbbadf16-651a-47d5-b308-a37e56aa07f2', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 15:46:57 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:46:57 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:46:57 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Volume reschedule parameters: True retry: {'num_attempts': 2, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 15:46:57 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Flow 'volume_create_manager' (b81145b7-9801-4b0e-a85a-a429119190e0) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:46:57 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (ef2fc1f0-7641-4986-a834-18c3b09a924e) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:46:57 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Volume reschedule parameters: True retry: {'num_attempts': 2, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 15:46:57 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Flow 'volume_create_manager' (e02c3aee-6db0-49bb-9cf8-8f921800fe6f) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:46:57 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (4a96466f-ae69-4442-b1ee-51acc9db10b5) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:46:57 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (ef2fc1f0-7641-4986-a834-18c3b09a924e) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:44:38Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-CreateVolumesFromImageTest-Volume-1463431787',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=dfd745ed-b0c4-4e35-b77c-a911c90ce33e,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='75babbbca67f4035a48ee8ee816b18db',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:46:58Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:46:58Z,use_quota=True,user_id='d5299006144f4ee6817349348a489017',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:46:57 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (0da1616d-8a2c-46bb-aea1-8287b7af25cf) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:46:57 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (0da1616d-8a2c-46bb-aea1-8287b7af25cf) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:46:57 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (cedda478-02cf-4380-b073-53be7ff18594) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:46:57 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (cedda478-02cf-4380-b073-53be7ff18594) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': 'dfd745ed-b0c4-4e35-b77c-a911c90ce33e', 'volume_name': 'volume-dfd745ed-b0c4-4e35-b77c-a911c90ce33e', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:46:57 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (674f7152-823d-456b-a444-a8406df5280b) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:46:57 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (674f7152-823d-456b-a444-a8406df5280b) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:46:57 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (be26eb99-6fec-4d2f-ad74-1ffbc1521d7e) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:46:57 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Volume dfd745ed-b0c4-4e35-b77c-a911c90ce33e: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-dfd745ed-b0c4-4e35-b77c-a911c90ce33e', 'volume_size': 1} Apr 17 15:46:57 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] creating volume v_dfd745ed-b0c4-4e35-b77c-a911c90ce33e. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 15:46:57 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] create volume {'name': 'v_dfd745ed-b0c4-4e35-b77c-a911c90ce33e', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 15:46:57 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:46:57 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:46:57 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (4a96466f-ae69-4442-b1ee-51acc9db10b5) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:44:39Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-CreateVolumesFromImageTest-Volume-800604710',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=aea45c70-7c05-4146-847e-cbb777d3dfbe,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='75babbbca67f4035a48ee8ee816b18db',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:46:58Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:46:58Z,use_quota=True,user_id='d5299006144f4ee6817349348a489017',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:46:57 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (b5bdf986-2fce-4bd2-a19a-0e363a52af33) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:46:57 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (b5bdf986-2fce-4bd2-a19a-0e363a52af33) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:46:57 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (bea6314b-56a0-447d-87e7-f8136533d6e0) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:46:57 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (bea6314b-56a0-447d-87e7-f8136533d6e0) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': 'aea45c70-7c05-4146-847e-cbb777d3dfbe', 'volume_name': 'volume-aea45c70-7c05-4146-847e-cbb777d3dfbe', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:46:57 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (2f57f3d6-f99e-4921-8e08-fe8ce0153780) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:46:57 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (2f57f3d6-f99e-4921-8e08-fe8ce0153780) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:46:57 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (841e870b-81ae-4ee0-be56-d91eaaa8981b) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:46:57 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Volume aea45c70-7c05-4146-847e-cbb777d3dfbe: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-aea45c70-7c05-4146-847e-cbb777d3dfbe', 'volume_size': 1} Apr 17 15:46:57 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] creating volume v_aea45c70-7c05-4146-847e-cbb777d3dfbe. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 15:46:57 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] create volume {'name': 'v_aea45c70-7c05-4146-847e-cbb777d3dfbe', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 15:46:57 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:46:57 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:47:00 user cinder-volume[84696]: DEBUG cinder.utils [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 140.668(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:47:00 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:47:00 user cinder-volume[84696]: INFO cinder.message.api [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Creating message record for request_id = req-27d7b7b3-4675-424c-af48-cc2422ffc813 Apr 17 15:47:00 user cinder-volume[84696]: DEBUG cinder.utils [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.844(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:47:00 user cinder-volume[84696]: DEBUG cinder.utils [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_dfd745ed-b0c4-4e35-b77c-a911c90ce33e/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:47:00 user cinder-volume[84696]: DEBUG cinder.utils [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.867(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:47:00 user cinder-volume[84696]: DEBUG cinder.utils [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_fbbadf16-651a-47d5-b308-a37e56aa07f2/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:47:00 user cinder-volume[84696]: DEBUG cinder.utils [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.633(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:47:00 user cinder-volume[84696]: DEBUG cinder.utils [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:47:00 user cinder-volume[84696]: DEBUG cinder.utils [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.478(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:47:00 user cinder-volume[84696]: DEBUG cinder.utils [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:47:00 user cinder-volume[84696]: DEBUG cinder.utils [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 140.890(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:47:00 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:47:00 user cinder-volume[84696]: INFO cinder.message.api [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Creating message record for request_id = req-9b871d6d-58e3-4523-b328-b6ffa7aca14e Apr 17 15:47:00 user cinder-volume[84696]: DEBUG cinder.utils [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.843(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:47:00 user cinder-volume[84696]: DEBUG cinder.utils [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_aea45c70-7c05-4146-847e-cbb777d3dfbe/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:47:00 user cinder-volume[84696]: DEBUG cinder.utils [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.522(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:47:00 user cinder-volume[84696]: DEBUG cinder.utils [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:47:00 user cinder-volume[84696]: DEBUG cinder.utils [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 72.490(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:47:00 user cinder-volume[84696]: DEBUG cinder.utils [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:47:00 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 15:47:00 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 15:47:00 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (33b88de9-f6af-4460-b2a0-b6ca9b4da0ac) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 15:47:00 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 15:47:00 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:44:39Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-CreateVolumesFromImageTest-Volume-1478171265',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=ab5889f8-b6c7-4673-a666-f4b4e5ff5fd5,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='75babbbca67f4035a48ee8ee816b18db',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:44:39Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:44:39Z,use_quota=True,user_id='d5299006144f4ee6817349348a489017',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 15:47:00 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:44:39Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-CreateVolumesFromImageTest-Volume-1478171265',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=ab5889f8-b6c7-4673-a666-f4b4e5ff5fd5,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='75babbbca67f4035a48ee8ee816b18db',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:44:39Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:44:39Z,use_quota=True,user_id='d5299006144f4ee6817349348a489017',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(ab5889f8-b6c7-4673-a666-f4b4e5ff5fd5),volume_id=ab5889f8-b6c7-4673-a666-f4b4e5ff5fd5,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': 'ab5889f8-b6c7-4673-a666-f4b4e5ff5fd5', 'volume_name': 'volume-ab5889f8-b6c7-4673-a666-f4b4e5ff5fd5', 'volume_size': 1}} Apr 17 15:47:00 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'retry': {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']}, 'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': 'ab5889f8-b6c7-4673-a666-f4b4e5ff5fd5', 'volume': {'id': 'ab5889f8-b6c7-4673-a666-f4b4e5ff5fd5', '_name_id': None, 'ec2_id': None, 'user_id': 'd5299006144f4ee6817349348a489017', 'project_id': '75babbbca67f4035a48ee8ee816b18db', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-CreateVolumesFromImageTest-Volume-1478171265', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T15:44:39.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-ab5889f8-b6c7-4673-a666-f4b4e5ff5fd5', 'name_id': 'ab5889f8-b6c7-4673-a666-f4b4e5ff5fd5', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-CreateVolumesFromImageTest-Volume-1478171265', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '75babbbca67f4035a48ee8ee816b18db', 'qos_specs': None, 'reservations': ['a64639ba-df1a-40e6-bded-21cad01b745f', '4323d77e-0f64-483e-8433-0f03b9e38ddd', '7b2c692c-9fb9-4f7b-9b9c-94383941c4df', '45cf8435-1a25-41db-aac5-ff8bf8698729'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'd5299006144f4ee6817349348a489017', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-CreateVolumesFromImageTest-Volume-1478171265', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '75babbbca67f4035a48ee8ee816b18db', 'qos_specs': None, 'reservations': ['a64639ba-df1a-40e6-bded-21cad01b745f', '4323d77e-0f64-483e-8433-0f03b9e38ddd', '7b2c692c-9fb9-4f7b-9b9c-94383941c4df', '45cf8435-1a25-41db-aac5-ff8bf8698729'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'd5299006144f4ee6817349348a489017', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': 'd5299006144f4ee6817349348a489017', 'metadata': {}, 'qos_specs': None}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(ab5889f8-b6c7-4673-a666-f4b4e5ff5fd5),volume_id=ab5889f8-b6c7-4673-a666-f4b4e5ff5fd5,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:44:39Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-CreateVolumesFromImageTest-Volume-1478171265',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=ab5889f8-b6c7-4673-a666-f4b4e5ff5fd5,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='75babbbca67f4035a48ee8ee816b18db',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:44:39Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:44:39Z,use_quota=True,user_id='d5299006144f4ee6817349348a489017',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 15:47:00 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:44:39Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-CreateVolumesFromImageTest-Volume-1478171265',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=ab5889f8-b6c7-4673-a666-f4b4e5ff5fd5,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='75babbbca67f4035a48ee8ee816b18db',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:44:39Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:44:39Z,use_quota=True,user_id='d5299006144f4ee6817349348a489017',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:44:39Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-CreateVolumesFromImageTest-Volume-1478171265',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=ab5889f8-b6c7-4673-a666-f4b4e5ff5fd5,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='75babbbca67f4035a48ee8ee816b18db',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:44:39Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:44:39Z,use_quota=True,user_id='d5299006144f4ee6817349348a489017',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 15:47:00 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 15:47:00 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 15:47:00 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:47:00 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 15:47:00 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 15:47:00 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 15:47:00 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 15:47:00 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 15:47:00 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 15:47:00 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 15:47:00 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 15:47:00 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 15:47:00 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 15:47:00 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 15:47:00 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 15:47:00 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 15:47:00 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 15:47:00 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 15:47:00 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 15:47:00 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 15:47:00 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (33b88de9-f6af-4460-b2a0-b6ca9b4da0ac) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:47:00 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (33b88de9-f6af-4460-b2a0-b6ca9b4da0ac) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:47:00 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (a2033970-30aa-483e-908d-181c3c80d670) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:47:00 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (a2033970-30aa-483e-908d-181c3c80d670) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:47:00 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (aa4c31be-b709-4965-91f8-2f89acf992d8) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:47:00 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (aa4c31be-b709-4965-91f8-2f89acf992d8) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:47:00 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (34f4f4ff-2793-4469-b09b-4747e98a8a60) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:47:00 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Updating volume ab5889f8-b6c7-4673-a666-f4b4e5ff5fd5 with {'scheduled_at': datetime.datetime(2023, 4, 17, 15, 47, 0, 321054), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 15:47:00 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (6003413c-9a6d-4d1d-bc80-b0476dbdfa23) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 15:47:00 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 15:47:00 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:44:39Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-CreateVolumesFromImageTest-Volume-1776307206',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=13cc73df-7780-4934-8cbd-7f372b1cf57a,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='75babbbca67f4035a48ee8ee816b18db',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:44:39Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:44:39Z,use_quota=True,user_id='d5299006144f4ee6817349348a489017',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 15:47:00 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:44:39Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-CreateVolumesFromImageTest-Volume-1776307206',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=13cc73df-7780-4934-8cbd-7f372b1cf57a,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='75babbbca67f4035a48ee8ee816b18db',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:44:39Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:44:39Z,use_quota=True,user_id='d5299006144f4ee6817349348a489017',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(13cc73df-7780-4934-8cbd-7f372b1cf57a),volume_id=13cc73df-7780-4934-8cbd-7f372b1cf57a,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': '13cc73df-7780-4934-8cbd-7f372b1cf57a', 'volume_name': 'volume-13cc73df-7780-4934-8cbd-7f372b1cf57a', 'volume_size': 1}} Apr 17 15:47:00 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'retry': {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']}, 'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': '13cc73df-7780-4934-8cbd-7f372b1cf57a', 'volume': {'id': '13cc73df-7780-4934-8cbd-7f372b1cf57a', '_name_id': None, 'ec2_id': None, 'user_id': 'd5299006144f4ee6817349348a489017', 'project_id': '75babbbca67f4035a48ee8ee816b18db', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-CreateVolumesFromImageTest-Volume-1776307206', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T15:44:39.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-13cc73df-7780-4934-8cbd-7f372b1cf57a', 'name_id': '13cc73df-7780-4934-8cbd-7f372b1cf57a', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-CreateVolumesFromImageTest-Volume-1776307206', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '75babbbca67f4035a48ee8ee816b18db', 'qos_specs': None, 'reservations': ['a8c0981a-3509-44ee-887e-5a4c15f5fc31', 'be48fc25-1b54-4ebf-aef6-6988c7432c47', '45f69ba8-0c40-4500-b132-878446677716', '5143bd10-b0c0-4336-a02c-d9f5b822c37b'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'd5299006144f4ee6817349348a489017', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-CreateVolumesFromImageTest-Volume-1776307206', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '75babbbca67f4035a48ee8ee816b18db', 'qos_specs': None, 'reservations': ['a8c0981a-3509-44ee-887e-5a4c15f5fc31', 'be48fc25-1b54-4ebf-aef6-6988c7432c47', '45f69ba8-0c40-4500-b132-878446677716', '5143bd10-b0c0-4336-a02c-d9f5b822c37b'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'd5299006144f4ee6817349348a489017', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': 'd5299006144f4ee6817349348a489017', 'metadata': {}, 'qos_specs': None}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(13cc73df-7780-4934-8cbd-7f372b1cf57a),volume_id=13cc73df-7780-4934-8cbd-7f372b1cf57a,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:44:39Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-CreateVolumesFromImageTest-Volume-1776307206',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=13cc73df-7780-4934-8cbd-7f372b1cf57a,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='75babbbca67f4035a48ee8ee816b18db',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:44:39Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:44:39Z,use_quota=True,user_id='d5299006144f4ee6817349348a489017',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 15:47:00 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:44:39Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-CreateVolumesFromImageTest-Volume-1776307206',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=13cc73df-7780-4934-8cbd-7f372b1cf57a,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='75babbbca67f4035a48ee8ee816b18db',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:44:39Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:44:39Z,use_quota=True,user_id='d5299006144f4ee6817349348a489017',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:44:39Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-CreateVolumesFromImageTest-Volume-1776307206',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=13cc73df-7780-4934-8cbd-7f372b1cf57a,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='75babbbca67f4035a48ee8ee816b18db',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:44:39Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:44:39Z,use_quota=True,user_id='d5299006144f4ee6817349348a489017',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 15:47:00 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 15:47:00 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 15:47:00 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:47:00 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 15:47:00 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 15:47:00 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 15:47:00 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 15:47:00 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 15:47:00 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 15:47:00 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 15:47:00 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 15:47:00 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 15:47:00 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 15:47:00 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 15:47:00 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 15:47:00 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 15:47:00 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 15:47:00 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 15:47:00 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 15:47:00 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 15:47:00 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (6003413c-9a6d-4d1d-bc80-b0476dbdfa23) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:47:00 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (6003413c-9a6d-4d1d-bc80-b0476dbdfa23) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:47:00 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (9c125e0a-3123-49ab-b2f3-c2cf229225b5) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:47:00 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (9c125e0a-3123-49ab-b2f3-c2cf229225b5) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:47:00 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (7060b283-2751-438b-acca-fdd61100417f) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:47:00 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (7060b283-2751-438b-acca-fdd61100417f) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:47:00 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (182b3545-4d79-41e7-8994-9e2a802d1769) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:47:00 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Updating volume 13cc73df-7780-4934-8cbd-7f372b1cf57a with {'scheduled_at': datetime.datetime(2023, 4, 17, 15, 47, 0, 373275), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 15:47:00 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Volume ab5889f8-b6c7-4673-a666-f4b4e5ff5fd5: re-scheduling SchedulerAPI.create_volume attempt 1 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 15:47:00 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Volume 13cc73df-7780-4934-8cbd-7f372b1cf57a: re-scheduling SchedulerAPI.create_volume attempt 1 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 15:47:00 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Volume ab5889f8-b6c7-4673-a666-f4b4e5ff5fd5: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 15:47:00 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] deleating volume v_ab5889f8-b6c7-4673-a666-f4b4e5ff5fd5 {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 15:47:00 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] get snapshots for volume v_ab5889f8-b6c7-4673-a666-f4b4e5ff5fd5 {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 15:47:00 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_ab5889f8-b6c7-4673-a666-f4b4e5ff5fd5/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:47:00 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_ab5889f8-b6c7-4673-a666-f4b4e5ff5fd5/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:47:00 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Volume 13cc73df-7780-4934-8cbd-7f372b1cf57a: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 15:47:00 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] deleating volume v_13cc73df-7780-4934-8cbd-7f372b1cf57a {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 15:47:00 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] get snapshots for volume v_13cc73df-7780-4934-8cbd-7f372b1cf57a {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 15:47:00 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_13cc73df-7780-4934-8cbd-7f372b1cf57a/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:47:00 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_13cc73df-7780-4934-8cbd-7f372b1cf57a/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:47:00 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Volume reschedule parameters: True retry: {'num_attempts': 2, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 15:47:00 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Flow 'volume_create_manager' (36004948-f6dd-4fbc-80e8-70e49699879b) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:47:00 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (f5a58638-0fab-4fbe-83df-d20129322641) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:47:00 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (f5a58638-0fab-4fbe-83df-d20129322641) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:44:39Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-CreateVolumesFromImageTest-Volume-1478171265',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=ab5889f8-b6c7-4673-a666-f4b4e5ff5fd5,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='75babbbca67f4035a48ee8ee816b18db',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:47:01Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:47:01Z,use_quota=True,user_id='d5299006144f4ee6817349348a489017',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:47:00 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (ed61cc6c-4259-46de-8e73-3b5f80096571) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:47:00 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (ed61cc6c-4259-46de-8e73-3b5f80096571) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:47:00 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (b4424a4c-7b95-41aa-9426-18ccacfa2eee) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:47:00 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (b4424a4c-7b95-41aa-9426-18ccacfa2eee) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': 'ab5889f8-b6c7-4673-a666-f4b4e5ff5fd5', 'volume_name': 'volume-ab5889f8-b6c7-4673-a666-f4b4e5ff5fd5', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:47:00 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (323a2d66-260c-4d6f-bd28-1732b006d05d) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:47:00 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (323a2d66-260c-4d6f-bd28-1732b006d05d) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:47:00 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (79c3e531-6757-4e5a-ab4f-9b1fed099fd3) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:47:00 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Volume ab5889f8-b6c7-4673-a666-f4b4e5ff5fd5: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-ab5889f8-b6c7-4673-a666-f4b4e5ff5fd5', 'volume_size': 1} Apr 17 15:47:00 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] creating volume v_ab5889f8-b6c7-4673-a666-f4b4e5ff5fd5. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 15:47:00 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] create volume {'name': 'v_ab5889f8-b6c7-4673-a666-f4b4e5ff5fd5', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 15:47:00 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:47:00 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:47:00 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Volume reschedule parameters: True retry: {'num_attempts': 2, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 15:47:00 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Flow 'volume_create_manager' (93b410ca-1864-4f6c-9a57-32151032e466) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:47:00 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (b14fdc3c-2a4d-4b8c-bde6-41ea57565678) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:47:00 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (b14fdc3c-2a4d-4b8c-bde6-41ea57565678) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:44:39Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-CreateVolumesFromImageTest-Volume-1776307206',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=13cc73df-7780-4934-8cbd-7f372b1cf57a,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='75babbbca67f4035a48ee8ee816b18db',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:47:01Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:47:01Z,use_quota=True,user_id='d5299006144f4ee6817349348a489017',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:47:00 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (f9ed0951-88b0-4fb9-aaab-6be33f2e9366) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:47:00 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (f9ed0951-88b0-4fb9-aaab-6be33f2e9366) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:47:00 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (fc33ef75-005e-4d38-804e-1fcfa611f564) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:47:00 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (fc33ef75-005e-4d38-804e-1fcfa611f564) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': '13cc73df-7780-4934-8cbd-7f372b1cf57a', 'volume_name': 'volume-13cc73df-7780-4934-8cbd-7f372b1cf57a', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:47:00 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (72aede64-9af0-4067-8946-dad96767f88d) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:47:00 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (72aede64-9af0-4067-8946-dad96767f88d) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:47:00 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (62b34036-79c9-4679-bc37-81e483756477) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:47:00 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Volume 13cc73df-7780-4934-8cbd-7f372b1cf57a: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-13cc73df-7780-4934-8cbd-7f372b1cf57a', 'volume_size': 1} Apr 17 15:47:00 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] creating volume v_13cc73df-7780-4934-8cbd-7f372b1cf57a. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 15:47:00 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] create volume {'name': 'v_13cc73df-7780-4934-8cbd-7f372b1cf57a', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 15:47:00 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:47:00 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:47:03 user cinder-volume[84696]: DEBUG cinder.utils [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 140.640(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:47:03 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:47:03 user cinder-volume[84696]: INFO cinder.message.api [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Creating message record for request_id = req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 Apr 17 15:47:03 user cinder-volume[84696]: DEBUG cinder.utils [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.931(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:47:03 user cinder-volume[84696]: DEBUG cinder.utils [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_ab5889f8-b6c7-4673-a666-f4b4e5ff5fd5/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:47:03 user cinder-volume[84696]: DEBUG cinder.utils [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.923(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:47:03 user cinder-volume[84696]: DEBUG cinder.utils [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_dfd745ed-b0c4-4e35-b77c-a911c90ce33e/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:47:03 user cinder-volume[84696]: DEBUG cinder.utils [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.339(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:47:03 user cinder-volume[84696]: DEBUG cinder.utils [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:47:03 user cinder-volume[84696]: DEBUG cinder.utils [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.927(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:47:03 user cinder-volume[84696]: DEBUG cinder.utils [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_13cc73df-7780-4934-8cbd-7f372b1cf57a/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:47:03 user cinder-volume[84696]: DEBUG cinder.utils [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.440(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:47:03 user cinder-volume[84696]: DEBUG cinder.utils [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:47:03 user cinder-volume[84696]: DEBUG cinder.utils [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.951(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:47:03 user cinder-volume[84696]: DEBUG cinder.utils [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_fbbadf16-651a-47d5-b308-a37e56aa07f2/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:47:03 user cinder-volume[84696]: DEBUG cinder.utils [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.600(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:47:03 user cinder-volume[84696]: DEBUG cinder.utils [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:47:03 user cinder-volume[84696]: DEBUG cinder.utils [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.923(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:47:03 user cinder-volume[84696]: DEBUG cinder.utils [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_aea45c70-7c05-4146-847e-cbb777d3dfbe/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:47:03 user cinder-volume[84696]: DEBUG cinder.utils [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.720(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:47:03 user cinder-volume[84696]: DEBUG cinder.utils [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:47:03 user cinder-volume[84696]: WARNING urllib3.connectionpool [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Connection pool is full, discarding connection: 172.16.0.220. Connection pool size: 10: queue.Full Apr 17 15:47:03 user cinder-volume[84696]: DEBUG cinder.utils [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.569(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:47:03 user cinder-volume[84696]: DEBUG cinder.utils [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:47:03 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 15:47:03 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (f33783fd-fc7d-43a3-9b4e-f4003948152d) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 15:47:03 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 15:47:03 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:44:42Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-ServerRescueNegativeTestJSON-volume-1878278176',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=68f814b0-1291-4a12-ae65-dc0892203d36,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='9324c99bd4d040bdb5a7f1db782b2783',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:44:43Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:44:43Z,use_quota=True,user_id='8e3487d6b32d46beac556617050fd664',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 15:47:03 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:44:42Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-ServerRescueNegativeTestJSON-volume-1878278176',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=68f814b0-1291-4a12-ae65-dc0892203d36,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='9324c99bd4d040bdb5a7f1db782b2783',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:44:43Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:44:43Z,use_quota=True,user_id='8e3487d6b32d46beac556617050fd664',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(68f814b0-1291-4a12-ae65-dc0892203d36),volume_id=68f814b0-1291-4a12-ae65-dc0892203d36,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': '68f814b0-1291-4a12-ae65-dc0892203d36', 'volume_name': 'volume-68f814b0-1291-4a12-ae65-dc0892203d36', 'volume_size': 1}} Apr 17 15:47:03 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'retry': {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']}, 'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': '68f814b0-1291-4a12-ae65-dc0892203d36', 'volume': {'id': '68f814b0-1291-4a12-ae65-dc0892203d36', '_name_id': None, 'ec2_id': None, 'user_id': '8e3487d6b32d46beac556617050fd664', 'project_id': '9324c99bd4d040bdb5a7f1db782b2783', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-ServerRescueNegativeTestJSON-volume-1878278176', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T15:44:42.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-68f814b0-1291-4a12-ae65-dc0892203d36', 'name_id': '68f814b0-1291-4a12-ae65-dc0892203d36', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-ServerRescueNegativeTestJSON-volume-1878278176', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '9324c99bd4d040bdb5a7f1db782b2783', 'qos_specs': None, 'reservations': ['2f994b27-d987-48a4-ad61-f3ddbb20a152', 'b47baa6b-4689-4254-948c-f073028527b3', '59e249a4-01b6-44aa-a2e4-029e54867201', '148031dd-8a91-4041-9ef3-af4b5a029444'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': '8e3487d6b32d46beac556617050fd664', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-ServerRescueNegativeTestJSON-volume-1878278176', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '9324c99bd4d040bdb5a7f1db782b2783', 'qos_specs': None, 'reservations': ['2f994b27-d987-48a4-ad61-f3ddbb20a152', 'b47baa6b-4689-4254-948c-f073028527b3', '59e249a4-01b6-44aa-a2e4-029e54867201', '148031dd-8a91-4041-9ef3-af4b5a029444'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': '8e3487d6b32d46beac556617050fd664', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': '8e3487d6b32d46beac556617050fd664', 'metadata': {}, 'qos_specs': None}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(68f814b0-1291-4a12-ae65-dc0892203d36),volume_id=68f814b0-1291-4a12-ae65-dc0892203d36,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:44:42Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-ServerRescueNegativeTestJSON-volume-1878278176',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=68f814b0-1291-4a12-ae65-dc0892203d36,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='9324c99bd4d040bdb5a7f1db782b2783',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:44:43Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:44:43Z,use_quota=True,user_id='8e3487d6b32d46beac556617050fd664',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 15:47:03 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:44:42Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-ServerRescueNegativeTestJSON-volume-1878278176',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=68f814b0-1291-4a12-ae65-dc0892203d36,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='9324c99bd4d040bdb5a7f1db782b2783',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:44:43Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:44:43Z,use_quota=True,user_id='8e3487d6b32d46beac556617050fd664',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:44:42Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-ServerRescueNegativeTestJSON-volume-1878278176',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=68f814b0-1291-4a12-ae65-dc0892203d36,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='9324c99bd4d040bdb5a7f1db782b2783',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:44:43Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:44:43Z,use_quota=True,user_id='8e3487d6b32d46beac556617050fd664',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 15:47:03 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 15:47:03 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 15:47:03 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:47:03 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 15:47:03 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 15:47:03 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 15:47:03 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 15:47:03 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 15:47:03 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 15:47:03 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 15:47:03 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 15:47:03 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 15:47:03 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 15:47:03 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 15:47:03 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 15:47:03 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 15:47:03 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 15:47:03 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 15:47:03 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 15:47:03 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 15:47:03 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (f33783fd-fc7d-43a3-9b4e-f4003948152d) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:47:03 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (f33783fd-fc7d-43a3-9b4e-f4003948152d) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:47:03 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (1ba33adb-82eb-4638-9fdf-8a225a8b9494) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:47:03 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (1ba33adb-82eb-4638-9fdf-8a225a8b9494) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:47:03 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (86c42ee2-e424-4a87-b832-e2cad1fe942d) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:47:03 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (86c42ee2-e424-4a87-b832-e2cad1fe942d) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:47:03 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (78cd626e-1d4d-4838-8d0e-2d6b218f83b8) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:47:03 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Updating volume 68f814b0-1291-4a12-ae65-dc0892203d36 with {'scheduled_at': datetime.datetime(2023, 4, 17, 15, 47, 3, 420079), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 15:47:03 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Volume 68f814b0-1291-4a12-ae65-dc0892203d36: re-scheduling SchedulerAPI.create_volume attempt 1 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 15:47:03 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Volume 68f814b0-1291-4a12-ae65-dc0892203d36: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 15:47:03 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] deleating volume v_68f814b0-1291-4a12-ae65-dc0892203d36 {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 15:47:03 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] get snapshots for volume v_68f814b0-1291-4a12-ae65-dc0892203d36 {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 15:47:03 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_68f814b0-1291-4a12-ae65-dc0892203d36/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:47:03 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_68f814b0-1291-4a12-ae65-dc0892203d36/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:47:03 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Volume reschedule parameters: True retry: {'num_attempts': 2, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 15:47:03 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Flow 'volume_create_manager' (ddf23f63-d343-4c4e-ab15-b09de6fa5ae4) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:47:03 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (762d4ee0-8819-457a-b6c8-bad3a12c96c6) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:47:03 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (762d4ee0-8819-457a-b6c8-bad3a12c96c6) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:44:42Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-ServerRescueNegativeTestJSON-volume-1878278176',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=68f814b0-1291-4a12-ae65-dc0892203d36,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='9324c99bd4d040bdb5a7f1db782b2783',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:47:04Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:47:04Z,use_quota=True,user_id='8e3487d6b32d46beac556617050fd664',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:47:03 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (48c4ff00-053c-4a8b-8d01-bf8c90352d82) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:47:03 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (48c4ff00-053c-4a8b-8d01-bf8c90352d82) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:47:03 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (7cf39b0e-9ed4-4a29-98cc-a65170d845c4) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:47:03 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (7cf39b0e-9ed4-4a29-98cc-a65170d845c4) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': '68f814b0-1291-4a12-ae65-dc0892203d36', 'volume_name': 'volume-68f814b0-1291-4a12-ae65-dc0892203d36', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:47:03 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (7d1a95d8-4a1e-4f1c-b49a-9dfe89751746) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:47:03 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (7d1a95d8-4a1e-4f1c-b49a-9dfe89751746) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:47:03 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (da83f184-2ca5-4d3d-999d-9dff082efa0b) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:47:03 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Volume 68f814b0-1291-4a12-ae65-dc0892203d36: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-68f814b0-1291-4a12-ae65-dc0892203d36', 'volume_size': 1} Apr 17 15:47:03 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] creating volume v_68f814b0-1291-4a12-ae65-dc0892203d36. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 15:47:03 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] create volume {'name': 'v_68f814b0-1291-4a12-ae65-dc0892203d36', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 15:47:03 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:47:03 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:47:06 user cinder-volume[84696]: DEBUG cinder.utils [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.398(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:47:06 user cinder-volume[84696]: DEBUG cinder.utils [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:47:06 user cinder-volume[84696]: DEBUG cinder.utils [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.945(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:47:06 user cinder-volume[84696]: DEBUG cinder.utils [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_68f814b0-1291-4a12-ae65-dc0892203d36/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:47:06 user cinder-volume[84696]: DEBUG cinder.utils [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.996(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:47:06 user cinder-volume[84696]: DEBUG cinder.utils [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_ab5889f8-b6c7-4673-a666-f4b4e5ff5fd5/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:47:06 user cinder-volume[84696]: DEBUG cinder.utils [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.771(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:47:06 user cinder-volume[84696]: DEBUG cinder.utils [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:47:06 user cinder-volume[84696]: DEBUG cinder.utils [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.501(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:47:06 user cinder-volume[84696]: DEBUG cinder.utils [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:47:06 user cinder-volume[84696]: DEBUG cinder.utils [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.990(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:47:06 user cinder-volume[84696]: DEBUG cinder.utils [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_13cc73df-7780-4934-8cbd-7f372b1cf57a/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:47:09 user cinder-volume[84696]: DEBUG cinder.utils [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.077(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:47:09 user cinder-volume[84696]: DEBUG cinder.utils [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_fbbadf16-651a-47d5-b308-a37e56aa07f2/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:47:09 user cinder-volume[84696]: DEBUG cinder.utils [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.056(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:47:09 user cinder-volume[84696]: DEBUG cinder.utils [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_dfd745ed-b0c4-4e35-b77c-a911c90ce33e/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:47:09 user cinder-volume[84696]: DEBUG cinder.utils [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 72.422(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:47:09 user cinder-volume[84696]: DEBUG cinder.utils [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:47:09 user cinder-volume[84696]: DEBUG cinder.utils [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 11.846(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:47:09 user cinder-volume[84696]: DEBUG cinder.utils [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:47:09 user cinder-volume[84696]: DEBUG cinder.utils [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.843(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:47:09 user cinder-volume[84696]: DEBUG cinder.utils [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:47:09 user cinder-volume[84696]: DEBUG cinder.utils [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 11.731(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:47:09 user cinder-volume[84696]: DEBUG cinder.utils [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:47:09 user cinder-volume[84696]: DEBUG cinder.utils [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 72.599(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:47:09 user cinder-volume[84696]: DEBUG cinder.utils [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_441910f8-d29c-4c9e-bc5b-780ad228c85d/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:47:09 user cinder-volume[84696]: DEBUG cinder.utils [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 11.694(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:47:09 user cinder-volume[84696]: DEBUG cinder.utils [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:47:09 user cinder-volume[84696]: DEBUG cinder.utils [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.023(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:47:09 user cinder-volume[84696]: DEBUG cinder.utils [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_68f814b0-1291-4a12-ae65-dc0892203d36/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:47:09 user cinder-volume[84696]: DEBUG cinder.utils [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.056(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:47:09 user cinder-volume[84696]: DEBUG cinder.utils [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_aea45c70-7c05-4146-847e-cbb777d3dfbe/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:47:12 user cinder-volume[84696]: DEBUG cinder.utils [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.137(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:47:12 user cinder-volume[84696]: DEBUG cinder.utils [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_ab5889f8-b6c7-4673-a666-f4b4e5ff5fd5/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:47:12 user cinder-volume[84696]: DEBUG cinder.utils [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 11.544(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:47:12 user cinder-volume[84696]: DEBUG cinder.utils [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:47:12 user cinder-volume[84696]: DEBUG cinder.utils [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 11.643(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:47:12 user cinder-volume[84696]: DEBUG cinder.utils [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:47:12 user cinder-volume[84696]: DEBUG cinder.utils [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.134(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:47:12 user cinder-volume[84696]: DEBUG cinder.utils [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_13cc73df-7780-4934-8cbd-7f372b1cf57a/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:47:14 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:INSERT=4,cinder:SELECT=25,cinder:UPDATE=7 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:47:15 user cinder-volume[84696]: DEBUG cinder.utils [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 72.608(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:47:15 user cinder-volume[84696]: DEBUG cinder.utils [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_5c16bb82-43c5-4d91-9320-0e1f2bbec574/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:47:15 user cinder-volume[84696]: DEBUG cinder.utils [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.165(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:47:15 user cinder-volume[84696]: DEBUG cinder.utils [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_68f814b0-1291-4a12-ae65-dc0892203d36/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:47:15 user cinder-volume[84696]: DEBUG cinder.utils [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 72.371(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:47:15 user cinder-volume[84696]: DEBUG cinder.utils [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:47:15 user cinder-volume[84696]: DEBUG cinder.utils [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 11.991(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:47:15 user cinder-volume[84696]: DEBUG cinder.utils [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:47:15 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 140.932(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:47:15 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:47:15 user cinder-volume[84696]: ERROR oslo_service.periodic_task [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Error during VolumeManager.publish_service_capabilities: TypeError: 'NoneType' object is not subscriptable Apr 17 15:47:15 user cinder-volume[84696]: ERROR oslo_service.periodic_task Traceback (most recent call last): Apr 17 15:47:15 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/usr/local/lib/python3.10/dist-packages/oslo_service/periodic_task.py", line 216, in run_periodic_tasks Apr 17 15:47:15 user cinder-volume[84696]: ERROR oslo_service.periodic_task task(self, context) Apr 17 15:47:15 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/manager.py", line 2876, in publish_service_capabilities Apr 17 15:47:15 user cinder-volume[84696]: ERROR oslo_service.periodic_task self._report_driver_status(context) Apr 17 15:47:15 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/manager.py", line 2755, in _report_driver_status Apr 17 15:47:15 user cinder-volume[84696]: ERROR oslo_service.periodic_task volume_stats = get_stats() Apr 17 15:47:15 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/usr/local/lib/python3.10/dist-packages/oslo_utils/timeutils.py", line 319, in wrapper Apr 17 15:47:15 user cinder-volume[84696]: ERROR oslo_service.periodic_task result = func(*args, **kwargs) Apr 17 15:47:15 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/manager.py", line 2753, in get_stats Apr 17 15:47:15 user cinder-volume[84696]: ERROR oslo_service.periodic_task return self.driver.get_volume_stats(refresh=True) Apr 17 15:47:15 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/driver.py", line 666, in get_volume_stats Apr 17 15:47:15 user cinder-volume[84696]: ERROR oslo_service.periodic_task self._update_volume_stats() Apr 17 15:47:15 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 726, in _update_volume_stats Apr 17 15:47:15 user cinder-volume[84696]: ERROR oslo_service.periodic_task pool_stats = self.ra.get_pool_stats() Apr 17 15:47:15 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 897, in get_pool_stats Apr 17 15:47:15 user cinder-volume[84696]: ERROR oslo_service.periodic_task if not resp["error"] and resp["code"] == 200: Apr 17 15:47:15 user cinder-volume[84696]: ERROR oslo_service.periodic_task TypeError: 'NoneType' object is not subscriptable Apr 17 15:47:15 user cinder-volume[84696]: ERROR oslo_service.periodic_task Apr 17 15:47:15 user cinder-volume[84696]: WARNING oslo.service.loopingcall [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Function 'cinder.service.Service.periodic_tasks' run outlasted interval by 80.94 sec Apr 17 15:47:15 user cinder-volume[84696]: DEBUG oslo_service.periodic_task [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Running periodic task VolumeManager.publish_service_capabilities {{(pid=84696) run_periodic_tasks /usr/local/lib/python3.10/dist-packages/oslo_service/periodic_task.py:210}} Apr 17 15:47:15 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Updating volume stats {{(pid=84696) _update_volume_stats /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:724}} Apr 17 15:47:15 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Get pool Pool-0 fsprops {{(pid=84696) get_pool_stats /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:894}} Apr 17 15:47:15 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:47:15 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:47:18 user cinder-volume[84696]: DEBUG cinder.utils [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.059(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:47:18 user cinder-volume[84696]: DEBUG cinder.utils [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:47:18 user cinder-volume[84696]: DEBUG cinder.utils [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 20.905(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:47:18 user cinder-volume[84696]: DEBUG cinder.utils [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:47:18 user cinder-volume[84696]: DEBUG cinder.utils [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.268(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:47:18 user cinder-volume[84696]: DEBUG cinder.utils [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_aea45c70-7c05-4146-847e-cbb777d3dfbe/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:47:18 user cinder-volume[84696]: DEBUG cinder.utils [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 20.947(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:47:18 user cinder-volume[84696]: DEBUG cinder.utils [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:47:18 user cinder-volume[84696]: DEBUG cinder.utils [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.278(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:47:18 user cinder-volume[84696]: DEBUG cinder.utils [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_dfd745ed-b0c4-4e35-b77c-a911c90ce33e/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:47:18 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 3.064(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:47:18 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:47:18 user cinder-volume[84696]: DEBUG cinder.utils [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.304(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:47:18 user cinder-volume[84696]: DEBUG cinder.utils [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_fbbadf16-651a-47d5-b308-a37e56aa07f2/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:47:21 user cinder-volume[84696]: DEBUG cinder.utils [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 140.707(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:47:21 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_a50e9659-3e2d-4f04-b0ac-ae78b8f61fad/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:47:21 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (dfa120d7-a8b0-4768-af04-6169c215bc02) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 15:47:21 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (73dac3cd-a0e1-4f9c-9609-de8d355deb71) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:47:21 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (73dac3cd-a0e1-4f9c-9609-de8d355deb71) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:47:21 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Flow 'volume_create_manager' (c15a8b37-3406-4de3-92a0-bb523e3f9c64) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 15:47:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-e0039512-b662-485b-95d9-48de5642a75d tempest-CreateVolumesFromSourceVolumeTest-574348635 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 15:47:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 15:47:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 15:47:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 15:47:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 15:47:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 15:47:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 15:47:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 15:47:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 15:47:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 15:47:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 15:47:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 15:47:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 15:47:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 15:47:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 15:47:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 15:47:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 15:47:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 15:47:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 15:47:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 15:47:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 15:47:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 15:47:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 15:47:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 15:47:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:47:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 15:47:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 15:47:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 15:47:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 15:47:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 15:47:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 15:47:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 15:47:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 15:47:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 15:47:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 15:47:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 15:47:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 15:47:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 15:47:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 15:47:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 15:47:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 15:47:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 15:47:21 user cinder-volume[84696]: DEBUG cinder.utils [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 140.729(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:47:21 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_57070f23-6b40-47b9-b9b4-e0d77240f544/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:47:21 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (64ee410e-815a-4b3a-b712-8b21a6c707d8) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 15:47:21 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (21c2e379-aa21-4b1e-b467-1a7beae2f068) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:47:21 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (21c2e379-aa21-4b1e-b467-1a7beae2f068) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:47:21 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Flow 'volume_create_manager' (862dc8f8-09bd-4559-a622-c1bc1cdfcb4b) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 15:47:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-e55035f7-bed9-4dad-9cbc-b8b208b454bf tempest-VolumesActionsTest-707453245 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 15:47:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 15:47:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 15:47:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 15:47:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 15:47:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 15:47:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 15:47:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 15:47:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 15:47:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 15:47:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 15:47:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 15:47:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 15:47:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 15:47:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 15:47:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 15:47:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 15:47:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 15:47:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 15:47:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 15:47:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 15:47:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 15:47:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 15:47:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 15:47:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:47:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 15:47:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 15:47:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 15:47:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 15:47:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 15:47:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 15:47:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 15:47:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 15:47:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 15:47:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 15:47:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 15:47:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 15:47:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 15:47:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 15:47:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 15:47:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 15:47:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 15:47:21 user cinder-volume[84696]: DEBUG cinder.utils [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 20.892(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:47:21 user cinder-volume[84696]: DEBUG cinder.utils [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:47:21 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.165(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:47:21 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:47:21 user cinder-volume[84696]: DEBUG cinder.utils [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 20.796(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:47:21 user cinder-volume[84696]: DEBUG cinder.utils [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:47:21 user cinder-volume[84696]: DEBUG cinder.utils [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.383(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:47:21 user cinder-volume[84696]: DEBUG cinder.utils [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_13cc73df-7780-4934-8cbd-7f372b1cf57a/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:47:21 user cinder-volume[84696]: DEBUG cinder.utils [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.392(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:47:21 user cinder-volume[84696]: DEBUG cinder.utils [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_ab5889f8-b6c7-4673-a666-f4b4e5ff5fd5/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:47:24 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:INSERT=8,cinder:SELECT=34,cinder:UPDATE=13 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:47:24 user cinder-volume[84696]: DEBUG cinder.utils [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.199(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:47:24 user cinder-volume[84696]: DEBUG cinder.utils [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:47:24 user cinder-volume[84696]: DEBUG cinder.utils [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.376(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:47:24 user cinder-volume[84696]: DEBUG cinder.utils [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_68f814b0-1291-4a12-ae65-dc0892203d36/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:47:27 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.272(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:47:27 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:47:30 user cinder-volume[84696]: DEBUG cinder.utils [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 72.428(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:47:30 user cinder-volume[84696]: DEBUG cinder.utils [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:47:30 user cinder-volume[84696]: DEBUG cinder.utils [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 72.616(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:47:30 user cinder-volume[84696]: DEBUG cinder.utils [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_2c9714de-f8ea-4d1d-99f9-beb85126ad91/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:47:34 user cinder-volume[84696]: DEBUG cinder.utils [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 140.827(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:47:34 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_7ce815ce-9603-4fd6-8cc8-9fd798b2f99d/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:47:34 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (a6c87ba5-c4ef-454e-93ad-6732d94f5e50) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 15:47:34 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (af5be1b2-adf5-4c31-8f79-d06f505378fb) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:47:34 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (af5be1b2-adf5-4c31-8f79-d06f505378fb) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:47:34 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Flow 'volume_create_manager' (c4aa3ff4-61e1-444d-8ac9-70cbf23f00cf) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 15:47:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-f6a8cdbc-2e40-4ce2-aa75-ceab7e008d2b tempest-VolumesSnapshotTestJSON-1118243512 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 15:47:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 15:47:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 15:47:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 15:47:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 15:47:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 15:47:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 15:47:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 15:47:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 15:47:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 15:47:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 15:47:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 15:47:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 15:47:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 15:47:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 15:47:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 15:47:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 15:47:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 15:47:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 15:47:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 15:47:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 15:47:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 15:47:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 15:47:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 15:47:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:47:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 15:47:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 15:47:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 15:47:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 15:47:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 15:47:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 15:47:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 15:47:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 15:47:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 15:47:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 15:47:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 15:47:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 15:47:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 15:47:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 15:47:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 15:47:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 15:47:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 15:47:37 user cinder-volume[84696]: DEBUG cinder.utils [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.703(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:47:37 user cinder-volume[84696]: DEBUG cinder.utils [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_dfd745ed-b0c4-4e35-b77c-a911c90ce33e/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:47:37 user cinder-volume[84696]: DEBUG cinder.utils [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.696(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:47:37 user cinder-volume[84696]: DEBUG cinder.utils [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_aea45c70-7c05-4146-847e-cbb777d3dfbe/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:47:37 user cinder-volume[84696]: DEBUG cinder.utils [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.336(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:47:37 user cinder-volume[84696]: DEBUG cinder.utils [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:47:37 user cinder-volume[84696]: DEBUG cinder.utils [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.493(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:47:37 user cinder-volume[84696]: DEBUG cinder.utils [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:47:37 user cinder-volume[84696]: DEBUG cinder.utils [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.729(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:47:37 user cinder-volume[84696]: DEBUG cinder.utils [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_fbbadf16-651a-47d5-b308-a37e56aa07f2/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:47:37 user cinder-volume[84696]: DEBUG cinder.utils [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.378(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:47:37 user cinder-volume[84696]: DEBUG cinder.utils [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:47:37 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.494(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:47:37 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:47:40 user cinder-volume[84696]: DEBUG cinder.utils [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.190(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:47:40 user cinder-volume[84696]: DEBUG cinder.utils [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:47:40 user cinder-volume[84696]: DEBUG cinder.utils [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.786(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:47:40 user cinder-volume[84696]: DEBUG cinder.utils [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_ab5889f8-b6c7-4673-a666-f4b4e5ff5fd5/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:47:40 user cinder-volume[84696]: DEBUG cinder.utils [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.778(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:47:40 user cinder-volume[84696]: DEBUG cinder.utils [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_13cc73df-7780-4934-8cbd-7f372b1cf57a/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:47:40 user cinder-volume[84696]: DEBUG cinder.utils [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.290(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:47:40 user cinder-volume[84696]: DEBUG cinder.utils [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:47:43 user cinder-volume[84696]: DEBUG cinder.utils [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.631(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:47:43 user cinder-volume[84696]: DEBUG cinder.utils [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:47:43 user cinder-volume[84696]: DEBUG cinder.utils [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.808(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:47:43 user cinder-volume[84696]: DEBUG cinder.utils [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_68f814b0-1291-4a12-ae65-dc0892203d36/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:47:43 user cinder-volume[84696]: DEBUG cinder.utils [req-6ee9fe55-7b3a-42cf-906e-c25ad8813b64 req-9a73de69-ad8a-45b1-8113-a76d843c2607 tempest-TestShelveInstance-1683612675 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 139.205(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:47:43 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-6ee9fe55-7b3a-42cf-906e-c25ad8813b64 req-9a73de69-ad8a-45b1-8113-a76d843c2607 tempest-TestShelveInstance-1683612675 None] Geting None from DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:d895d11d-6880-4418-a9ad-feca813ed5d6/luns/v_d895d11d-6880-4418-a9ad-feca813ed5d6 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:47:43 user cinder-volume[84696]: DEBUG cinder.coordination [req-6ee9fe55-7b3a-42cf-906e-c25ad8813b64 req-9a73de69-ad8a-45b1-8113-a76d843c2607 tempest-TestShelveInstance-1683612675 None] Lock "cinder-d895d11d-6880-4418-a9ad-feca813ed5d6-delete_volume" released by "delete_volume" :: held 139.264s {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:207}} Apr 17 15:47:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [req-6ee9fe55-7b3a-42cf-906e-c25ad8813b64 req-9a73de69-ad8a-45b1-8113-a76d843c2607 tempest-TestShelveInstance-1683612675 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 15:47:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 15:47:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 15:47:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 15:47:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 15:47:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 15:47:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 15:47:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 15:47:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 203, in wrapper Apr 17 15:47:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server skip_clean = func(self, context, volume, *args, **kwargs) Apr 17 15:47:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 15:47:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 15:47:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/coordination.py", line 200, in _synchronized Apr 17 15:47:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return f(*a, **k) Apr 17 15:47:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 15:47:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 15:47:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 15:47:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 15:47:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 1035, in delete_volume Apr 17 15:47:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 15:47:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 15:47:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 15:47:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 15:47:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 15:47:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 1004, in delete_volume Apr 17 15:47:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.driver.remove_export(context, volume) Apr 17 15:47:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 720, in remove_export Apr 17 15:47:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self._remove_target_volume(volume) Apr 17 15:47:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 918, in _remove_target_volume Apr 17 15:47:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.detach_target_vol(target_name, jcom.vname(volume.id)) Apr 17 15:47:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 612, in detach_target_vol Apr 17 15:47:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if resp["code"] in (200, 201, 204): Apr 17 15:47:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 15:47:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 15:47:44 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=6,cinder:UPDATE=2 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:47:53 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 37.584(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:47:53 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:47:54 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=7,cinder:UPDATE=2,cinder:DELETE=1 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:47:59 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Volume reschedule parameters: True retry: {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 15:47:59 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Flow 'volume_create_manager' (b731ba4a-4006-467b-913f-f8ef3652da50) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:47:59 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (86278ba8-51c9-49d1-ad15-468a700d90e8) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:47:59 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (86278ba8-51c9-49d1-ad15-468a700d90e8) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:47:59Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-ServerRescueNegativeTestJSON-volume-7015736',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=cdedb820-e1c7-4b86-8286-43b3640b3a46,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='9324c99bd4d040bdb5a7f1db782b2783',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:47:59Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:47:59Z,use_quota=True,user_id='8e3487d6b32d46beac556617050fd664',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:47:59 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (427e4daa-48b1-40f1-b755-ad1fe82a7c04) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:47:59 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (427e4daa-48b1-40f1-b755-ad1fe82a7c04) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:47:59 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (873c476f-2a43-474c-96b7-18cbcbead4a4) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:47:59 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (873c476f-2a43-474c-96b7-18cbcbead4a4) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': 'cdedb820-e1c7-4b86-8286-43b3640b3a46', 'volume_name': 'volume-cdedb820-e1c7-4b86-8286-43b3640b3a46', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:47:59 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (dec98a2e-2088-4a24-9523-f2350094d5a9) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:47:59 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (dec98a2e-2088-4a24-9523-f2350094d5a9) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:47:59 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (9d5a63ca-cfb7-4210-880b-f0307fc50723) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:47:59 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Volume cdedb820-e1c7-4b86-8286-43b3640b3a46: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-cdedb820-e1c7-4b86-8286-43b3640b3a46', 'volume_size': 1} Apr 17 15:47:59 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] creating volume v_cdedb820-e1c7-4b86-8286-43b3640b3a46. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 15:47:59 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] create volume {'name': 'v_cdedb820-e1c7-4b86-8286-43b3640b3a46', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 15:47:59 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:47:59 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:48:00 user cinder-volume[84696]: DEBUG cinder.utils [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 0.811(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:48:00 user cinder-volume[84696]: DEBUG cinder.utils [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:48:03 user cinder-volume[84696]: DEBUG cinder.utils [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 3.882(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:48:03 user cinder-volume[84696]: DEBUG cinder.utils [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:48:03 user cinder-volume[84696]: DEBUG cinder.utils [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 138.671(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:48:03 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_a27d0ff3-5de9-4feb-82f7-d4571af88bb9/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:48:03 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (00be8551-29ba-4eb4-bc63-b825cf11ce67) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 15:48:03 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (7fc90935-2625-4047-95ba-0dbd29cfe095) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:48:03 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (7fc90935-2625-4047-95ba-0dbd29cfe095) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:48:03 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Flow 'volume_create_manager' (a8d98d40-62ca-452b-87fd-500fc7aa8bbc) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 15:48:03 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-52df28ef-59c1-47f6-831b-cf97d2a31df8 tempest-ServersNegativeTestJSON-1985071109 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 15:48:03 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 15:48:03 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 15:48:03 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 15:48:03 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 15:48:03 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 15:48:03 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 15:48:03 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 15:48:03 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 15:48:03 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 15:48:03 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 15:48:03 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 15:48:03 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 15:48:03 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 15:48:03 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 15:48:03 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 15:48:03 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 15:48:03 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 15:48:03 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 15:48:03 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 15:48:03 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 15:48:03 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 15:48:03 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 15:48:03 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 15:48:03 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:48:03 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 15:48:03 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 15:48:03 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 15:48:03 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 15:48:03 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 15:48:03 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 15:48:03 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 15:48:03 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 15:48:03 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 15:48:03 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 15:48:03 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 15:48:03 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 15:48:03 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 15:48:03 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 15:48:03 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 15:48:03 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 15:48:03 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 15:48:06 user cinder-volume[84696]: DEBUG cinder.utils [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 138.589(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:48:06 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:48:06 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (f5db5269-5ad7-4f1b-be93-2e00babdad94) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 15:48:06 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 15:48:06 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:45:47Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TestVolumeBootPattern-volume-origin-14805124',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=93a32a7c-4c6a-472a-8f91-9d34558ba232,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='c8a836b47d954cb9a7c1cd7c0f790426',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:45:48Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:45:48Z,use_quota=True,user_id='a80e753322224ec1a897598b5b9ed6ea',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 15:48:06 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:45:47Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TestVolumeBootPattern-volume-origin-14805124',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=93a32a7c-4c6a-472a-8f91-9d34558ba232,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='c8a836b47d954cb9a7c1cd7c0f790426',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:45:48Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:45:48Z,use_quota=True,user_id='a80e753322224ec1a897598b5b9ed6ea',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=b3f0dec1-7758-44a8-82e0-76ada8837428,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(93a32a7c-4c6a-472a-8f91-9d34558ba232),volume_id=93a32a7c-4c6a-472a-8f91-9d34558ba232,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'image', 'volume_id': '93a32a7c-4c6a-472a-8f91-9d34558ba232', 'volume_name': 'volume-93a32a7c-4c6a-472a-8f91-9d34558ba232', 'volume_size': 1, 'image_id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'image_location': (None, None), 'image_meta': {'name': 'cirros-0.5.2-x86_64-disk', 'disk_format': 'qcow2', 'container_format': 'bare', 'visibility': 'public', 'size': 16300544, 'virtual_size': 117440512, 'status': 'active', 'checksum': 'b874c39491a2377b8490f5f1e89761a4', 'protected': False, 'min_ram': 0, 'min_disk': 0, 'owner': 'a43c211b4ed542538799349489b1d884', 'os_hidden': False, 'os_hash_algo': 'sha512', 'os_hash_value': '6b813aa46bb90b4da216a4d19376593fa3f4fc7e617f03a92b7fe11e9a3981cbe8f0959dbebe36225e5f53dc4492341a4863cac4ed1ee0909f3fc78ef9c3e869', 'id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'created_at': datetime.datetime(2023, 4, 17, 15, 19, 53, tzinfo=datetime.timezone.utc), 'updated_at': datetime.datetime(2023, 4, 17, 15, 19, 55, tzinfo=datetime.timezone.utc), 'tags': [], 'file': '/v2/images/b3f0dec1-7758-44a8-82e0-76ada8837428/file', 'properties': {'hw_rng_model': 'virtio', 'owner_specified.openstack.md5': '', 'owner_specified.openstack.object': 'images/cirros-0.5.2-x86_64-disk', 'owner_specified.openstack.sha256': ''}}, 'image_service': }} Apr 17 15:48:06 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'retry': {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']}, 'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'snapshot_id': None, 'source_volid': None, 'volume_id': '93a32a7c-4c6a-472a-8f91-9d34558ba232', 'volume': {'id': '93a32a7c-4c6a-472a-8f91-9d34558ba232', '_name_id': None, 'ec2_id': None, 'user_id': 'a80e753322224ec1a897598b5b9ed6ea', 'project_id': 'c8a836b47d954cb9a7c1cd7c0f790426', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-TestVolumeBootPattern-volume-origin-14805124', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T15:45:47.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-93a32a7c-4c6a-472a-8f91-9d34558ba232', 'name_id': '93a32a7c-4c6a-472a-8f91-9d34558ba232', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-TestVolumeBootPattern-volume-origin-14805124', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': 'c8a836b47d954cb9a7c1cd7c0f790426', 'qos_specs': None, 'reservations': ['c48dcd83-2285-48af-a8d0-ff11edc29cb9', 'd1463595-875c-453d-8e19-7c670de227bd', '59caf45d-ddd8-48db-b7c0-c2b483358ca9', 'deb0cce9-9fd9-4e11-8704-901c42ef169b'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'a80e753322224ec1a897598b5b9ed6ea', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-TestVolumeBootPattern-volume-origin-14805124', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': 'c8a836b47d954cb9a7c1cd7c0f790426', 'qos_specs': None, 'reservations': ['c48dcd83-2285-48af-a8d0-ff11edc29cb9', 'd1463595-875c-453d-8e19-7c670de227bd', '59caf45d-ddd8-48db-b7c0-c2b483358ca9', 'deb0cce9-9fd9-4e11-8704-901c42ef169b'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'a80e753322224ec1a897598b5b9ed6ea', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': 'a80e753322224ec1a897598b5b9ed6ea', 'metadata': {}, 'qos_specs': None}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=b3f0dec1-7758-44a8-82e0-76ada8837428,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(93a32a7c-4c6a-472a-8f91-9d34558ba232),volume_id=93a32a7c-4c6a-472a-8f91-9d34558ba232,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:45:47Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TestVolumeBootPattern-volume-origin-14805124',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=93a32a7c-4c6a-472a-8f91-9d34558ba232,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='c8a836b47d954cb9a7c1cd7c0f790426',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:45:48Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:45:48Z,use_quota=True,user_id='a80e753322224ec1a897598b5b9ed6ea',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 15:48:06 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:45:47Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TestVolumeBootPattern-volume-origin-14805124',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=93a32a7c-4c6a-472a-8f91-9d34558ba232,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='c8a836b47d954cb9a7c1cd7c0f790426',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:45:48Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:45:48Z,use_quota=True,user_id='a80e753322224ec1a897598b5b9ed6ea',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:45:47Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TestVolumeBootPattern-volume-origin-14805124',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=93a32a7c-4c6a-472a-8f91-9d34558ba232,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='c8a836b47d954cb9a7c1cd7c0f790426',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:45:48Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:45:48Z,use_quota=True,user_id='a80e753322224ec1a897598b5b9ed6ea',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 15:48:06 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 15:48:06 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 15:48:06 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:48:06 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 15:48:06 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1237, in execute Apr 17 15:48:06 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_from_image(context, Apr 17 15:48:06 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/utils.py", line 670, in _wrapper Apr 17 15:48:06 user cinder-volume[84696]: ERROR cinder.volume.manager return r(f, *args, **kwargs) Apr 17 15:48:06 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/tenacity/__init__.py", line 423, in __call__ Apr 17 15:48:06 user cinder-volume[84696]: ERROR cinder.volume.manager do = self.iter(retry_state=retry_state) Apr 17 15:48:06 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/tenacity/__init__.py", line 360, in iter Apr 17 15:48:06 user cinder-volume[84696]: ERROR cinder.volume.manager return fut.result() Apr 17 15:48:06 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/lib/python3.10/concurrent/futures/_base.py", line 451, in result Apr 17 15:48:06 user cinder-volume[84696]: ERROR cinder.volume.manager return self.__get_result() Apr 17 15:48:06 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result Apr 17 15:48:06 user cinder-volume[84696]: ERROR cinder.volume.manager raise self._exception Apr 17 15:48:06 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/tenacity/__init__.py", line 426, in __call__ Apr 17 15:48:06 user cinder-volume[84696]: ERROR cinder.volume.manager result = fn(*args, **kwargs) Apr 17 15:48:06 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1125, in _create_from_image Apr 17 15:48:06 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_from_image_cache_or_download( Apr 17 15:48:06 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 944, in _create_from_image_cache_or_download Apr 17 15:48:06 user cinder-volume[84696]: ERROR cinder.volume.manager model_update, cloned = self._create_from_image_cache( Apr 17 15:48:06 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 843, in _create_from_image_cache Apr 17 15:48:06 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_from_source_volume( Apr 17 15:48:06 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 659, in _create_from_source_volume Apr 17 15:48:06 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self.driver.create_cloned_volume(volume, srcvol_ref) Apr 17 15:48:06 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 525, in create_cloned_volume Apr 17 15:48:06 user cinder-volume[84696]: ERROR cinder.volume.manager self._clone_object(vname, cvname) Apr 17 15:48:06 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 461, in _clone_object Apr 17 15:48:06 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_snapshot(oname, coname) Apr 17 15:48:06 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 639, in create_snapshot Apr 17 15:48:06 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201, 204): Apr 17 15:48:06 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 15:48:06 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 15:48:06 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (f5db5269-5ad7-4f1b-be93-2e00babdad94) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:48:06 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (f5db5269-5ad7-4f1b-be93-2e00babdad94) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:48:06 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (a07076e3-7672-44e5-9253-90d6448b71bc) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:48:06 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (a07076e3-7672-44e5-9253-90d6448b71bc) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:48:06 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (a85a3b7a-915c-43f4-8b75-f1fb81ad62ee) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:48:06 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (a85a3b7a-915c-43f4-8b75-f1fb81ad62ee) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:48:06 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (fd6372ff-955f-45b3-a7c4-551f96bdd404) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:48:06 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Updating volume 93a32a7c-4c6a-472a-8f91-9d34558ba232 with {'scheduled_at': datetime.datetime(2023, 4, 17, 15, 48, 6, 415716), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 15:48:06 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Volume 93a32a7c-4c6a-472a-8f91-9d34558ba232: re-scheduling SchedulerAPI.create_volume attempt 1 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 15:48:06 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Volume 93a32a7c-4c6a-472a-8f91-9d34558ba232: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 15:48:06 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] deleating volume v_93a32a7c-4c6a-472a-8f91-9d34558ba232 {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 15:48:06 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] get snapshots for volume v_93a32a7c-4c6a-472a-8f91-9d34558ba232 {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 15:48:06 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_93a32a7c-4c6a-472a-8f91-9d34558ba232/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:48:06 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_93a32a7c-4c6a-472a-8f91-9d34558ba232/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:48:06 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Volume reschedule parameters: True retry: {'num_attempts': 2, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 15:48:06 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Flow 'volume_create_manager' (9854c8a1-2cb7-4529-964f-3d4607c1552f) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:48:06 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (66bdcfbf-3b5a-4eeb-8a0e-92299cac60a6) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:48:06 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (66bdcfbf-3b5a-4eeb-8a0e-92299cac60a6) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:45:47Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TestVolumeBootPattern-volume-origin-14805124',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=93a32a7c-4c6a-472a-8f91-9d34558ba232,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='c8a836b47d954cb9a7c1cd7c0f790426',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:48:07Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:48:07Z,use_quota=True,user_id='a80e753322224ec1a897598b5b9ed6ea',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:48:06 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (5a647616-587e-451f-ad41-772bf631bc61) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:48:06 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (5a647616-587e-451f-ad41-772bf631bc61) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:48:06 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (e8421914-d534-4f63-acac-2d12132d3230) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:48:06 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (e8421914-d534-4f63-acac-2d12132d3230) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'image', 'volume_id': '93a32a7c-4c6a-472a-8f91-9d34558ba232', 'volume_name': 'volume-93a32a7c-4c6a-472a-8f91-9d34558ba232', 'volume_size': 1, 'image_id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'image_location': (None, None), 'image_meta': {'name': 'cirros-0.5.2-x86_64-disk', 'disk_format': 'qcow2', 'container_format': 'bare', 'visibility': 'public', 'size': 16300544, 'virtual_size': 117440512, 'status': 'active', 'checksum': 'b874c39491a2377b8490f5f1e89761a4', 'protected': False, 'min_ram': 0, 'min_disk': 0, 'owner': 'a43c211b4ed542538799349489b1d884', 'os_hidden': False, 'os_hash_algo': 'sha512', 'os_hash_value': '6b813aa46bb90b4da216a4d19376593fa3f4fc7e617f03a92b7fe11e9a3981cbe8f0959dbebe36225e5f53dc4492341a4863cac4ed1ee0909f3fc78ef9c3e869', 'id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'created_at': datetime.datetime(2023, 4, 17, 15, 19, 53, tzinfo=datetime.timezone.utc), 'updated_at': datetime.datetime(2023, 4, 17, 15, 19, 55, tzinfo=datetime.timezone.utc), 'tags': [], 'file': '/v2/images/b3f0dec1-7758-44a8-82e0-76ada8837428/file', 'properties': {'hw_rng_model': 'virtio', 'owner_specified.openstack.md5': '', 'owner_specified.openstack.object': 'images/cirros-0.5.2-x86_64-disk', 'owner_specified.openstack.sha256': ''}}, 'image_service': }' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:48:06 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (81e5f561-0a54-4025-b482-e64744649b65) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:48:06 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (81e5f561-0a54-4025-b482-e64744649b65) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:48:06 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (ffd4194b-0692-4180-b2f2-0e5ec1eae1ed) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:48:06 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Volume 93a32a7c-4c6a-472a-8f91-9d34558ba232: being created as image with specification: {'status': 'creating', 'volume_name': 'volume-93a32a7c-4c6a-472a-8f91-9d34558ba232', 'volume_size': 1, 'image_id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'image_location': (None, None), 'image_meta': {'name': 'cirros-0.5.2-x86_64-disk', 'disk_format': 'qcow2', 'container_format': 'bare', 'visibility': 'public', 'size': 16300544, 'virtual_size': 117440512, 'status': 'active', 'checksum': 'b874c39491a2377b8490f5f1e89761a4', 'protected': False, 'min_ram': 0, 'min_disk': 0, 'owner': 'a43c211b4ed542538799349489b1d884', 'os_hidden': False, 'os_hash_algo': 'sha512', 'os_hash_value': '6b813aa46bb90b4da216a4d19376593fa3f4fc7e617f03a92b7fe11e9a3981cbe8f0959dbebe36225e5f53dc4492341a4863cac4ed1ee0909f3fc78ef9c3e869', 'id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'created_at': datetime.datetime(2023, 4, 17, 15, 19, 53, tzinfo=datetime.timezone.utc), 'updated_at': datetime.datetime(2023, 4, 17, 15, 19, 55, tzinfo=datetime.timezone.utc), 'tags': [], 'file': '/v2/images/b3f0dec1-7758-44a8-82e0-76ada8837428/file', 'properties': {'hw_rng_model': 'virtio', 'owner_specified.openstack.md5': '', 'owner_specified.openstack.object': 'images/cirros-0.5.2-x86_64-disk', 'owner_specified.openstack.sha256': ''}}, 'image_service': } Apr 17 15:48:06 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Cloning 93a32a7c-4c6a-472a-8f91-9d34558ba232 from image b3f0dec1-7758-44a8-82e0-76ada8837428 at location (None, None). {{(pid=84696) _create_from_image /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1071}} Apr 17 15:48:06 user cinder-volume[84696]: DEBUG cinder.coordination [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Acquiring lock "cinder-b3f0dec1-7758-44a8-82e0-76ada8837428" by "_prepare_image_cache_entry" {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:192}} Apr 17 15:48:06 user cinder-volume[84696]: DEBUG cinder.coordination [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Lock "cinder-b3f0dec1-7758-44a8-82e0-76ada8837428" acquired by "_prepare_image_cache_entry" :: waited 0.012s {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:196}} Apr 17 15:48:06 user cinder-volume[84696]: DEBUG cinder.image.cache [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Found image-volume cache entry: {'id': 1, 'image_id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'volume_id': '006f20ae-5306-44fe-afdf-e433f79284e8', 'host': 'user@joviandss-2#joviandss-2', 'size': 1, 'image_updated_at': datetime.datetime(2023, 4, 17, 15, 19, 55), 'last_used': datetime.datetime(2023, 4, 17, 15, 48, 6, 731555)}. {{(pid=84696) get_entry /opt/stack/cinder/cinder/image/cache.py:76}} Apr 17 15:48:06 user cinder-volume[84696]: DEBUG cinder.image.cache [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Image-volume cache entry image_update_at = 2023-04-17 15:19:55+00:00, requested image updated_at = 2023-04-17 15:19:55+00:00. {{(pid=84696) _should_update_entry /opt/stack/cinder/cinder/image/cache.py:259}} Apr 17 15:48:06 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Found cache entry for image = b3f0dec1-7758-44a8-82e0-76ada8837428 on host user@joviandss-2#joviandss-2. {{(pid=84696) _prepare_image_cache_entry /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:886}} Apr 17 15:48:06 user cinder-volume[84696]: DEBUG cinder.coordination [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Lock "cinder-b3f0dec1-7758-44a8-82e0-76ada8837428" released by "_prepare_image_cache_entry" :: held 0.017s {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:207}} Apr 17 15:48:06 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Attempting to retrieve cache entry for image = b3f0dec1-7758-44a8-82e0-76ada8837428 on host user@joviandss-2#joviandss-2. {{(pid=84696) _create_from_image_cache /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:827}} Apr 17 15:48:06 user cinder-volume[84696]: DEBUG cinder.image.cache [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Found image-volume cache entry: {'id': 1, 'image_id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'volume_id': '006f20ae-5306-44fe-afdf-e433f79284e8', 'host': 'user@joviandss-2#joviandss-2', 'size': 1, 'image_updated_at': datetime.datetime(2023, 4, 17, 15, 19, 55), 'last_used': datetime.datetime(2023, 4, 17, 15, 48, 6, 747229)}. {{(pid=84696) get_entry /opt/stack/cinder/cinder/image/cache.py:76}} Apr 17 15:48:06 user cinder-volume[84696]: DEBUG cinder.image.cache [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Image-volume cache entry image_update_at = 2023-04-17 15:19:55+00:00, requested image updated_at = 2023-04-17 15:19:55+00:00. {{(pid=84696) _should_update_entry /opt/stack/cinder/cinder/image/cache.py:259}} Apr 17 15:48:06 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Creating from source image-volume 006f20ae-5306-44fe-afdf-e433f79284e8 {{(pid=84696) _create_from_image_cache /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:841}} Apr 17 15:48:06 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] cloned volume 006f20ae-5306-44fe-afdf-e433f79284e8 to 93a32a7c-4c6a-472a-8f91-9d34558ba232 {{(pid=84696) create_cloned_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:521}} Apr 17 15:48:06 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] cloning v_006f20ae-5306-44fe-afdf-e433f79284e8 to v_93a32a7c-4c6a-472a-8f91-9d34558ba232 {{(pid=84696) _clone_object /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:456}} Apr 17 15:48:06 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] create snapshot v_93a32a7c-4c6a-472a-8f91-9d34558ba232 {{(pid=84696) create_snapshot /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:635}} Apr 17 15:48:06 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:48:06 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:48:09 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-778c62be-8963-4e6d-97f1-f456b759ce16 req-d9cf97f7-129f-4cb2-9bc2-f51b1339159a tempest-AttachVolumeShelveTestJSON-1745617049 None] terminate connection for 13bf568b-585c-4556-ab78-e9930ad50214 {{(pid=84696) terminate_connection /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:1032}} Apr 17 15:48:09 user cinder-volume[84696]: INFO cinder.volume.manager [req-778c62be-8963-4e6d-97f1-f456b759ce16 req-d9cf97f7-129f-4cb2-9bc2-f51b1339159a tempest-AttachVolumeShelveTestJSON-1745617049 None] Terminate volume connection completed successfully. Apr 17 15:48:09 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-778c62be-8963-4e6d-97f1-f456b759ce16 req-d9cf97f7-129f-4cb2-9bc2-f51b1339159a tempest-AttachVolumeShelveTestJSON-1745617049 None] Deleting attachment 1634a7d0-3313-448e-9888-8de015f268af. {{(pid=84696) attachment_delete /opt/stack/cinder/cinder/volume/manager.py:5006}} Apr 17 15:48:09 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-778c62be-8963-4e6d-97f1-f456b759ce16 req-d9cf97f7-129f-4cb2-9bc2-f51b1339159a tempest-AttachVolumeShelveTestJSON-1745617049 None] remove_export for volume: 13bf568b-585c-4556-ab78-e9930ad50214. {{(pid=84696) remove_export /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:718}} Apr 17 15:48:09 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-778c62be-8963-4e6d-97f1-f456b759ce16 req-d9cf97f7-129f-4cb2-9bc2-f51b1339159a tempest-AttachVolumeShelveTestJSON-1745617049 None] remove export {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:912}} Apr 17 15:48:09 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-778c62be-8963-4e6d-97f1-f456b759ce16 req-d9cf97f7-129f-4cb2-9bc2-f51b1339159a tempest-AttachVolumeShelveTestJSON-1745617049 None] detach volume:Volume(_name_id=None,admin_metadata=,attach_status='attached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:25:53Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-AttachVolumeShelveTestJSON-volume-1457837636',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=13bf568b-585c-4556-ab78-e9930ad50214,launched_at=2023-04-17T15:25:55Z,metadata=,migration_status=None,multiattach=False,previous_status=None,project_id='214a0f233f114d7aa22e64048a837c0b',provider_auth='CHAP cajgFQvq cTCKdDR0kw38ze',provider_geometry=None,provider_id=None,provider_location='172.16.0.220:3260,1 iqn.2010-10.org.openstack:13bf568b-585c-4556-ab78-e9930ad50214 0',replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:25:53Z,service_uuid='e628352e-c4cf-40eb-8389-cce40663026a',shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='in-use',terminated_at=None,updated_at=2023-04-17T15:31:43Z,use_quota=True,user_id='83b9d77de46143d8b9277a55f5ad4d72',volume_attachment=,volume_type=,volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c) from target:iqn.2010-10.org.openstack:13bf568b-585c-4556-ab78-e9930ad50214. {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:913}} Apr 17 15:48:09 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [req-778c62be-8963-4e6d-97f1-f456b759ce16 req-d9cf97f7-129f-4cb2-9bc2-f51b1339159a tempest-AttachVolumeShelveTestJSON-1745617049 None] detach volume v_13bf568b-585c-4556-ab78-e9930ad50214 from target iqn.2010-10.org.openstack:13bf568b-585c-4556-ab78-e9930ad50214 {{(pid=84696) detach_target_vol /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:606}} Apr 17 15:48:09 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-778c62be-8963-4e6d-97f1-f456b759ce16 req-d9cf97f7-129f-4cb2-9bc2-f51b1339159a tempest-AttachVolumeShelveTestJSON-1745617049 None] Sending pool request DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:13bf568b-585c-4556-ab78-e9930ad50214/luns/v_13bf568b-585c-4556-ab78-e9930ad50214 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:48:09 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-778c62be-8963-4e6d-97f1-f456b759ce16 req-d9cf97f7-129f-4cb2-9bc2-f51b1339159a tempest-AttachVolumeShelveTestJSON-1745617049 None] Sending DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:13bf568b-585c-4556-ab78-e9930ad50214/luns/v_13bf568b-585c-4556-ab78-e9930ad50214 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:48:09 user cinder-volume[84696]: DEBUG cinder.utils [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 71.698(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:48:09 user cinder-volume[84696]: DEBUG cinder.utils [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:48:09 user cinder-volume[84696]: DEBUG cinder.utils [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.683(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:48:09 user cinder-volume[84696]: DEBUG cinder.utils [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:48:09 user cinder-volume[84696]: DEBUG cinder.utils [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 72.026(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:48:09 user cinder-volume[84696]: DEBUG cinder.utils [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_aea45c70-7c05-4146-847e-cbb777d3dfbe/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:48:09 user cinder-volume[84696]: DEBUG cinder.utils [req-778c62be-8963-4e6d-97f1-f456b759ce16 req-d9cf97f7-129f-4cb2-9bc2-f51b1339159a tempest-AttachVolumeShelveTestJSON-1745617049 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 0.438(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:48:09 user cinder-volume[84696]: DEBUG cinder.utils [req-778c62be-8963-4e6d-97f1-f456b759ce16 req-d9cf97f7-129f-4cb2-9bc2-f51b1339159a tempest-AttachVolumeShelveTestJSON-1745617049 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:13bf568b-585c-4556-ab78-e9930ad50214/luns/v_13bf568b-585c-4556-ab78-e9930ad50214 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:48:09 user cinder-volume[84696]: DEBUG cinder.utils [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 10.041(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:48:09 user cinder-volume[84696]: DEBUG cinder.utils [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:48:09 user cinder-volume[84696]: DEBUG cinder.utils [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 71.825(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:48:09 user cinder-volume[84696]: DEBUG cinder.utils [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:48:09 user cinder-volume[84696]: DEBUG cinder.utils [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 3.006(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:48:09 user cinder-volume[84696]: DEBUG cinder.utils [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_93a32a7c-4c6a-472a-8f91-9d34558ba232/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:48:09 user cinder-volume[84696]: DEBUG cinder.utils [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 71.672(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:48:09 user cinder-volume[84696]: DEBUG cinder.utils [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:48:09 user cinder-volume[84696]: DEBUG cinder.utils [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 72.042(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:48:09 user cinder-volume[84696]: DEBUG cinder.utils [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_dfd745ed-b0c4-4e35-b77c-a911c90ce33e/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:48:09 user cinder-volume[84696]: DEBUG cinder.utils [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 72.065(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:48:09 user cinder-volume[84696]: DEBUG cinder.utils [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_fbbadf16-651a-47d5-b308-a37e56aa07f2/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:48:12 user cinder-volume[84696]: DEBUG cinder.utils [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.746(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:48:12 user cinder-volume[84696]: DEBUG cinder.utils [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:48:12 user cinder-volume[84696]: DEBUG cinder.utils [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 71.511(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:48:12 user cinder-volume[84696]: DEBUG cinder.utils [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:48:12 user cinder-volume[84696]: DEBUG cinder.utils [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.064(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:48:12 user cinder-volume[84696]: DEBUG cinder.utils [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_93a32a7c-4c6a-472a-8f91-9d34558ba232/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:48:12 user cinder-volume[84696]: DEBUG cinder.utils [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 72.108(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:48:12 user cinder-volume[84696]: DEBUG cinder.utils [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_ab5889f8-b6c7-4673-a666-f4b4e5ff5fd5/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:48:12 user cinder-volume[84696]: DEBUG cinder.utils [req-778c62be-8963-4e6d-97f1-f456b759ce16 req-d9cf97f7-129f-4cb2-9bc2-f51b1339159a tempest-AttachVolumeShelveTestJSON-1745617049 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 3.502(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:48:12 user cinder-volume[84696]: DEBUG cinder.utils [req-778c62be-8963-4e6d-97f1-f456b759ce16 req-d9cf97f7-129f-4cb2-9bc2-f51b1339159a tempest-AttachVolumeShelveTestJSON-1745617049 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:13bf568b-585c-4556-ab78-e9930ad50214/luns/v_13bf568b-585c-4556-ab78-e9930ad50214 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:48:12 user cinder-volume[84696]: DEBUG cinder.utils [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 71.612(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:48:12 user cinder-volume[84696]: DEBUG cinder.utils [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:48:12 user cinder-volume[84696]: DEBUG cinder.utils [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 72.102(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:48:12 user cinder-volume[84696]: DEBUG cinder.utils [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_13cc73df-7780-4934-8cbd-7f372b1cf57a/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:48:14 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=15,cinder:UPDATE=5,cinder:INSERT=1 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:48:15 user cinder-volume[84696]: DEBUG cinder.utils [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 138.704(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:48:15 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_441910f8-d29c-4c9e-bc5b-780ad228c85d/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:48:15 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (4d0ba699-1018-4d1d-a6d5-05df49808c59) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 15:48:15 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (7b2b9368-64b3-4560-ba5b-59fd9f0b0863) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:48:15 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (7b2b9368-64b3-4560-ba5b-59fd9f0b0863) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:48:15 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Flow 'volume_create_manager' (ec4c4cd2-651f-4aaf-b76f-a389f625dcdc) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 15:48:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 15:48:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 15:48:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 15:48:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 15:48:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 15:48:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 15:48:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 15:48:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 15:48:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 15:48:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 15:48:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 15:48:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 15:48:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 15:48:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 15:48:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 15:48:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 15:48:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 15:48:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 15:48:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 15:48:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 15:48:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 15:48:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 15:48:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 15:48:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 15:48:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:48:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 15:48:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 15:48:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 15:48:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 15:48:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 15:48:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 15:48:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 15:48:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 15:48:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 15:48:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 15:48:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 15:48:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 15:48:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 15:48:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 15:48:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 15:48:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 15:48:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 15:48:15 user cinder-volume[84696]: DEBUG cinder.utils [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 71.974(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:48:15 user cinder-volume[84696]: DEBUG cinder.utils [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:48:15 user cinder-volume[84696]: DEBUG cinder.utils [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 138.557(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:48:15 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:48:15 user cinder-volume[84696]: INFO cinder.message.api [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Creating message record for request_id = req-51924a9e-5094-404b-8828-f76cf8bc761a Apr 17 15:48:15 user cinder-volume[84696]: DEBUG cinder.utils [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 72.156(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:48:15 user cinder-volume[84696]: DEBUG cinder.utils [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_68f814b0-1291-4a12-ae65-dc0892203d36/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:48:15 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 15:48:15 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (82bda95f-132e-4edd-b2c1-b8ce90491f71) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 15:48:15 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 15:48:15 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:41:18Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesActionsTest-Volume-1624790909',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=441910f8-d29c-4c9e-bc5b-780ad228c85d,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='a8b97ff6ed6047f0a33e74684a1c162b',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:45:57Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:45:57Z,use_quota=True,user_id='c993ceeb4a5c49eea14057e2125518ad',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 15:48:15 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:41:18Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesActionsTest-Volume-1624790909',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=441910f8-d29c-4c9e-bc5b-780ad228c85d,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='a8b97ff6ed6047f0a33e74684a1c162b',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:45:57Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:45:57Z,use_quota=True,user_id='c993ceeb4a5c49eea14057e2125518ad',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(441910f8-d29c-4c9e-bc5b-780ad228c85d),volume_id=441910f8-d29c-4c9e-bc5b-780ad228c85d,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': '441910f8-d29c-4c9e-bc5b-780ad228c85d', 'volume_name': 'volume-441910f8-d29c-4c9e-bc5b-780ad228c85d', 'volume_size': 1}} Apr 17 15:48:15 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': '441910f8-d29c-4c9e-bc5b-780ad228c85d', 'volume': {'id': '441910f8-d29c-4c9e-bc5b-780ad228c85d', '_name_id': None, 'ec2_id': None, 'user_id': 'c993ceeb4a5c49eea14057e2125518ad', 'project_id': 'a8b97ff6ed6047f0a33e74684a1c162b', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-VolumesActionsTest-Volume-1624790909', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'admin_metadata': {}, 'glance_metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T15:41:18.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-441910f8-d29c-4c9e-bc5b-780ad228c85d', 'name_id': '441910f8-d29c-4c9e-bc5b-780ad228c85d', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumesActionsTest-Volume-1624790909', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': 'a8b97ff6ed6047f0a33e74684a1c162b', 'qos_specs': None, 'reservations': ['78c7033e-ef76-463d-90b8-8624379b802e', '4c9ae8cb-21fa-4ffc-97d0-d16076593bb1', '7ea6c36f-c448-4bcd-9d91-ca9a7f03d807', '60f944ba-84eb-4ff9-b309-d6e9af0d6b61'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'c993ceeb4a5c49eea14057e2125518ad', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumesActionsTest-Volume-1624790909', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': 'a8b97ff6ed6047f0a33e74684a1c162b', 'qos_specs': None, 'reservations': ['78c7033e-ef76-463d-90b8-8624379b802e', '4c9ae8cb-21fa-4ffc-97d0-d16076593bb1', '7ea6c36f-c448-4bcd-9d91-ca9a7f03d807', '60f944ba-84eb-4ff9-b309-d6e9af0d6b61'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'c993ceeb4a5c49eea14057e2125518ad', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': 'c993ceeb4a5c49eea14057e2125518ad', 'metadata': {}, 'qos_specs': None, 'retry': {'num_attempts': 3, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']}}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(441910f8-d29c-4c9e-bc5b-780ad228c85d),volume_id=441910f8-d29c-4c9e-bc5b-780ad228c85d,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:41:18Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesActionsTest-Volume-1624790909',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=441910f8-d29c-4c9e-bc5b-780ad228c85d,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='a8b97ff6ed6047f0a33e74684a1c162b',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:45:57Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:45:57Z,use_quota=True,user_id='c993ceeb4a5c49eea14057e2125518ad',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 15:48:15 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:41:18Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesActionsTest-Volume-1624790909',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=441910f8-d29c-4c9e-bc5b-780ad228c85d,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='a8b97ff6ed6047f0a33e74684a1c162b',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:45:57Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:45:57Z,use_quota=True,user_id='c993ceeb4a5c49eea14057e2125518ad',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:41:18Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesActionsTest-Volume-1624790909',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=441910f8-d29c-4c9e-bc5b-780ad228c85d,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='a8b97ff6ed6047f0a33e74684a1c162b',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:45:57Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:45:57Z,use_quota=True,user_id='c993ceeb4a5c49eea14057e2125518ad',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 15:48:15 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 15:48:15 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 15:48:15 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:48:15 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 15:48:15 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 15:48:15 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 15:48:15 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 15:48:15 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 15:48:15 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 15:48:15 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 15:48:15 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 15:48:15 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 15:48:15 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 15:48:15 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 15:48:15 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 15:48:15 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 15:48:15 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 15:48:15 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 15:48:15 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 15:48:15 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 15:48:15 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (82bda95f-132e-4edd-b2c1-b8ce90491f71) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:48:15 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (82bda95f-132e-4edd-b2c1-b8ce90491f71) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:48:15 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (7ff4de90-0cac-49ae-8514-c1d398eed82e) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:48:15 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (7ff4de90-0cac-49ae-8514-c1d398eed82e) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:48:15 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (70f464cd-d9ff-4fa4-aa19-dccc1bdaefa2) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:48:15 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (70f464cd-d9ff-4fa4-aa19-dccc1bdaefa2) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:48:15 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (282c69f7-04a6-489b-b737-2ce95b429beb) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:48:15 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Updating volume 441910f8-d29c-4c9e-bc5b-780ad228c85d with {'scheduled_at': datetime.datetime(2023, 4, 17, 15, 48, 15, 669528), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 15:48:15 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Volume 441910f8-d29c-4c9e-bc5b-780ad228c85d: re-scheduling SchedulerAPI.create_volume attempt 3 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 15:48:15 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Volume 441910f8-d29c-4c9e-bc5b-780ad228c85d: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 15:48:15 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] deleating volume v_441910f8-d29c-4c9e-bc5b-780ad228c85d {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 15:48:15 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] get snapshots for volume v_441910f8-d29c-4c9e-bc5b-780ad228c85d {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 15:48:15 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_441910f8-d29c-4c9e-bc5b-780ad228c85d/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:48:15 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_441910f8-d29c-4c9e-bc5b-780ad228c85d/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:48:18 user cinder-volume[84696]: DEBUG cinder.utils [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.955(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:48:18 user cinder-volume[84696]: DEBUG cinder.utils [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_441910f8-d29c-4c9e-bc5b-780ad228c85d/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:48:18 user cinder-volume[84696]: DEBUG cinder.utils [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 11.892(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:48:18 user cinder-volume[84696]: DEBUG cinder.utils [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:48:18 user cinder-volume[84696]: DEBUG cinder.utils [req-778c62be-8963-4e6d-97f1-f456b759ce16 req-d9cf97f7-129f-4cb2-9bc2-f51b1339159a tempest-AttachVolumeShelveTestJSON-1745617049 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 9.644(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:48:18 user cinder-volume[84696]: DEBUG cinder.utils [req-778c62be-8963-4e6d-97f1-f456b759ce16 req-d9cf97f7-129f-4cb2-9bc2-f51b1339159a tempest-AttachVolumeShelveTestJSON-1745617049 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:13bf568b-585c-4556-ab78-e9930ad50214/luns/v_13bf568b-585c-4556-ab78-e9930ad50214 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:48:18 user cinder-volume[84696]: DEBUG cinder.utils [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 19.246(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:48:18 user cinder-volume[84696]: DEBUG cinder.utils [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:48:18 user cinder-volume[84696]: DEBUG cinder.utils [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.210(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:48:18 user cinder-volume[84696]: DEBUG cinder.utils [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_93a32a7c-4c6a-472a-8f91-9d34558ba232/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:48:21 user cinder-volume[84696]: DEBUG cinder.utils [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 138.717(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:48:21 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_5c16bb82-43c5-4d91-9320-0e1f2bbec574/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:48:21 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (ba07b316-c816-4732-abc1-488bfdbe265a) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 15:48:21 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (9e7517b4-e533-4710-8e8a-12463f40b91d) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:48:21 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (9e7517b4-e533-4710-8e8a-12463f40b91d) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:48:21 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Flow 'volume_create_manager' (52235598-0fc7-45e2-8428-7d140671d028) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 15:48:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 15:48:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 15:48:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 15:48:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 15:48:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 15:48:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 15:48:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 15:48:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 15:48:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 15:48:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 15:48:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 15:48:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 15:48:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 15:48:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 15:48:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 15:48:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 15:48:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 15:48:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 15:48:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 15:48:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 15:48:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 15:48:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 15:48:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 15:48:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 15:48:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:48:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 15:48:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 15:48:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 15:48:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 15:48:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 15:48:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 15:48:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 15:48:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 15:48:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 15:48:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 15:48:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 15:48:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 15:48:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 15:48:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 15:48:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 15:48:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 15:48:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 15:48:21 user cinder-volume[84696]: DEBUG cinder.utils [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 138.496(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:48:21 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:48:21 user cinder-volume[84696]: INFO cinder.message.api [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Creating message record for request_id = req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b Apr 17 15:48:21 user cinder-volume[84696]: DEBUG cinder.utils [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.050(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:48:21 user cinder-volume[84696]: DEBUG cinder.utils [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_441910f8-d29c-4c9e-bc5b-780ad228c85d/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:48:21 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 15:48:21 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (313e3521-7297-4c03-bcef-83a05d743c7b) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 15:48:21 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 15:48:21 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:43:43Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TestMinimumBasicScenario-volume-1067752463',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=5c16bb82-43c5-4d91-9320-0e1f2bbec574,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='17d1dae0ad7149bbbd160525d64c0c20',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:46:03Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:46:03Z,use_quota=True,user_id='3c407d9d494d485c9c95dc3981284fba',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 15:48:21 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:43:43Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TestMinimumBasicScenario-volume-1067752463',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=5c16bb82-43c5-4d91-9320-0e1f2bbec574,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='17d1dae0ad7149bbbd160525d64c0c20',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:46:03Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:46:03Z,use_quota=True,user_id='3c407d9d494d485c9c95dc3981284fba',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(5c16bb82-43c5-4d91-9320-0e1f2bbec574),volume_id=5c16bb82-43c5-4d91-9320-0e1f2bbec574,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': '5c16bb82-43c5-4d91-9320-0e1f2bbec574', 'volume_name': 'volume-5c16bb82-43c5-4d91-9320-0e1f2bbec574', 'volume_size': 1}} Apr 17 15:48:21 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': '5c16bb82-43c5-4d91-9320-0e1f2bbec574', 'volume': {'id': '5c16bb82-43c5-4d91-9320-0e1f2bbec574', '_name_id': None, 'ec2_id': None, 'user_id': '3c407d9d494d485c9c95dc3981284fba', 'project_id': '17d1dae0ad7149bbbd160525d64c0c20', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-TestMinimumBasicScenario-volume-1067752463', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'admin_metadata': {}, 'glance_metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T15:43:43.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-5c16bb82-43c5-4d91-9320-0e1f2bbec574', 'name_id': '5c16bb82-43c5-4d91-9320-0e1f2bbec574', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-TestMinimumBasicScenario-volume-1067752463', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '17d1dae0ad7149bbbd160525d64c0c20', 'qos_specs': None, 'reservations': ['dc14bf14-c68a-4eb7-b82a-f5cc6f1cd0d8', 'bc18e142-51f5-4393-aeb8-9697eb8e95e4', '3bbb7ba4-5edd-410f-beff-181109d36e4a', 'd2dbd02b-21f5-42b4-a139-2855ceadc14e'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': '3c407d9d494d485c9c95dc3981284fba', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-TestMinimumBasicScenario-volume-1067752463', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '17d1dae0ad7149bbbd160525d64c0c20', 'qos_specs': None, 'reservations': ['dc14bf14-c68a-4eb7-b82a-f5cc6f1cd0d8', 'bc18e142-51f5-4393-aeb8-9697eb8e95e4', '3bbb7ba4-5edd-410f-beff-181109d36e4a', 'd2dbd02b-21f5-42b4-a139-2855ceadc14e'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': '3c407d9d494d485c9c95dc3981284fba', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': '3c407d9d494d485c9c95dc3981284fba', 'metadata': {}, 'qos_specs': None, 'retry': {'num_attempts': 2, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']}}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(5c16bb82-43c5-4d91-9320-0e1f2bbec574),volume_id=5c16bb82-43c5-4d91-9320-0e1f2bbec574,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:43:43Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TestMinimumBasicScenario-volume-1067752463',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=5c16bb82-43c5-4d91-9320-0e1f2bbec574,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='17d1dae0ad7149bbbd160525d64c0c20',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:46:03Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:46:03Z,use_quota=True,user_id='3c407d9d494d485c9c95dc3981284fba',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 15:48:21 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:43:43Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TestMinimumBasicScenario-volume-1067752463',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=5c16bb82-43c5-4d91-9320-0e1f2bbec574,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='17d1dae0ad7149bbbd160525d64c0c20',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:46:03Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:46:03Z,use_quota=True,user_id='3c407d9d494d485c9c95dc3981284fba',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:43:43Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TestMinimumBasicScenario-volume-1067752463',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=5c16bb82-43c5-4d91-9320-0e1f2bbec574,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='17d1dae0ad7149bbbd160525d64c0c20',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:46:03Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:46:03Z,use_quota=True,user_id='3c407d9d494d485c9c95dc3981284fba',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 15:48:21 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 15:48:21 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 15:48:21 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:48:21 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 15:48:21 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 15:48:21 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 15:48:21 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 15:48:21 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 15:48:21 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 15:48:21 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 15:48:21 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 15:48:21 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 15:48:21 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 15:48:21 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 15:48:21 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 15:48:21 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 15:48:21 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 15:48:21 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 15:48:21 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 15:48:21 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 15:48:21 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (313e3521-7297-4c03-bcef-83a05d743c7b) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:48:21 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (313e3521-7297-4c03-bcef-83a05d743c7b) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:48:21 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (2622fe9a-32d4-49a4-b140-79612e7d3cb9) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:48:21 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (2622fe9a-32d4-49a4-b140-79612e7d3cb9) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:48:21 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (56e57616-c597-47ee-a18a-2db3962f40ee) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:48:21 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (56e57616-c597-47ee-a18a-2db3962f40ee) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:48:21 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (23135497-50b1-444f-9668-b80977e69ae3) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:48:21 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Updating volume 5c16bb82-43c5-4d91-9320-0e1f2bbec574 with {'scheduled_at': datetime.datetime(2023, 4, 17, 15, 48, 21, 803711), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 15:48:21 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Volume 5c16bb82-43c5-4d91-9320-0e1f2bbec574: re-scheduling SchedulerAPI.create_volume attempt 2 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 15:48:21 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Volume 5c16bb82-43c5-4d91-9320-0e1f2bbec574: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 15:48:21 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] deleating volume v_5c16bb82-43c5-4d91-9320-0e1f2bbec574 {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 15:48:21 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] get snapshots for volume v_5c16bb82-43c5-4d91-9320-0e1f2bbec574 {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 15:48:21 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_5c16bb82-43c5-4d91-9320-0e1f2bbec574/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:48:21 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_5c16bb82-43c5-4d91-9320-0e1f2bbec574/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:48:21 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Volume reschedule parameters: True retry: {'num_attempts': 3, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 15:48:21 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Flow 'volume_create_manager' (9decf911-1891-4460-a859-237cd7e861bd) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:48:21 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (52d382d0-f289-4b52-aacc-943b882e8e39) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:48:21 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (52d382d0-f289-4b52-aacc-943b882e8e39) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:43:43Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TestMinimumBasicScenario-volume-1067752463',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=5c16bb82-43c5-4d91-9320-0e1f2bbec574,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='17d1dae0ad7149bbbd160525d64c0c20',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:48:22Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:48:22Z,use_quota=True,user_id='3c407d9d494d485c9c95dc3981284fba',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:48:21 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (e09b31ab-cc40-4bfc-95b8-19c7269b9325) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:48:21 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (e09b31ab-cc40-4bfc-95b8-19c7269b9325) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:48:21 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (ec3d1ee5-01d1-4a00-b734-8a021097f480) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:48:21 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (ec3d1ee5-01d1-4a00-b734-8a021097f480) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': '5c16bb82-43c5-4d91-9320-0e1f2bbec574', 'volume_name': 'volume-5c16bb82-43c5-4d91-9320-0e1f2bbec574', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:48:21 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (194a7a34-0f2c-4b9b-a785-ee7c4c4bb522) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:48:22 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (194a7a34-0f2c-4b9b-a785-ee7c4c4bb522) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:48:22 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (f40cf3e8-b8e2-46c1-bf46-336c8a2ab7bf) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:48:22 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Volume 5c16bb82-43c5-4d91-9320-0e1f2bbec574: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-5c16bb82-43c5-4d91-9320-0e1f2bbec574', 'volume_size': 1} Apr 17 15:48:22 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] creating volume v_5c16bb82-43c5-4d91-9320-0e1f2bbec574. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 15:48:22 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] create volume {'name': 'v_5c16bb82-43c5-4d91-9320-0e1f2bbec574', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 15:48:22 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:48:22 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:48:24 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=25,cinder:UPDATE=5,cinder:INSERT=2 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:48:24 user cinder-volume[84696]: DEBUG cinder.utils [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.791(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:48:24 user cinder-volume[84696]: DEBUG cinder.utils [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:48:24 user cinder-volume[84696]: DEBUG cinder.utils [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.966(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:48:24 user cinder-volume[84696]: DEBUG cinder.utils [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_5c16bb82-43c5-4d91-9320-0e1f2bbec574/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:48:27 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 72.240(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:48:27 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:48:27 user cinder-volume[84696]: DEBUG cinder.utils [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.172(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:48:27 user cinder-volume[84696]: DEBUG cinder.utils [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_441910f8-d29c-4c9e-bc5b-780ad228c85d/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:48:27 user cinder-volume[84696]: DEBUG cinder.utils [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.424(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:48:27 user cinder-volume[84696]: DEBUG cinder.utils [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_93a32a7c-4c6a-472a-8f91-9d34558ba232/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:48:27 user cinder-volume[84696]: DEBUG cinder.utils [req-778c62be-8963-4e6d-97f1-f456b759ce16 req-d9cf97f7-129f-4cb2-9bc2-f51b1339159a tempest-AttachVolumeShelveTestJSON-1745617049 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 18.861(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:48:27 user cinder-volume[84696]: DEBUG cinder.utils [req-778c62be-8963-4e6d-97f1-f456b759ce16 req-d9cf97f7-129f-4cb2-9bc2-f51b1339159a tempest-AttachVolumeShelveTestJSON-1745617049 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:13bf568b-585c-4556-ab78-e9930ad50214/luns/v_13bf568b-585c-4556-ab78-e9930ad50214 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:48:27 user cinder-volume[84696]: DEBUG cinder.utils [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.110(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:48:27 user cinder-volume[84696]: DEBUG cinder.utils [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:48:27 user cinder-volume[84696]: DEBUG cinder.utils [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.864(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:48:27 user cinder-volume[84696]: DEBUG cinder.utils [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:48:27 user cinder-volume[84696]: DEBUG cinder.utils [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.038(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:48:27 user cinder-volume[84696]: DEBUG cinder.utils [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_5c16bb82-43c5-4d91-9320-0e1f2bbec574/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:48:34 user cinder-volume[84696]: DEBUG cinder.utils [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.003(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:48:34 user cinder-volume[84696]: DEBUG cinder.utils [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:48:34 user cinder-volume[84696]: DEBUG cinder.utils [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.178(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:48:34 user cinder-volume[84696]: DEBUG cinder.utils [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_5c16bb82-43c5-4d91-9320-0e1f2bbec574/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:48:37 user cinder-volume[84696]: DEBUG cinder.utils [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 37.675(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:48:37 user cinder-volume[84696]: DEBUG cinder.utils [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:48:37 user cinder-volume[84696]: DEBUG cinder.utils [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 138.542(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:48:37 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:48:37 user cinder-volume[84696]: INFO cinder.message.api [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Creating message record for request_id = req-636827e5-ca50-4d75-a738-9b54469bf4fe Apr 17 15:48:37 user cinder-volume[84696]: DEBUG cinder.utils [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 138.734(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:48:37 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_2c9714de-f8ea-4d1d-99f9-beb85126ad91/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:48:37 user cinder-volume[84696]: WARNING cinder.volume.manager [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (91eb764b-f46d-44d6-b36e-ef5e6fb68451) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 15:48:37 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (997e1140-d27b-408f-85ff-51beb6aecd35) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:48:37 user cinder-volume[84696]: WARNING cinder.volume.manager [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (997e1140-d27b-408f-85ff-51beb6aecd35) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:48:37 user cinder-volume[84696]: WARNING cinder.volume.manager [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Flow 'volume_create_manager' (398ad45c-928a-4370-97f4-8e626bb3991b) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 15:48:37 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 15:48:37 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 15:48:37 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 15:48:37 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 15:48:37 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 15:48:37 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 15:48:37 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 15:48:37 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 15:48:37 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 15:48:37 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 15:48:37 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 15:48:37 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 15:48:37 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 15:48:37 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 15:48:37 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 15:48:37 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 15:48:37 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 15:48:37 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 15:48:37 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 15:48:37 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 15:48:37 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 15:48:37 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 15:48:37 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 15:48:37 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 15:48:37 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:48:37 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 15:48:37 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 15:48:37 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 15:48:37 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 15:48:37 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 15:48:37 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 15:48:37 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 15:48:37 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 15:48:37 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 15:48:37 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 15:48:37 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 15:48:37 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 15:48:37 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 15:48:37 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 15:48:37 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 15:48:37 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 15:48:37 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 15:48:37 user cinder-volume[84696]: DEBUG cinder.utils [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.410(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:48:37 user cinder-volume[84696]: DEBUG cinder.utils [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_441910f8-d29c-4c9e-bc5b-780ad228c85d/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:48:37 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 15:48:37 user cinder-volume[84696]: WARNING cinder.volume.manager [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (2eecdd60-c8ce-49c2-8382-d7abd644f212) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 15:48:37 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 15:48:37 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:44:03Z,deleted=False,deleted_at=None,display_description='',display_name='05e08eea-cc2c-4332-9e97-07cb9b466af6-blank-vol',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=2c9714de-f8ea-4d1d-99f9-beb85126ad91,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='540826be4baf46dba9617b2fd47821a3',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:46:18Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:46:18Z,use_quota=True,user_id='d819e3f51368413c99691d1b45bbcfb1',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 15:48:37 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:44:03Z,deleted=False,deleted_at=None,display_description='',display_name='05e08eea-cc2c-4332-9e97-07cb9b466af6-blank-vol',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=2c9714de-f8ea-4d1d-99f9-beb85126ad91,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='540826be4baf46dba9617b2fd47821a3',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:46:18Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:46:18Z,use_quota=True,user_id='d819e3f51368413c99691d1b45bbcfb1',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(2c9714de-f8ea-4d1d-99f9-beb85126ad91),volume_id=2c9714de-f8ea-4d1d-99f9-beb85126ad91,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': '2c9714de-f8ea-4d1d-99f9-beb85126ad91', 'volume_name': 'volume-2c9714de-f8ea-4d1d-99f9-beb85126ad91', 'volume_size': 1}} Apr 17 15:48:37 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': '2c9714de-f8ea-4d1d-99f9-beb85126ad91', 'volume': {'id': '2c9714de-f8ea-4d1d-99f9-beb85126ad91', '_name_id': None, 'ec2_id': None, 'user_id': 'd819e3f51368413c99691d1b45bbcfb1', 'project_id': '540826be4baf46dba9617b2fd47821a3', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': '05e08eea-cc2c-4332-9e97-07cb9b466af6-blank-vol', 'display_description': '', 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'admin_metadata': {}, 'glance_metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T15:44:02.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-2c9714de-f8ea-4d1d-99f9-beb85126ad91', 'name_id': '2c9714de-f8ea-4d1d-99f9-beb85126ad91', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': '', 'display_name': '05e08eea-cc2c-4332-9e97-07cb9b466af6-blank-vol', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '540826be4baf46dba9617b2fd47821a3', 'qos_specs': None, 'reservations': ['e20dc1ca-0a9e-4413-be1e-c7f19faae9e1', '81c416d5-515b-4265-888a-740c186d6f45', 'da7244e3-def8-4c3b-b96c-0088e02b1914', 'afef167b-9548-4f1a-a499-b206d13c826a'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'd819e3f51368413c99691d1b45bbcfb1', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': '', 'display_name': '05e08eea-cc2c-4332-9e97-07cb9b466af6-blank-vol', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '540826be4baf46dba9617b2fd47821a3', 'qos_specs': None, 'reservations': ['e20dc1ca-0a9e-4413-be1e-c7f19faae9e1', '81c416d5-515b-4265-888a-740c186d6f45', 'da7244e3-def8-4c3b-b96c-0088e02b1914', 'afef167b-9548-4f1a-a499-b206d13c826a'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'd819e3f51368413c99691d1b45bbcfb1', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': 'd819e3f51368413c99691d1b45bbcfb1', 'metadata': {}, 'qos_specs': None, 'retry': {'num_attempts': 2, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']}}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(2c9714de-f8ea-4d1d-99f9-beb85126ad91),volume_id=2c9714de-f8ea-4d1d-99f9-beb85126ad91,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:44:03Z,deleted=False,deleted_at=None,display_description='',display_name='05e08eea-cc2c-4332-9e97-07cb9b466af6-blank-vol',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=2c9714de-f8ea-4d1d-99f9-beb85126ad91,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='540826be4baf46dba9617b2fd47821a3',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:46:18Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:46:18Z,use_quota=True,user_id='d819e3f51368413c99691d1b45bbcfb1',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 15:48:37 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:44:03Z,deleted=False,deleted_at=None,display_description='',display_name='05e08eea-cc2c-4332-9e97-07cb9b466af6-blank-vol',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=2c9714de-f8ea-4d1d-99f9-beb85126ad91,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='540826be4baf46dba9617b2fd47821a3',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:46:18Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:46:18Z,use_quota=True,user_id='d819e3f51368413c99691d1b45bbcfb1',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:44:03Z,deleted=False,deleted_at=None,display_description='',display_name='05e08eea-cc2c-4332-9e97-07cb9b466af6-blank-vol',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=2c9714de-f8ea-4d1d-99f9-beb85126ad91,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='540826be4baf46dba9617b2fd47821a3',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:46:18Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:46:18Z,use_quota=True,user_id='d819e3f51368413c99691d1b45bbcfb1',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 15:48:37 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 15:48:37 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 15:48:37 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:48:37 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 15:48:37 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 15:48:37 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 15:48:37 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 15:48:37 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 15:48:37 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 15:48:37 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 15:48:37 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 15:48:37 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 15:48:37 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 15:48:37 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 15:48:37 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 15:48:37 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 15:48:37 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 15:48:37 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 15:48:37 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 15:48:37 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 15:48:37 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (2eecdd60-c8ce-49c2-8382-d7abd644f212) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:48:37 user cinder-volume[84696]: WARNING cinder.volume.manager [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (2eecdd60-c8ce-49c2-8382-d7abd644f212) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:48:37 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (e0532b43-faf5-4c9a-bef5-86d349478076) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:48:37 user cinder-volume[84696]: WARNING cinder.volume.manager [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (e0532b43-faf5-4c9a-bef5-86d349478076) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:48:37 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (709c2d10-0581-4c42-964c-79370796a7ff) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:48:37 user cinder-volume[84696]: WARNING cinder.volume.manager [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (709c2d10-0581-4c42-964c-79370796a7ff) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:48:37 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (6d6245bf-af9c-4f2e-b7ff-5d36fbf9d72d) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:48:37 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Updating volume 2c9714de-f8ea-4d1d-99f9-beb85126ad91 with {'scheduled_at': datetime.datetime(2023, 4, 17, 15, 48, 37, 161817), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 15:48:37 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Volume 2c9714de-f8ea-4d1d-99f9-beb85126ad91: re-scheduling SchedulerAPI.create_volume attempt 2 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 15:48:37 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Volume 2c9714de-f8ea-4d1d-99f9-beb85126ad91: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 15:48:37 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] deleating volume v_2c9714de-f8ea-4d1d-99f9-beb85126ad91 {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 15:48:37 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] get snapshots for volume v_2c9714de-f8ea-4d1d-99f9-beb85126ad91 {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 15:48:37 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_2c9714de-f8ea-4d1d-99f9-beb85126ad91/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:48:37 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_2c9714de-f8ea-4d1d-99f9-beb85126ad91/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:48:37 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Volume reschedule parameters: True retry: {'num_attempts': 3, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 15:48:37 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Flow 'volume_create_manager' (84ab4fd6-b295-4c89-8f6c-5cf3b6f20b59) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:48:37 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (78d194dd-793d-4ef0-b16a-5e0e901e6740) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:48:37 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (78d194dd-793d-4ef0-b16a-5e0e901e6740) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:44:03Z,deleted=False,deleted_at=None,display_description='',display_name='05e08eea-cc2c-4332-9e97-07cb9b466af6-blank-vol',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=2c9714de-f8ea-4d1d-99f9-beb85126ad91,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='540826be4baf46dba9617b2fd47821a3',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:48:37Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:48:37Z,use_quota=True,user_id='d819e3f51368413c99691d1b45bbcfb1',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:48:37 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (2e236988-b8a4-435d-9f21-a584f5eac774) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:48:37 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (2e236988-b8a4-435d-9f21-a584f5eac774) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:48:37 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (8f68d0d2-69e4-463b-a42e-eb323b7280a2) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:48:37 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (8f68d0d2-69e4-463b-a42e-eb323b7280a2) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': '2c9714de-f8ea-4d1d-99f9-beb85126ad91', 'volume_name': 'volume-2c9714de-f8ea-4d1d-99f9-beb85126ad91', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:48:37 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (b9052dc1-84b3-4996-9500-5074d44e289a) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:48:37 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (b9052dc1-84b3-4996-9500-5074d44e289a) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:48:37 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (4efd3e1a-02eb-469a-a755-003065d9b7d2) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:48:37 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Volume 2c9714de-f8ea-4d1d-99f9-beb85126ad91: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-2c9714de-f8ea-4d1d-99f9-beb85126ad91', 'volume_size': 1} Apr 17 15:48:37 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] creating volume v_2c9714de-f8ea-4d1d-99f9-beb85126ad91. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 15:48:37 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] create volume {'name': 'v_2c9714de-f8ea-4d1d-99f9-beb85126ad91', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 15:48:37 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:48:37 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:48:40 user cinder-volume[84696]: DEBUG cinder.utils [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.966(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:48:40 user cinder-volume[84696]: DEBUG cinder.utils [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_2c9714de-f8ea-4d1d-99f9-beb85126ad91/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:48:40 user cinder-volume[84696]: DEBUG cinder.utils [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.784(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:48:40 user cinder-volume[84696]: DEBUG cinder.utils [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:48:43 user cinder-volume[84696]: DEBUG cinder.utils [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.219(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:48:43 user cinder-volume[84696]: DEBUG cinder.utils [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:48:43 user cinder-volume[84696]: DEBUG cinder.utils [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.041(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:48:43 user cinder-volume[84696]: DEBUG cinder.utils [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_2c9714de-f8ea-4d1d-99f9-beb85126ad91/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:48:43 user cinder-volume[84696]: DEBUG cinder.utils [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.396(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:48:43 user cinder-volume[84696]: DEBUG cinder.utils [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_5c16bb82-43c5-4d91-9320-0e1f2bbec574/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:48:43 user cinder-volume[84696]: DEBUG cinder.utils [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.859(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:48:43 user cinder-volume[84696]: DEBUG cinder.utils [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:48:44 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=13,cinder:UPDATE=8,cinder:INSERT=3 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:48:46 user cinder-volume[84696]: DEBUG cinder.utils [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.854(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:48:46 user cinder-volume[84696]: DEBUG cinder.utils [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_93a32a7c-4c6a-472a-8f91-9d34558ba232/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:48:46 user cinder-volume[84696]: DEBUG cinder.utils [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.541(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:48:46 user cinder-volume[84696]: DEBUG cinder.utils [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:48:46 user cinder-volume[84696]: DEBUG cinder.utils [req-778c62be-8963-4e6d-97f1-f456b759ce16 req-d9cf97f7-129f-4cb2-9bc2-f51b1339159a tempest-AttachVolumeShelveTestJSON-1745617049 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 37.294(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:48:46 user cinder-volume[84696]: DEBUG cinder.utils [req-778c62be-8963-4e6d-97f1-f456b759ce16 req-d9cf97f7-129f-4cb2-9bc2-f51b1339159a tempest-AttachVolumeShelveTestJSON-1745617049 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:13bf568b-585c-4556-ab78-e9930ad50214/luns/v_13bf568b-585c-4556-ab78-e9930ad50214 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:48:49 user cinder-volume[84696]: DEBUG cinder.utils [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.183(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:48:49 user cinder-volume[84696]: DEBUG cinder.utils [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_2c9714de-f8ea-4d1d-99f9-beb85126ad91/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:48:49 user cinder-volume[84696]: DEBUG cinder.utils [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 11.999(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:48:49 user cinder-volume[84696]: DEBUG cinder.utils [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:48:56 user cinder-volume[84696]: DEBUG cinder.utils [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 40.491(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:48:56 user cinder-volume[84696]: DEBUG cinder.utils [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_441910f8-d29c-4c9e-bc5b-780ad228c85d/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:48:59 user cinder-volume[84696]: DEBUG cinder.utils [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 37.424(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:48:59 user cinder-volume[84696]: DEBUG cinder.utils [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_5c16bb82-43c5-4d91-9320-0e1f2bbec574/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:48:59 user cinder-volume[84696]: DEBUG cinder.utils [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.886(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:48:59 user cinder-volume[84696]: DEBUG cinder.utils [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:48:59 user cinder-volume[84696]: DEBUG cinder.utils [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 37.253(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:48:59 user cinder-volume[84696]: DEBUG cinder.utils [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:48:59 user cinder-volume[84696]: DEBUG cinder.utils [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 22.073(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:48:59 user cinder-volume[84696]: DEBUG cinder.utils [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_2c9714de-f8ea-4d1d-99f9-beb85126ad91/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:49:04 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=9,cinder:UPDATE=2,cinder:INSERT=1 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:49:09 user cinder-volume[84696]: DEBUG cinder.utils [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 69.962(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:49:09 user cinder-volume[84696]: DEBUG cinder.utils [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:49:15 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=8,cinder:UPDATE=2 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:49:15 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Volume reschedule parameters: True retry: {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 15:49:15 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Flow 'volume_create_manager' (bd64ca5c-c8d5-4e5a-ab0a-49ca63e7ecb8) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:49:15 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (d61e31b9-a310-449f-be85-a8c6409e5f39) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:49:15 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (d61e31b9-a310-449f-be85-a8c6409e5f39) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:49:15Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-AttachVolumeNegativeTest-volume-1650632398',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=4a2d1c00-a134-4f73-8508-86f35b58acf7,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='8586cc10142d4743874f43cdb0f37aba',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:49:15Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:49:15Z,use_quota=True,user_id='9fdc7c80b4654692a52c8a4c4149324f',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:49:15 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (057c958c-d875-42c6-8087-149e232298b2) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:49:15 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (057c958c-d875-42c6-8087-149e232298b2) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:49:15 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (cc3875b0-1df3-4f2c-9e20-c10a31bc194c) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:49:15 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (cc3875b0-1df3-4f2c-9e20-c10a31bc194c) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': '4a2d1c00-a134-4f73-8508-86f35b58acf7', 'volume_name': 'volume-4a2d1c00-a134-4f73-8508-86f35b58acf7', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:49:15 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (18626c41-c279-460e-b48f-ac0f1e256fdc) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:49:15 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (18626c41-c279-460e-b48f-ac0f1e256fdc) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:49:15 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (65a59a4e-335e-48a1-a7c5-bc52c1c31f2c) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:49:15 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Volume 4a2d1c00-a134-4f73-8508-86f35b58acf7: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-4a2d1c00-a134-4f73-8508-86f35b58acf7', 'volume_size': 1} Apr 17 15:49:15 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] creating volume v_4a2d1c00-a134-4f73-8508-86f35b58acf7. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 15:49:15 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] create volume {'name': 'v_4a2d1c00-a134-4f73-8508-86f35b58acf7', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 15:49:15 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:49:15 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:49:16 user cinder-volume[84696]: DEBUG cinder.utils [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 138.927(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:49:16 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_aea45c70-7c05-4146-847e-cbb777d3dfbe/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:49:16 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (947d63b2-51cb-4ef3-83c3-96e17bebdb4c) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 15:49:16 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (824dee1d-7348-4235-a659-2415ca567d67) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:49:16 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (824dee1d-7348-4235-a659-2415ca567d67) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:49:16 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Flow 'volume_create_manager' (c41a7b6c-c6fe-4d33-a8a4-0f6ddd345296) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 15:49:16 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 15:49:16 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 15:49:16 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 15:49:16 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 15:49:16 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 15:49:16 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 15:49:16 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 15:49:16 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 15:49:16 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 15:49:16 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 15:49:16 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 15:49:16 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 15:49:16 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 15:49:16 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 15:49:16 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 15:49:16 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 15:49:16 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 15:49:16 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 15:49:16 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 15:49:16 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 15:49:16 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 15:49:16 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 15:49:16 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 15:49:16 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 15:49:16 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:49:16 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 15:49:16 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 15:49:16 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 15:49:16 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 15:49:16 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 15:49:16 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 15:49:16 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 15:49:16 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 15:49:16 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 15:49:16 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 15:49:16 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 15:49:16 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 15:49:16 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 15:49:16 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 15:49:16 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 15:49:16 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 15:49:16 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 15:49:16 user cinder-volume[84696]: DEBUG cinder.utils [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 138.624(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:49:16 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:49:16 user cinder-volume[84696]: INFO cinder.message.api [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Creating message record for request_id = req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 Apr 17 15:49:16 user cinder-volume[84696]: DEBUG cinder.utils [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 138.745(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:49:16 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:49:16 user cinder-volume[84696]: INFO cinder.message.api [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Creating message record for request_id = req-be98cecd-d344-4ab9-b5f6-c268a7041adb Apr 17 15:49:16 user cinder-volume[84696]: DEBUG cinder.utils [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 138.591(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:49:16 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:49:16 user cinder-volume[84696]: INFO cinder.message.api [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Creating message record for request_id = req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f Apr 17 15:49:16 user cinder-volume[84696]: DEBUG cinder.utils [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 138.962(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:49:16 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_dfd745ed-b0c4-4e35-b77c-a911c90ce33e/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:49:16 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (b63ce73b-bcf8-4cfb-87d9-296580b1da26) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 15:49:16 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (cc5db28e-1f6b-4789-b35a-420b13e30f80) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:49:16 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (cc5db28e-1f6b-4789-b35a-420b13e30f80) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:49:16 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Flow 'volume_create_manager' (1a9a0e5d-c7b0-4ded-a0f8-8dd4d09ecaf4) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 15:49:16 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 15:49:16 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 15:49:16 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 15:49:16 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 15:49:16 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 15:49:16 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 15:49:16 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 15:49:16 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 15:49:16 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 15:49:16 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 15:49:16 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 15:49:16 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 15:49:16 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 15:49:16 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 15:49:16 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 15:49:16 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 15:49:16 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 15:49:16 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 15:49:16 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 15:49:16 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 15:49:16 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 15:49:16 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 15:49:16 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 15:49:16 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 15:49:16 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:49:16 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 15:49:16 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 15:49:16 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 15:49:16 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 15:49:16 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 15:49:16 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 15:49:16 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 15:49:16 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 15:49:16 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 15:49:16 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 15:49:16 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 15:49:16 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 15:49:16 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 15:49:16 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 15:49:16 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 15:49:16 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 15:49:16 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 15:49:16 user cinder-volume[84696]: DEBUG cinder.utils [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.019(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:49:16 user cinder-volume[84696]: DEBUG cinder.utils [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:49:16 user cinder-volume[84696]: DEBUG cinder.utils [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 0.802(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:49:16 user cinder-volume[84696]: DEBUG cinder.utils [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:49:16 user cinder-volume[84696]: DEBUG cinder.utils [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 139.007(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:49:16 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_fbbadf16-651a-47d5-b308-a37e56aa07f2/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:49:16 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (07ade4d8-0421-45fb-9322-2eed895f7d00) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 15:49:16 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (dfed3372-6304-4ab9-89dd-8b11f2799c62) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:49:16 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (dfed3372-6304-4ab9-89dd-8b11f2799c62) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:49:16 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Flow 'volume_create_manager' (150566c6-b9f6-438e-9f32-f77b92c1a70e) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 15:49:16 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 15:49:16 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 15:49:16 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 15:49:16 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 15:49:16 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 15:49:16 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 15:49:16 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 15:49:16 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 15:49:16 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 15:49:16 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 15:49:16 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 15:49:16 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 15:49:16 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 15:49:16 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 15:49:16 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 15:49:16 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 15:49:16 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 15:49:16 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 15:49:16 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 15:49:16 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 15:49:16 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 15:49:16 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 15:49:16 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 15:49:16 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 15:49:16 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:49:16 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 15:49:16 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 15:49:16 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 15:49:16 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 15:49:16 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 15:49:16 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 15:49:16 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 15:49:16 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 15:49:16 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 15:49:16 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 15:49:16 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 15:49:16 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 15:49:16 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 15:49:16 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 15:49:16 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 15:49:16 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 15:49:16 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 15:49:16 user cinder-volume[84696]: DEBUG cinder.utils [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.226(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:49:16 user cinder-volume[84696]: DEBUG cinder.utils [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_2c9714de-f8ea-4d1d-99f9-beb85126ad91/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:49:16 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 15:49:16 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 15:49:16 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 15:49:16 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (841e870b-81ae-4ee0-be56-d91eaaa8981b) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 15:49:16 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 15:49:16 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:44:39Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-CreateVolumesFromImageTest-Volume-800604710',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=aea45c70-7c05-4146-847e-cbb777d3dfbe,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='75babbbca67f4035a48ee8ee816b18db',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:46:58Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:46:58Z,use_quota=True,user_id='d5299006144f4ee6817349348a489017',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 15:49:16 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:44:39Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-CreateVolumesFromImageTest-Volume-800604710',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=aea45c70-7c05-4146-847e-cbb777d3dfbe,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='75babbbca67f4035a48ee8ee816b18db',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:46:58Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:46:58Z,use_quota=True,user_id='d5299006144f4ee6817349348a489017',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(aea45c70-7c05-4146-847e-cbb777d3dfbe),volume_id=aea45c70-7c05-4146-847e-cbb777d3dfbe,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': 'aea45c70-7c05-4146-847e-cbb777d3dfbe', 'volume_name': 'volume-aea45c70-7c05-4146-847e-cbb777d3dfbe', 'volume_size': 1}} Apr 17 15:49:16 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': 'aea45c70-7c05-4146-847e-cbb777d3dfbe', 'volume': {'id': 'aea45c70-7c05-4146-847e-cbb777d3dfbe', '_name_id': None, 'ec2_id': None, 'user_id': 'd5299006144f4ee6817349348a489017', 'project_id': '75babbbca67f4035a48ee8ee816b18db', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-CreateVolumesFromImageTest-Volume-800604710', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'admin_metadata': {}, 'glance_metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T15:44:38.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-aea45c70-7c05-4146-847e-cbb777d3dfbe', 'name_id': 'aea45c70-7c05-4146-847e-cbb777d3dfbe', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-CreateVolumesFromImageTest-Volume-800604710', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '75babbbca67f4035a48ee8ee816b18db', 'qos_specs': None, 'reservations': ['0a40a0f0-8514-4f1f-ac7e-cbc2d47cbe5f', 'b703b541-7376-4c99-8896-74ebf0e7fb9f', '4c0d25ea-51e7-41c5-8f64-d45e42a842b1', '32031e87-40eb-4f2d-aa17-2ea553f6a406'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'd5299006144f4ee6817349348a489017', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-CreateVolumesFromImageTest-Volume-800604710', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '75babbbca67f4035a48ee8ee816b18db', 'qos_specs': None, 'reservations': ['0a40a0f0-8514-4f1f-ac7e-cbc2d47cbe5f', 'b703b541-7376-4c99-8896-74ebf0e7fb9f', '4c0d25ea-51e7-41c5-8f64-d45e42a842b1', '32031e87-40eb-4f2d-aa17-2ea553f6a406'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'd5299006144f4ee6817349348a489017', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': 'd5299006144f4ee6817349348a489017', 'metadata': {}, 'qos_specs': None, 'retry': {'num_attempts': 2, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']}}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(aea45c70-7c05-4146-847e-cbb777d3dfbe),volume_id=aea45c70-7c05-4146-847e-cbb777d3dfbe,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:44:39Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-CreateVolumesFromImageTest-Volume-800604710',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=aea45c70-7c05-4146-847e-cbb777d3dfbe,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='75babbbca67f4035a48ee8ee816b18db',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:46:58Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:46:58Z,use_quota=True,user_id='d5299006144f4ee6817349348a489017',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 15:49:16 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:44:39Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-CreateVolumesFromImageTest-Volume-800604710',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=aea45c70-7c05-4146-847e-cbb777d3dfbe,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='75babbbca67f4035a48ee8ee816b18db',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:46:58Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:46:58Z,use_quota=True,user_id='d5299006144f4ee6817349348a489017',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:44:39Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-CreateVolumesFromImageTest-Volume-800604710',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=aea45c70-7c05-4146-847e-cbb777d3dfbe,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='75babbbca67f4035a48ee8ee816b18db',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:46:58Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:46:58Z,use_quota=True,user_id='d5299006144f4ee6817349348a489017',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 15:49:16 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 15:49:16 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 15:49:16 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:49:16 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 15:49:16 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 15:49:16 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 15:49:16 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 15:49:16 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 15:49:16 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 15:49:16 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 15:49:16 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 15:49:16 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 15:49:16 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 15:49:16 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 15:49:16 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 15:49:16 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 15:49:16 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 15:49:16 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 15:49:16 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 15:49:16 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 15:49:16 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (841e870b-81ae-4ee0-be56-d91eaaa8981b) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:49:16 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (841e870b-81ae-4ee0-be56-d91eaaa8981b) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:49:16 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (2f57f3d6-f99e-4921-8e08-fe8ce0153780) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:49:16 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (2f57f3d6-f99e-4921-8e08-fe8ce0153780) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:49:16 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (bea6314b-56a0-447d-87e7-f8136533d6e0) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:49:16 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (bea6314b-56a0-447d-87e7-f8136533d6e0) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:49:16 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (b5bdf986-2fce-4bd2-a19a-0e363a52af33) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:49:16 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Updating volume aea45c70-7c05-4146-847e-cbb777d3dfbe with {'scheduled_at': datetime.datetime(2023, 4, 17, 15, 49, 16, 471865), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 15:49:16 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (a5831bc7-05e8-4397-ac61-3fc78593eda3) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 15:49:16 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 15:49:16 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:44:39Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-CreateVolumesFromImageTest-Volume-2146530532',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=fbbadf16-651a-47d5-b308-a37e56aa07f2,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='75babbbca67f4035a48ee8ee816b18db',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:46:57Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:46:57Z,use_quota=True,user_id='d5299006144f4ee6817349348a489017',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 15:49:16 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:44:39Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-CreateVolumesFromImageTest-Volume-2146530532',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=fbbadf16-651a-47d5-b308-a37e56aa07f2,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='75babbbca67f4035a48ee8ee816b18db',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:46:57Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:46:57Z,use_quota=True,user_id='d5299006144f4ee6817349348a489017',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(fbbadf16-651a-47d5-b308-a37e56aa07f2),volume_id=fbbadf16-651a-47d5-b308-a37e56aa07f2,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': 'fbbadf16-651a-47d5-b308-a37e56aa07f2', 'volume_name': 'volume-fbbadf16-651a-47d5-b308-a37e56aa07f2', 'volume_size': 1}} Apr 17 15:49:16 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': 'fbbadf16-651a-47d5-b308-a37e56aa07f2', 'volume': {'id': 'fbbadf16-651a-47d5-b308-a37e56aa07f2', '_name_id': None, 'ec2_id': None, 'user_id': 'd5299006144f4ee6817349348a489017', 'project_id': '75babbbca67f4035a48ee8ee816b18db', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-CreateVolumesFromImageTest-Volume-2146530532', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'admin_metadata': {}, 'glance_metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T15:44:38.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-fbbadf16-651a-47d5-b308-a37e56aa07f2', 'name_id': 'fbbadf16-651a-47d5-b308-a37e56aa07f2', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-CreateVolumesFromImageTest-Volume-2146530532', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '75babbbca67f4035a48ee8ee816b18db', 'qos_specs': None, 'reservations': ['ebe2c0e3-e7c3-4d72-b0ff-d7918e651f9b', '11cb75cb-f9b7-46b2-bc19-f76f314b9643', '9657a0be-fa65-4315-b170-50972324b109', 'b1d14f83-c9a6-448d-acae-a1bf415819e5'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'd5299006144f4ee6817349348a489017', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-CreateVolumesFromImageTest-Volume-2146530532', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '75babbbca67f4035a48ee8ee816b18db', 'qos_specs': None, 'reservations': ['ebe2c0e3-e7c3-4d72-b0ff-d7918e651f9b', '11cb75cb-f9b7-46b2-bc19-f76f314b9643', '9657a0be-fa65-4315-b170-50972324b109', 'b1d14f83-c9a6-448d-acae-a1bf415819e5'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'd5299006144f4ee6817349348a489017', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': 'd5299006144f4ee6817349348a489017', 'metadata': {}, 'qos_specs': None, 'retry': {'num_attempts': 2, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']}}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(fbbadf16-651a-47d5-b308-a37e56aa07f2),volume_id=fbbadf16-651a-47d5-b308-a37e56aa07f2,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:44:39Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-CreateVolumesFromImageTest-Volume-2146530532',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=fbbadf16-651a-47d5-b308-a37e56aa07f2,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='75babbbca67f4035a48ee8ee816b18db',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:46:57Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:46:57Z,use_quota=True,user_id='d5299006144f4ee6817349348a489017',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 15:49:16 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:44:39Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-CreateVolumesFromImageTest-Volume-2146530532',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=fbbadf16-651a-47d5-b308-a37e56aa07f2,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='75babbbca67f4035a48ee8ee816b18db',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:46:57Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:46:57Z,use_quota=True,user_id='d5299006144f4ee6817349348a489017',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:44:39Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-CreateVolumesFromImageTest-Volume-2146530532',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=fbbadf16-651a-47d5-b308-a37e56aa07f2,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='75babbbca67f4035a48ee8ee816b18db',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:46:57Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:46:57Z,use_quota=True,user_id='d5299006144f4ee6817349348a489017',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 15:49:16 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 15:49:16 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 15:49:16 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:49:16 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 15:49:16 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 15:49:16 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 15:49:16 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 15:49:16 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 15:49:16 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 15:49:16 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 15:49:16 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 15:49:16 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 15:49:16 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 15:49:16 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 15:49:16 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 15:49:16 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 15:49:16 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 15:49:16 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 15:49:16 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 15:49:16 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 15:49:16 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (a5831bc7-05e8-4397-ac61-3fc78593eda3) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:49:16 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (a5831bc7-05e8-4397-ac61-3fc78593eda3) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:49:16 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (9cc722d5-3458-4059-b7c9-a8df875a18a8) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:49:16 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (9cc722d5-3458-4059-b7c9-a8df875a18a8) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:49:16 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (a1ae4143-8324-4ec4-92b5-17e7ab558456) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:49:16 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (a1ae4143-8324-4ec4-92b5-17e7ab558456) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:49:16 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (6af14be6-309b-47d8-9e72-4d46f866188d) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:49:16 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Updating volume fbbadf16-651a-47d5-b308-a37e56aa07f2 with {'scheduled_at': datetime.datetime(2023, 4, 17, 15, 49, 16, 522963), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 15:49:16 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (be26eb99-6fec-4d2f-ad74-1ffbc1521d7e) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 15:49:16 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 15:49:16 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:44:38Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-CreateVolumesFromImageTest-Volume-1463431787',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=dfd745ed-b0c4-4e35-b77c-a911c90ce33e,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='75babbbca67f4035a48ee8ee816b18db',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:46:58Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:46:58Z,use_quota=True,user_id='d5299006144f4ee6817349348a489017',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 15:49:16 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:44:38Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-CreateVolumesFromImageTest-Volume-1463431787',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=dfd745ed-b0c4-4e35-b77c-a911c90ce33e,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='75babbbca67f4035a48ee8ee816b18db',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:46:58Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:46:58Z,use_quota=True,user_id='d5299006144f4ee6817349348a489017',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(dfd745ed-b0c4-4e35-b77c-a911c90ce33e),volume_id=dfd745ed-b0c4-4e35-b77c-a911c90ce33e,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': 'dfd745ed-b0c4-4e35-b77c-a911c90ce33e', 'volume_name': 'volume-dfd745ed-b0c4-4e35-b77c-a911c90ce33e', 'volume_size': 1}} Apr 17 15:49:16 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': 'dfd745ed-b0c4-4e35-b77c-a911c90ce33e', 'volume': {'id': 'dfd745ed-b0c4-4e35-b77c-a911c90ce33e', '_name_id': None, 'ec2_id': None, 'user_id': 'd5299006144f4ee6817349348a489017', 'project_id': '75babbbca67f4035a48ee8ee816b18db', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-CreateVolumesFromImageTest-Volume-1463431787', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'admin_metadata': {}, 'glance_metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T15:44:38.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-dfd745ed-b0c4-4e35-b77c-a911c90ce33e', 'name_id': 'dfd745ed-b0c4-4e35-b77c-a911c90ce33e', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-CreateVolumesFromImageTest-Volume-1463431787', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '75babbbca67f4035a48ee8ee816b18db', 'qos_specs': None, 'reservations': ['24de6078-218c-412e-8c05-3e1cd2d51a18', '25d60ea2-888f-44b7-a809-be58574b0515', '8d3c3175-5e14-45c1-aac7-a222d3122bbb', '7c2bd4f1-736b-4589-98c3-db4b057ca99b'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'd5299006144f4ee6817349348a489017', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-CreateVolumesFromImageTest-Volume-1463431787', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '75babbbca67f4035a48ee8ee816b18db', 'qos_specs': None, 'reservations': ['24de6078-218c-412e-8c05-3e1cd2d51a18', '25d60ea2-888f-44b7-a809-be58574b0515', '8d3c3175-5e14-45c1-aac7-a222d3122bbb', '7c2bd4f1-736b-4589-98c3-db4b057ca99b'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'd5299006144f4ee6817349348a489017', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': 'd5299006144f4ee6817349348a489017', 'metadata': {}, 'qos_specs': None, 'retry': {'num_attempts': 2, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']}}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(dfd745ed-b0c4-4e35-b77c-a911c90ce33e),volume_id=dfd745ed-b0c4-4e35-b77c-a911c90ce33e,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:44:38Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-CreateVolumesFromImageTest-Volume-1463431787',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=dfd745ed-b0c4-4e35-b77c-a911c90ce33e,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='75babbbca67f4035a48ee8ee816b18db',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:46:58Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:46:58Z,use_quota=True,user_id='d5299006144f4ee6817349348a489017',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 15:49:16 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:44:38Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-CreateVolumesFromImageTest-Volume-1463431787',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=dfd745ed-b0c4-4e35-b77c-a911c90ce33e,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='75babbbca67f4035a48ee8ee816b18db',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:46:58Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:46:58Z,use_quota=True,user_id='d5299006144f4ee6817349348a489017',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:44:38Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-CreateVolumesFromImageTest-Volume-1463431787',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=dfd745ed-b0c4-4e35-b77c-a911c90ce33e,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='75babbbca67f4035a48ee8ee816b18db',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:46:58Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:46:58Z,use_quota=True,user_id='d5299006144f4ee6817349348a489017',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 15:49:16 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 15:49:16 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 15:49:16 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:49:16 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 15:49:16 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 15:49:16 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 15:49:16 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 15:49:16 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 15:49:16 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 15:49:16 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 15:49:16 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 15:49:16 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 15:49:16 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 15:49:16 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 15:49:16 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 15:49:16 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 15:49:16 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 15:49:16 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 15:49:16 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 15:49:16 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 15:49:16 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (be26eb99-6fec-4d2f-ad74-1ffbc1521d7e) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:49:16 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (be26eb99-6fec-4d2f-ad74-1ffbc1521d7e) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:49:16 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (674f7152-823d-456b-a444-a8406df5280b) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:49:16 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (674f7152-823d-456b-a444-a8406df5280b) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:49:16 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (cedda478-02cf-4380-b073-53be7ff18594) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:49:16 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (cedda478-02cf-4380-b073-53be7ff18594) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:49:16 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (0da1616d-8a2c-46bb-aea1-8287b7af25cf) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:49:16 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Updating volume dfd745ed-b0c4-4e35-b77c-a911c90ce33e with {'scheduled_at': datetime.datetime(2023, 4, 17, 15, 49, 16, 588398), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 15:49:16 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Volume aea45c70-7c05-4146-847e-cbb777d3dfbe: re-scheduling SchedulerAPI.create_volume attempt 2 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 15:49:16 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Volume dfd745ed-b0c4-4e35-b77c-a911c90ce33e: re-scheduling SchedulerAPI.create_volume attempt 2 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 15:49:16 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Volume fbbadf16-651a-47d5-b308-a37e56aa07f2: re-scheduling SchedulerAPI.create_volume attempt 2 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 15:49:16 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Volume aea45c70-7c05-4146-847e-cbb777d3dfbe: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 15:49:16 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] deleating volume v_aea45c70-7c05-4146-847e-cbb777d3dfbe {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 15:49:16 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] get snapshots for volume v_aea45c70-7c05-4146-847e-cbb777d3dfbe {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 15:49:16 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_aea45c70-7c05-4146-847e-cbb777d3dfbe/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:49:16 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_aea45c70-7c05-4146-847e-cbb777d3dfbe/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:49:16 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Volume dfd745ed-b0c4-4e35-b77c-a911c90ce33e: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 15:49:16 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] deleating volume v_dfd745ed-b0c4-4e35-b77c-a911c90ce33e {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 15:49:16 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] get snapshots for volume v_dfd745ed-b0c4-4e35-b77c-a911c90ce33e {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 15:49:16 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_dfd745ed-b0c4-4e35-b77c-a911c90ce33e/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:49:16 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_dfd745ed-b0c4-4e35-b77c-a911c90ce33e/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:49:16 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Volume fbbadf16-651a-47d5-b308-a37e56aa07f2: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 15:49:16 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] deleating volume v_fbbadf16-651a-47d5-b308-a37e56aa07f2 {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 15:49:16 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] get snapshots for volume v_fbbadf16-651a-47d5-b308-a37e56aa07f2 {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 15:49:16 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_fbbadf16-651a-47d5-b308-a37e56aa07f2/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:49:16 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_fbbadf16-651a-47d5-b308-a37e56aa07f2/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:49:16 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Volume reschedule parameters: True retry: {'num_attempts': 3, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 15:49:16 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Flow 'volume_create_manager' (0671f5b5-330e-4c8a-9cbf-9a77a8054324) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:49:16 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (6afe5bba-b183-49c9-80a8-fd35dab8ea03) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:49:16 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (6afe5bba-b183-49c9-80a8-fd35dab8ea03) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:44:39Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-CreateVolumesFromImageTest-Volume-800604710',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=aea45c70-7c05-4146-847e-cbb777d3dfbe,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='75babbbca67f4035a48ee8ee816b18db',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:49:17Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:49:17Z,use_quota=True,user_id='d5299006144f4ee6817349348a489017',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:49:16 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (86606587-6131-4612-b937-e2c0d95449ef) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:49:16 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (86606587-6131-4612-b937-e2c0d95449ef) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:49:16 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (8b8e51f7-7191-4940-b14c-7b43492c1509) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:49:16 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (8b8e51f7-7191-4940-b14c-7b43492c1509) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': 'aea45c70-7c05-4146-847e-cbb777d3dfbe', 'volume_name': 'volume-aea45c70-7c05-4146-847e-cbb777d3dfbe', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:49:16 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (cbe2b0c6-a93e-4dab-a0a2-6ae209ccea00) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:49:16 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (cbe2b0c6-a93e-4dab-a0a2-6ae209ccea00) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:49:16 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (4681bef3-39fb-4da6-8f26-0a7615c4b427) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:49:16 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Volume aea45c70-7c05-4146-847e-cbb777d3dfbe: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-aea45c70-7c05-4146-847e-cbb777d3dfbe', 'volume_size': 1} Apr 17 15:49:16 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] creating volume v_aea45c70-7c05-4146-847e-cbb777d3dfbe. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 15:49:16 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] create volume {'name': 'v_aea45c70-7c05-4146-847e-cbb777d3dfbe', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 15:49:16 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:49:16 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:49:16 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Volume reschedule parameters: True retry: {'num_attempts': 3, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 15:49:16 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Flow 'volume_create_manager' (5d8d0a95-6117-4590-9a91-1e9b6fe32bbf) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:49:16 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (e1d68394-eeed-45b1-b8b8-6c999c8fede3) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:49:16 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Volume reschedule parameters: True retry: {'num_attempts': 3, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 15:49:16 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Flow 'volume_create_manager' (9ffa0202-8736-4944-9017-f7af69c224fa) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:49:16 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (41666f3e-e2c3-4d4e-a469-9a5d210a57e9) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:49:16 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (e1d68394-eeed-45b1-b8b8-6c999c8fede3) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:44:39Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-CreateVolumesFromImageTest-Volume-2146530532',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=fbbadf16-651a-47d5-b308-a37e56aa07f2,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='75babbbca67f4035a48ee8ee816b18db',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:49:17Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:49:17Z,use_quota=True,user_id='d5299006144f4ee6817349348a489017',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:49:16 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (0a72d5cb-ba2d-4dde-a6c4-034bae01c38e) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:49:16 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (0a72d5cb-ba2d-4dde-a6c4-034bae01c38e) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:49:16 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (5fd6737e-92c8-44b2-9cf9-05fd83bdd49d) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:49:16 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (5fd6737e-92c8-44b2-9cf9-05fd83bdd49d) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': 'fbbadf16-651a-47d5-b308-a37e56aa07f2', 'volume_name': 'volume-fbbadf16-651a-47d5-b308-a37e56aa07f2', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:49:16 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (b9897d71-e285-4657-a1e8-85f152afdcf0) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:49:16 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (b9897d71-e285-4657-a1e8-85f152afdcf0) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:49:16 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (56e4cca7-6494-499c-96e7-627fee9eb451) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:49:16 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Volume fbbadf16-651a-47d5-b308-a37e56aa07f2: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-fbbadf16-651a-47d5-b308-a37e56aa07f2', 'volume_size': 1} Apr 17 15:49:16 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] creating volume v_fbbadf16-651a-47d5-b308-a37e56aa07f2. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 15:49:16 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] create volume {'name': 'v_fbbadf16-651a-47d5-b308-a37e56aa07f2', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 15:49:16 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:49:16 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:49:17 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (41666f3e-e2c3-4d4e-a469-9a5d210a57e9) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:44:38Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-CreateVolumesFromImageTest-Volume-1463431787',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=dfd745ed-b0c4-4e35-b77c-a911c90ce33e,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='75babbbca67f4035a48ee8ee816b18db',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:49:17Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:49:17Z,use_quota=True,user_id='d5299006144f4ee6817349348a489017',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:49:17 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (68d65eb2-d76e-4966-8ef0-9d655bd4d1a2) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:49:17 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (68d65eb2-d76e-4966-8ef0-9d655bd4d1a2) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:49:17 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (a4663525-c05a-4433-b540-b8941b417aca) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:49:17 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (a4663525-c05a-4433-b540-b8941b417aca) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': 'dfd745ed-b0c4-4e35-b77c-a911c90ce33e', 'volume_name': 'volume-dfd745ed-b0c4-4e35-b77c-a911c90ce33e', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:49:17 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (7418534d-6ff2-442f-ac11-83607405a06c) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:49:17 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (7418534d-6ff2-442f-ac11-83607405a06c) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:49:17 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (46a7e832-4174-4a49-9cc7-2aa79369b69a) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:49:17 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Volume dfd745ed-b0c4-4e35-b77c-a911c90ce33e: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-dfd745ed-b0c4-4e35-b77c-a911c90ce33e', 'volume_size': 1} Apr 17 15:49:17 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] creating volume v_dfd745ed-b0c4-4e35-b77c-a911c90ce33e. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 15:49:17 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] create volume {'name': 'v_dfd745ed-b0c4-4e35-b77c-a911c90ce33e', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 15:49:17 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:49:17 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:49:19 user cinder-volume[84696]: DEBUG cinder.utils [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.777(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:49:19 user cinder-volume[84696]: DEBUG cinder.utils [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_aea45c70-7c05-4146-847e-cbb777d3dfbe/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:49:19 user cinder-volume[84696]: DEBUG cinder.utils [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 139.019(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:49:19 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_ab5889f8-b6c7-4673-a666-f4b4e5ff5fd5/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:49:19 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (34f4f4ff-2793-4469-b09b-4747e98a8a60) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 15:49:19 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (3086b358-7bdb-42d8-83a9-53a9139ba127) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:49:19 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (3086b358-7bdb-42d8-83a9-53a9139ba127) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:49:19 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Flow 'volume_create_manager' (b73b5d1b-f361-4c7c-b68e-e1a4dfc5a76a) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 15:49:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 15:49:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 15:49:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 15:49:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 15:49:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 15:49:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 15:49:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 15:49:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 15:49:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 15:49:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 15:49:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 15:49:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 15:49:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 15:49:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 15:49:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 15:49:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 15:49:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 15:49:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 15:49:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 15:49:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 15:49:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 15:49:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 15:49:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 15:49:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 15:49:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:49:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 15:49:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 15:49:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 15:49:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 15:49:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 15:49:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 15:49:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 15:49:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 15:49:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 15:49:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 15:49:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 15:49:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 15:49:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 15:49:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 15:49:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 15:49:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 15:49:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 15:49:19 user cinder-volume[84696]: DEBUG cinder.utils [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 138.539(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:49:19 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:49:19 user cinder-volume[84696]: INFO cinder.message.api [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Creating message record for request_id = req-27d7b7b3-4675-424c-af48-cc2422ffc813 Apr 17 15:49:19 user cinder-volume[84696]: DEBUG cinder.utils [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.782(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:49:19 user cinder-volume[84696]: DEBUG cinder.utils [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_dfd745ed-b0c4-4e35-b77c-a911c90ce33e/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:49:19 user cinder-volume[84696]: DEBUG cinder.utils [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 139.033(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:49:19 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_13cc73df-7780-4934-8cbd-7f372b1cf57a/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:49:19 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (182b3545-4d79-41e7-8994-9e2a802d1769) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 15:49:19 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (f29bb2ab-3b18-4e88-81f3-09c13e394081) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:49:19 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (f29bb2ab-3b18-4e88-81f3-09c13e394081) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:49:19 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Flow 'volume_create_manager' (70451a17-1a5a-4c30-9d62-5035d2806cfe) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 15:49:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 15:49:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 15:49:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 15:49:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 15:49:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 15:49:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 15:49:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 15:49:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 15:49:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 15:49:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 15:49:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 15:49:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 15:49:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 15:49:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 15:49:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 15:49:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 15:49:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 15:49:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 15:49:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 15:49:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 15:49:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 15:49:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 15:49:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 15:49:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 15:49:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:49:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 15:49:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 15:49:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 15:49:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 15:49:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 15:49:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 15:49:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 15:49:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 15:49:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 15:49:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 15:49:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 15:49:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 15:49:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 15:49:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 15:49:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 15:49:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 15:49:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 15:49:19 user cinder-volume[84696]: DEBUG cinder.utils [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 138.464(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:49:19 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:49:19 user cinder-volume[84696]: INFO cinder.message.api [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Creating message record for request_id = req-9b871d6d-58e3-4523-b328-b6ffa7aca14e Apr 17 15:49:19 user cinder-volume[84696]: DEBUG cinder.utils [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 73.019(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:49:19 user cinder-volume[84696]: DEBUG cinder.utils [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_93a32a7c-4c6a-472a-8f91-9d34558ba232/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:49:19 user cinder-volume[84696]: DEBUG cinder.utils [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.579(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:49:19 user cinder-volume[84696]: DEBUG cinder.utils [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:49:19 user cinder-volume[84696]: DEBUG cinder.utils [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 3.875(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:49:19 user cinder-volume[84696]: DEBUG cinder.utils [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:49:19 user cinder-volume[84696]: DEBUG cinder.utils [req-778c62be-8963-4e6d-97f1-f456b759ce16 req-d9cf97f7-129f-4cb2-9bc2-f51b1339159a tempest-AttachVolumeShelveTestJSON-1745617049 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 70.458(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:49:19 user cinder-volume[84696]: DEBUG cinder.utils [req-778c62be-8963-4e6d-97f1-f456b759ce16 req-d9cf97f7-129f-4cb2-9bc2-f51b1339159a tempest-AttachVolumeShelveTestJSON-1745617049 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:13bf568b-585c-4556-ab78-e9930ad50214/luns/v_13bf568b-585c-4556-ab78-e9930ad50214 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:49:19 user cinder-volume[84696]: WARNING urllib3.connectionpool [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Connection pool is full, discarding connection: 172.16.0.220. Connection pool size: 10: queue.Full Apr 17 15:49:19 user cinder-volume[84696]: DEBUG cinder.utils [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 72.708(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:49:19 user cinder-volume[84696]: DEBUG cinder.utils [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:49:19 user cinder-volume[84696]: WARNING urllib3.connectionpool [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Connection pool is full, discarding connection: 172.16.0.220. Connection pool size: 10: queue.Full Apr 17 15:49:19 user cinder-volume[84696]: DEBUG cinder.utils [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.802(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:49:19 user cinder-volume[84696]: DEBUG cinder.utils [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_fbbadf16-651a-47d5-b308-a37e56aa07f2/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:49:19 user cinder-volume[84696]: WARNING urllib3.connectionpool [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Connection pool is full, discarding connection: 172.16.0.220. Connection pool size: 10: queue.Full Apr 17 15:49:19 user cinder-volume[84696]: DEBUG cinder.utils [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.436(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:49:19 user cinder-volume[84696]: DEBUG cinder.utils [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:49:19 user cinder-volume[84696]: WARNING urllib3.connectionpool [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Connection pool is full, discarding connection: 172.16.0.220. Connection pool size: 10: queue.Full Apr 17 15:49:19 user cinder-volume[84696]: DEBUG cinder.utils [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.479(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:49:19 user cinder-volume[84696]: DEBUG cinder.utils [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:49:19 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 15:49:19 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 15:49:19 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (62b34036-79c9-4679-bc37-81e483756477) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 15:49:19 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 15:49:19 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:44:39Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-CreateVolumesFromImageTest-Volume-1776307206',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=13cc73df-7780-4934-8cbd-7f372b1cf57a,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='75babbbca67f4035a48ee8ee816b18db',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:47:01Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:47:01Z,use_quota=True,user_id='d5299006144f4ee6817349348a489017',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 15:49:19 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:44:39Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-CreateVolumesFromImageTest-Volume-1776307206',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=13cc73df-7780-4934-8cbd-7f372b1cf57a,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='75babbbca67f4035a48ee8ee816b18db',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:47:01Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:47:01Z,use_quota=True,user_id='d5299006144f4ee6817349348a489017',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(13cc73df-7780-4934-8cbd-7f372b1cf57a),volume_id=13cc73df-7780-4934-8cbd-7f372b1cf57a,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': '13cc73df-7780-4934-8cbd-7f372b1cf57a', 'volume_name': 'volume-13cc73df-7780-4934-8cbd-7f372b1cf57a', 'volume_size': 1}} Apr 17 15:49:19 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': '13cc73df-7780-4934-8cbd-7f372b1cf57a', 'volume': {'id': '13cc73df-7780-4934-8cbd-7f372b1cf57a', '_name_id': None, 'ec2_id': None, 'user_id': 'd5299006144f4ee6817349348a489017', 'project_id': '75babbbca67f4035a48ee8ee816b18db', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-CreateVolumesFromImageTest-Volume-1776307206', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'admin_metadata': {}, 'glance_metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T15:44:39.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-13cc73df-7780-4934-8cbd-7f372b1cf57a', 'name_id': '13cc73df-7780-4934-8cbd-7f372b1cf57a', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-CreateVolumesFromImageTest-Volume-1776307206', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '75babbbca67f4035a48ee8ee816b18db', 'qos_specs': None, 'reservations': ['a8c0981a-3509-44ee-887e-5a4c15f5fc31', 'be48fc25-1b54-4ebf-aef6-6988c7432c47', '45f69ba8-0c40-4500-b132-878446677716', '5143bd10-b0c0-4336-a02c-d9f5b822c37b'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'd5299006144f4ee6817349348a489017', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-CreateVolumesFromImageTest-Volume-1776307206', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '75babbbca67f4035a48ee8ee816b18db', 'qos_specs': None, 'reservations': ['a8c0981a-3509-44ee-887e-5a4c15f5fc31', 'be48fc25-1b54-4ebf-aef6-6988c7432c47', '45f69ba8-0c40-4500-b132-878446677716', '5143bd10-b0c0-4336-a02c-d9f5b822c37b'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'd5299006144f4ee6817349348a489017', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': 'd5299006144f4ee6817349348a489017', 'metadata': {}, 'qos_specs': None, 'retry': {'num_attempts': 2, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']}}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(13cc73df-7780-4934-8cbd-7f372b1cf57a),volume_id=13cc73df-7780-4934-8cbd-7f372b1cf57a,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:44:39Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-CreateVolumesFromImageTest-Volume-1776307206',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=13cc73df-7780-4934-8cbd-7f372b1cf57a,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='75babbbca67f4035a48ee8ee816b18db',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:47:01Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:47:01Z,use_quota=True,user_id='d5299006144f4ee6817349348a489017',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 15:49:19 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:44:39Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-CreateVolumesFromImageTest-Volume-1776307206',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=13cc73df-7780-4934-8cbd-7f372b1cf57a,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='75babbbca67f4035a48ee8ee816b18db',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:47:01Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:47:01Z,use_quota=True,user_id='d5299006144f4ee6817349348a489017',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:44:39Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-CreateVolumesFromImageTest-Volume-1776307206',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=13cc73df-7780-4934-8cbd-7f372b1cf57a,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='75babbbca67f4035a48ee8ee816b18db',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:47:01Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:47:01Z,use_quota=True,user_id='d5299006144f4ee6817349348a489017',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 15:49:19 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 15:49:19 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 15:49:19 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:49:19 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 15:49:19 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 15:49:19 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 15:49:19 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 15:49:19 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 15:49:19 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 15:49:19 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 15:49:19 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 15:49:19 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 15:49:19 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 15:49:19 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 15:49:19 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 15:49:19 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 15:49:19 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 15:49:19 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 15:49:19 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 15:49:19 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 15:49:19 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (62b34036-79c9-4679-bc37-81e483756477) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:49:19 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (62b34036-79c9-4679-bc37-81e483756477) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:49:19 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (72aede64-9af0-4067-8946-dad96767f88d) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:49:19 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (72aede64-9af0-4067-8946-dad96767f88d) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:49:19 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (fc33ef75-005e-4d38-804e-1fcfa611f564) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:49:19 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (fc33ef75-005e-4d38-804e-1fcfa611f564) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:49:19 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (f9ed0951-88b0-4fb9-aaab-6be33f2e9366) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:49:19 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Updating volume 13cc73df-7780-4934-8cbd-7f372b1cf57a with {'scheduled_at': datetime.datetime(2023, 4, 17, 15, 49, 19, 532900), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 15:49:19 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (79c3e531-6757-4e5a-ab4f-9b1fed099fd3) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 15:49:19 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 15:49:19 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:44:39Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-CreateVolumesFromImageTest-Volume-1478171265',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=ab5889f8-b6c7-4673-a666-f4b4e5ff5fd5,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='75babbbca67f4035a48ee8ee816b18db',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:47:01Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:47:01Z,use_quota=True,user_id='d5299006144f4ee6817349348a489017',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 15:49:19 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:44:39Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-CreateVolumesFromImageTest-Volume-1478171265',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=ab5889f8-b6c7-4673-a666-f4b4e5ff5fd5,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='75babbbca67f4035a48ee8ee816b18db',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:47:01Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:47:01Z,use_quota=True,user_id='d5299006144f4ee6817349348a489017',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(ab5889f8-b6c7-4673-a666-f4b4e5ff5fd5),volume_id=ab5889f8-b6c7-4673-a666-f4b4e5ff5fd5,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': 'ab5889f8-b6c7-4673-a666-f4b4e5ff5fd5', 'volume_name': 'volume-ab5889f8-b6c7-4673-a666-f4b4e5ff5fd5', 'volume_size': 1}} Apr 17 15:49:19 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': 'ab5889f8-b6c7-4673-a666-f4b4e5ff5fd5', 'volume': {'id': 'ab5889f8-b6c7-4673-a666-f4b4e5ff5fd5', '_name_id': None, 'ec2_id': None, 'user_id': 'd5299006144f4ee6817349348a489017', 'project_id': '75babbbca67f4035a48ee8ee816b18db', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-CreateVolumesFromImageTest-Volume-1478171265', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'admin_metadata': {}, 'glance_metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T15:44:39.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-ab5889f8-b6c7-4673-a666-f4b4e5ff5fd5', 'name_id': 'ab5889f8-b6c7-4673-a666-f4b4e5ff5fd5', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-CreateVolumesFromImageTest-Volume-1478171265', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '75babbbca67f4035a48ee8ee816b18db', 'qos_specs': None, 'reservations': ['a64639ba-df1a-40e6-bded-21cad01b745f', '4323d77e-0f64-483e-8433-0f03b9e38ddd', '7b2c692c-9fb9-4f7b-9b9c-94383941c4df', '45cf8435-1a25-41db-aac5-ff8bf8698729'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'd5299006144f4ee6817349348a489017', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-CreateVolumesFromImageTest-Volume-1478171265', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '75babbbca67f4035a48ee8ee816b18db', 'qos_specs': None, 'reservations': ['a64639ba-df1a-40e6-bded-21cad01b745f', '4323d77e-0f64-483e-8433-0f03b9e38ddd', '7b2c692c-9fb9-4f7b-9b9c-94383941c4df', '45cf8435-1a25-41db-aac5-ff8bf8698729'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'd5299006144f4ee6817349348a489017', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': 'd5299006144f4ee6817349348a489017', 'metadata': {}, 'qos_specs': None, 'retry': {'num_attempts': 2, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']}}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(ab5889f8-b6c7-4673-a666-f4b4e5ff5fd5),volume_id=ab5889f8-b6c7-4673-a666-f4b4e5ff5fd5,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:44:39Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-CreateVolumesFromImageTest-Volume-1478171265',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=ab5889f8-b6c7-4673-a666-f4b4e5ff5fd5,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='75babbbca67f4035a48ee8ee816b18db',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:47:01Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:47:01Z,use_quota=True,user_id='d5299006144f4ee6817349348a489017',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 15:49:19 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:44:39Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-CreateVolumesFromImageTest-Volume-1478171265',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=ab5889f8-b6c7-4673-a666-f4b4e5ff5fd5,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='75babbbca67f4035a48ee8ee816b18db',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:47:01Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:47:01Z,use_quota=True,user_id='d5299006144f4ee6817349348a489017',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:44:39Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-CreateVolumesFromImageTest-Volume-1478171265',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=ab5889f8-b6c7-4673-a666-f4b4e5ff5fd5,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='75babbbca67f4035a48ee8ee816b18db',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:47:01Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:47:01Z,use_quota=True,user_id='d5299006144f4ee6817349348a489017',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 15:49:19 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 15:49:19 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 15:49:19 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:49:19 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 15:49:19 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 15:49:19 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 15:49:19 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 15:49:19 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 15:49:19 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 15:49:19 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 15:49:19 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 15:49:19 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 15:49:19 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 15:49:19 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 15:49:19 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 15:49:19 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 15:49:19 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 15:49:19 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 15:49:19 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 15:49:19 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 15:49:19 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (79c3e531-6757-4e5a-ab4f-9b1fed099fd3) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:49:19 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (79c3e531-6757-4e5a-ab4f-9b1fed099fd3) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:49:19 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (323a2d66-260c-4d6f-bd28-1732b006d05d) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:49:19 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (323a2d66-260c-4d6f-bd28-1732b006d05d) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:49:19 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (b4424a4c-7b95-41aa-9426-18ccacfa2eee) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:49:19 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (b4424a4c-7b95-41aa-9426-18ccacfa2eee) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:49:19 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (ed61cc6c-4259-46de-8e73-3b5f80096571) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:49:19 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Updating volume ab5889f8-b6c7-4673-a666-f4b4e5ff5fd5 with {'scheduled_at': datetime.datetime(2023, 4, 17, 15, 49, 19, 579306), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 15:49:19 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Volume 13cc73df-7780-4934-8cbd-7f372b1cf57a: re-scheduling SchedulerAPI.create_volume attempt 2 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 15:49:19 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Volume ab5889f8-b6c7-4673-a666-f4b4e5ff5fd5: re-scheduling SchedulerAPI.create_volume attempt 2 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 15:49:19 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Volume 13cc73df-7780-4934-8cbd-7f372b1cf57a: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 15:49:19 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] deleating volume v_13cc73df-7780-4934-8cbd-7f372b1cf57a {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 15:49:19 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] get snapshots for volume v_13cc73df-7780-4934-8cbd-7f372b1cf57a {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 15:49:19 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_13cc73df-7780-4934-8cbd-7f372b1cf57a/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:49:19 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_13cc73df-7780-4934-8cbd-7f372b1cf57a/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:49:19 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Volume ab5889f8-b6c7-4673-a666-f4b4e5ff5fd5: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 15:49:19 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] deleating volume v_ab5889f8-b6c7-4673-a666-f4b4e5ff5fd5 {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 15:49:19 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] get snapshots for volume v_ab5889f8-b6c7-4673-a666-f4b4e5ff5fd5 {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 15:49:19 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_ab5889f8-b6c7-4673-a666-f4b4e5ff5fd5/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:49:19 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_ab5889f8-b6c7-4673-a666-f4b4e5ff5fd5/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:49:19 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Volume reschedule parameters: True retry: {'num_attempts': 3, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 15:49:19 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Flow 'volume_create_manager' (30ac04b8-ce96-4283-ae25-58b4c523da77) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:49:19 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (5951d843-eff2-4552-a221-3aec16358929) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:49:19 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (5951d843-eff2-4552-a221-3aec16358929) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:44:39Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-CreateVolumesFromImageTest-Volume-1478171265',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=ab5889f8-b6c7-4673-a666-f4b4e5ff5fd5,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='75babbbca67f4035a48ee8ee816b18db',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:49:20Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:49:20Z,use_quota=True,user_id='d5299006144f4ee6817349348a489017',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:49:19 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (d63d7778-b87e-4d8f-b05d-c0bc4e4f2689) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:49:19 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (d63d7778-b87e-4d8f-b05d-c0bc4e4f2689) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:49:19 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (ffd36bbb-4c10-4144-b23b-1a77d8c93778) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:49:19 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (ffd36bbb-4c10-4144-b23b-1a77d8c93778) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': 'ab5889f8-b6c7-4673-a666-f4b4e5ff5fd5', 'volume_name': 'volume-ab5889f8-b6c7-4673-a666-f4b4e5ff5fd5', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:49:19 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (25c63d92-de74-424e-a068-d6dca2250db4) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:49:19 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (25c63d92-de74-424e-a068-d6dca2250db4) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:49:19 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (671165ad-ebcd-4131-93e4-5e514a6552b3) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:49:19 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Volume ab5889f8-b6c7-4673-a666-f4b4e5ff5fd5: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-ab5889f8-b6c7-4673-a666-f4b4e5ff5fd5', 'volume_size': 1} Apr 17 15:49:19 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] creating volume v_ab5889f8-b6c7-4673-a666-f4b4e5ff5fd5. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 15:49:19 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] create volume {'name': 'v_ab5889f8-b6c7-4673-a666-f4b4e5ff5fd5', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 15:49:19 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:49:19 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:49:19 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Volume reschedule parameters: True retry: {'num_attempts': 3, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 15:49:19 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Flow 'volume_create_manager' (f8b288af-2482-48ea-9d14-205bbbad1b17) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:49:19 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (712112e0-a44e-40c2-b397-553a821b91f0) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:49:19 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (712112e0-a44e-40c2-b397-553a821b91f0) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:44:39Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-CreateVolumesFromImageTest-Volume-1776307206',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=13cc73df-7780-4934-8cbd-7f372b1cf57a,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='75babbbca67f4035a48ee8ee816b18db',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:49:20Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:49:20Z,use_quota=True,user_id='d5299006144f4ee6817349348a489017',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:49:19 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (f20c37de-84d6-484b-8000-206b81409724) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:49:19 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (f20c37de-84d6-484b-8000-206b81409724) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:49:19 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (f5b299f1-d02a-4c3e-93b7-2ca59c3e291e) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:49:19 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (f5b299f1-d02a-4c3e-93b7-2ca59c3e291e) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': '13cc73df-7780-4934-8cbd-7f372b1cf57a', 'volume_name': 'volume-13cc73df-7780-4934-8cbd-7f372b1cf57a', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:49:19 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (2d570206-ca36-4c07-9215-efee2f5bb1ae) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:49:19 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (2d570206-ca36-4c07-9215-efee2f5bb1ae) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:49:20 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (36177b7f-40f9-4c25-9f43-6e183241c50c) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:49:20 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Volume 13cc73df-7780-4934-8cbd-7f372b1cf57a: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-13cc73df-7780-4934-8cbd-7f372b1cf57a', 'volume_size': 1} Apr 17 15:49:20 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] creating volume v_13cc73df-7780-4934-8cbd-7f372b1cf57a. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 15:49:20 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] create volume {'name': 'v_13cc73df-7780-4934-8cbd-7f372b1cf57a', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 15:49:20 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:49:20 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:49:22 user cinder-volume[84696]: DEBUG cinder.utils [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.870(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:49:22 user cinder-volume[84696]: DEBUG cinder.utils [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_ab5889f8-b6c7-4673-a666-f4b4e5ff5fd5/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:49:22 user cinder-volume[84696]: DEBUG cinder.utils [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 138.865(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:49:22 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:49:22 user cinder-volume[84696]: INFO cinder.message.api [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Creating message record for request_id = req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 Apr 17 15:49:22 user cinder-volume[84696]: DEBUG cinder.utils [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.886(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:49:22 user cinder-volume[84696]: DEBUG cinder.utils [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_13cc73df-7780-4934-8cbd-7f372b1cf57a/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:49:22 user cinder-volume[84696]: DEBUG cinder.utils [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 139.048(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:49:22 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_68f814b0-1291-4a12-ae65-dc0892203d36/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:49:22 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (78cd626e-1d4d-4838-8d0e-2d6b218f83b8) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 15:49:22 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (ab78d32b-45bb-465a-9c4d-e05cc2e00585) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:49:22 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (ab78d32b-45bb-465a-9c4d-e05cc2e00585) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:49:22 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Flow 'volume_create_manager' (c06604dc-2077-4b09-9327-56c070b16b86) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 15:49:22 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 15:49:22 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 15:49:22 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 15:49:22 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 15:49:22 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 15:49:22 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 15:49:22 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 15:49:22 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 15:49:22 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 15:49:22 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 15:49:22 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 15:49:22 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 15:49:22 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 15:49:22 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 15:49:22 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 15:49:22 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 15:49:22 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 15:49:22 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 15:49:22 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 15:49:22 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 15:49:22 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 15:49:22 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 15:49:22 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 15:49:22 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 15:49:22 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:49:22 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 15:49:22 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 15:49:22 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 15:49:22 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 15:49:22 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 15:49:22 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 15:49:22 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 15:49:22 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 15:49:22 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 15:49:22 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 15:49:22 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 15:49:22 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 15:49:22 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 15:49:22 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 15:49:22 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 15:49:22 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 15:49:22 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 15:49:22 user cinder-volume[84696]: DEBUG cinder.utils [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.877(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:49:22 user cinder-volume[84696]: DEBUG cinder.utils [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_aea45c70-7c05-4146-847e-cbb777d3dfbe/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:49:22 user cinder-volume[84696]: DEBUG cinder.utils [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.524(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:49:22 user cinder-volume[84696]: DEBUG cinder.utils [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:49:22 user cinder-volume[84696]: DEBUG cinder.utils [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.519(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:49:22 user cinder-volume[84696]: DEBUG cinder.utils [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:49:22 user cinder-volume[84696]: DEBUG cinder.utils [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.636(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:49:22 user cinder-volume[84696]: DEBUG cinder.utils [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:49:22 user cinder-volume[84696]: DEBUG cinder.utils [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.854(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:49:22 user cinder-volume[84696]: DEBUG cinder.utils [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_fbbadf16-651a-47d5-b308-a37e56aa07f2/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:49:22 user cinder-volume[84696]: DEBUG cinder.utils [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.488(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:49:22 user cinder-volume[84696]: DEBUG cinder.utils [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:49:22 user cinder-volume[84696]: WARNING urllib3.connectionpool [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Connection pool is full, discarding connection: 172.16.0.220. Connection pool size: 10: queue.Full Apr 17 15:49:22 user cinder-volume[84696]: DEBUG cinder.utils [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.595(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:49:22 user cinder-volume[84696]: DEBUG cinder.utils [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:49:22 user cinder-volume[84696]: WARNING urllib3.connectionpool [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Connection pool is full, discarding connection: 172.16.0.220. Connection pool size: 10: queue.Full Apr 17 15:49:22 user cinder-volume[84696]: DEBUG cinder.utils [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.866(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:49:22 user cinder-volume[84696]: DEBUG cinder.utils [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_dfd745ed-b0c4-4e35-b77c-a911c90ce33e/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:49:22 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 15:49:22 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (da83f184-2ca5-4d3d-999d-9dff082efa0b) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 15:49:22 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 15:49:22 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:44:42Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-ServerRescueNegativeTestJSON-volume-1878278176',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=68f814b0-1291-4a12-ae65-dc0892203d36,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='9324c99bd4d040bdb5a7f1db782b2783',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:47:04Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:47:04Z,use_quota=True,user_id='8e3487d6b32d46beac556617050fd664',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 15:49:22 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:44:42Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-ServerRescueNegativeTestJSON-volume-1878278176',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=68f814b0-1291-4a12-ae65-dc0892203d36,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='9324c99bd4d040bdb5a7f1db782b2783',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:47:04Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:47:04Z,use_quota=True,user_id='8e3487d6b32d46beac556617050fd664',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(68f814b0-1291-4a12-ae65-dc0892203d36),volume_id=68f814b0-1291-4a12-ae65-dc0892203d36,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': '68f814b0-1291-4a12-ae65-dc0892203d36', 'volume_name': 'volume-68f814b0-1291-4a12-ae65-dc0892203d36', 'volume_size': 1}} Apr 17 15:49:22 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': '68f814b0-1291-4a12-ae65-dc0892203d36', 'volume': {'id': '68f814b0-1291-4a12-ae65-dc0892203d36', '_name_id': None, 'ec2_id': None, 'user_id': '8e3487d6b32d46beac556617050fd664', 'project_id': '9324c99bd4d040bdb5a7f1db782b2783', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-ServerRescueNegativeTestJSON-volume-1878278176', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'admin_metadata': {}, 'glance_metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T15:44:42.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-68f814b0-1291-4a12-ae65-dc0892203d36', 'name_id': '68f814b0-1291-4a12-ae65-dc0892203d36', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-ServerRescueNegativeTestJSON-volume-1878278176', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '9324c99bd4d040bdb5a7f1db782b2783', 'qos_specs': None, 'reservations': ['2f994b27-d987-48a4-ad61-f3ddbb20a152', 'b47baa6b-4689-4254-948c-f073028527b3', '59e249a4-01b6-44aa-a2e4-029e54867201', '148031dd-8a91-4041-9ef3-af4b5a029444'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': '8e3487d6b32d46beac556617050fd664', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-ServerRescueNegativeTestJSON-volume-1878278176', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '9324c99bd4d040bdb5a7f1db782b2783', 'qos_specs': None, 'reservations': ['2f994b27-d987-48a4-ad61-f3ddbb20a152', 'b47baa6b-4689-4254-948c-f073028527b3', '59e249a4-01b6-44aa-a2e4-029e54867201', '148031dd-8a91-4041-9ef3-af4b5a029444'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': '8e3487d6b32d46beac556617050fd664', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': '8e3487d6b32d46beac556617050fd664', 'metadata': {}, 'qos_specs': None, 'retry': {'num_attempts': 2, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']}}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(68f814b0-1291-4a12-ae65-dc0892203d36),volume_id=68f814b0-1291-4a12-ae65-dc0892203d36,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:44:42Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-ServerRescueNegativeTestJSON-volume-1878278176',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=68f814b0-1291-4a12-ae65-dc0892203d36,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='9324c99bd4d040bdb5a7f1db782b2783',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:47:04Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:47:04Z,use_quota=True,user_id='8e3487d6b32d46beac556617050fd664',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 15:49:22 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:44:42Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-ServerRescueNegativeTestJSON-volume-1878278176',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=68f814b0-1291-4a12-ae65-dc0892203d36,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='9324c99bd4d040bdb5a7f1db782b2783',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:47:04Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:47:04Z,use_quota=True,user_id='8e3487d6b32d46beac556617050fd664',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:44:42Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-ServerRescueNegativeTestJSON-volume-1878278176',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=68f814b0-1291-4a12-ae65-dc0892203d36,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='9324c99bd4d040bdb5a7f1db782b2783',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:47:04Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:47:04Z,use_quota=True,user_id='8e3487d6b32d46beac556617050fd664',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 15:49:22 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 15:49:22 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 15:49:22 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:49:22 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 15:49:22 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 15:49:22 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 15:49:22 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 15:49:22 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 15:49:22 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 15:49:22 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 15:49:22 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 15:49:22 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 15:49:22 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 15:49:22 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 15:49:22 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 15:49:22 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 15:49:22 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 15:49:22 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 15:49:22 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 15:49:22 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 15:49:22 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (da83f184-2ca5-4d3d-999d-9dff082efa0b) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:49:22 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (da83f184-2ca5-4d3d-999d-9dff082efa0b) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:49:22 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (7d1a95d8-4a1e-4f1c-b49a-9dfe89751746) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:49:22 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (7d1a95d8-4a1e-4f1c-b49a-9dfe89751746) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:49:22 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (7cf39b0e-9ed4-4a29-98cc-a65170d845c4) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:49:22 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (7cf39b0e-9ed4-4a29-98cc-a65170d845c4) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:49:22 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (48c4ff00-053c-4a8b-8d01-bf8c90352d82) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:49:22 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Updating volume 68f814b0-1291-4a12-ae65-dc0892203d36 with {'scheduled_at': datetime.datetime(2023, 4, 17, 15, 49, 22, 598035), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 15:49:22 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Volume 68f814b0-1291-4a12-ae65-dc0892203d36: re-scheduling SchedulerAPI.create_volume attempt 2 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 15:49:22 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Volume 68f814b0-1291-4a12-ae65-dc0892203d36: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 15:49:22 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] deleating volume v_68f814b0-1291-4a12-ae65-dc0892203d36 {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 15:49:22 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] get snapshots for volume v_68f814b0-1291-4a12-ae65-dc0892203d36 {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 15:49:22 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_68f814b0-1291-4a12-ae65-dc0892203d36/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:49:22 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_68f814b0-1291-4a12-ae65-dc0892203d36/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:49:22 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Volume reschedule parameters: True retry: {'num_attempts': 3, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 15:49:22 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Flow 'volume_create_manager' (b6e14aff-6cac-440f-bb2e-0a23781024ab) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:49:22 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (1b0e124e-7a5a-4e51-93bb-74f1c5af474d) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:49:22 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (1b0e124e-7a5a-4e51-93bb-74f1c5af474d) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:44:42Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-ServerRescueNegativeTestJSON-volume-1878278176',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=68f814b0-1291-4a12-ae65-dc0892203d36,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='9324c99bd4d040bdb5a7f1db782b2783',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:49:23Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:49:23Z,use_quota=True,user_id='8e3487d6b32d46beac556617050fd664',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:49:22 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (d8132ee2-3794-49bc-8f00-ba6d46ed7cac) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:49:22 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (d8132ee2-3794-49bc-8f00-ba6d46ed7cac) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:49:22 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (2d709152-3d0a-456c-baa9-43704a23d108) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:49:22 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (2d709152-3d0a-456c-baa9-43704a23d108) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': '68f814b0-1291-4a12-ae65-dc0892203d36', 'volume_name': 'volume-68f814b0-1291-4a12-ae65-dc0892203d36', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:49:22 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (f9cd7bb4-0e3a-4851-b457-4e61c3a0f1e4) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:49:22 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (f9cd7bb4-0e3a-4851-b457-4e61c3a0f1e4) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:49:22 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (dbc74882-420b-4d9e-9ff9-82fe72629ce4) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:49:22 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Volume 68f814b0-1291-4a12-ae65-dc0892203d36: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-68f814b0-1291-4a12-ae65-dc0892203d36', 'volume_size': 1} Apr 17 15:49:22 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] creating volume v_68f814b0-1291-4a12-ae65-dc0892203d36. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 15:49:22 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] create volume {'name': 'v_68f814b0-1291-4a12-ae65-dc0892203d36', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 15:49:22 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:49:22 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:49:25 user cinder-volume[84696]: DEBUG cinder.utils [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.885(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:49:25 user cinder-volume[84696]: DEBUG cinder.utils [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_68f814b0-1291-4a12-ae65-dc0892203d36/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:49:25 user cinder-volume[84696]: DEBUG cinder.utils [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.706(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:49:25 user cinder-volume[84696]: DEBUG cinder.utils [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:49:25 user cinder-volume[84696]: DEBUG cinder.utils [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.957(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:49:25 user cinder-volume[84696]: DEBUG cinder.utils [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_13cc73df-7780-4934-8cbd-7f372b1cf57a/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:49:25 user cinder-volume[84696]: DEBUG cinder.utils [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.947(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:49:25 user cinder-volume[84696]: DEBUG cinder.utils [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_ab5889f8-b6c7-4673-a666-f4b4e5ff5fd5/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:49:25 user cinder-volume[84696]: DEBUG cinder.utils [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.567(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:49:25 user cinder-volume[84696]: DEBUG cinder.utils [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:49:25 user cinder-volume[84696]: DEBUG cinder.utils [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 9.980(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:49:25 user cinder-volume[84696]: DEBUG cinder.utils [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:49:25 user cinder-volume[84696]: DEBUG cinder.utils [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.641(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:49:25 user cinder-volume[84696]: DEBUG cinder.utils [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:49:28 user cinder-volume[84696]: DEBUG cinder.utils [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 11.640(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:49:28 user cinder-volume[84696]: DEBUG cinder.utils [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:49:28 user cinder-volume[84696]: DEBUG cinder.utils [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 11.974(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:49:28 user cinder-volume[84696]: DEBUG cinder.utils [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_fbbadf16-651a-47d5-b308-a37e56aa07f2/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:49:28 user cinder-volume[84696]: DEBUG cinder.utils [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.003(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:49:28 user cinder-volume[84696]: DEBUG cinder.utils [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_aea45c70-7c05-4146-847e-cbb777d3dfbe/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:49:28 user cinder-volume[84696]: DEBUG cinder.utils [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 11.760(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:49:28 user cinder-volume[84696]: DEBUG cinder.utils [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:49:28 user cinder-volume[84696]: DEBUG cinder.utils [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.970(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:49:28 user cinder-volume[84696]: DEBUG cinder.utils [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_68f814b0-1291-4a12-ae65-dc0892203d36/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:49:28 user cinder-volume[84696]: DEBUG cinder.utils [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 11.988(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:49:28 user cinder-volume[84696]: DEBUG cinder.utils [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_dfd745ed-b0c4-4e35-b77c-a911c90ce33e/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:49:28 user cinder-volume[84696]: DEBUG cinder.utils [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 72.954(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:49:28 user cinder-volume[84696]: DEBUG cinder.utils [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_441910f8-d29c-4c9e-bc5b-780ad228c85d/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:49:28 user cinder-volume[84696]: DEBUG cinder.utils [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 11.615(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:49:28 user cinder-volume[84696]: DEBUG cinder.utils [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:49:28 user cinder-volume[84696]: DEBUG cinder.utils [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.793(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:49:28 user cinder-volume[84696]: DEBUG cinder.utils [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:49:31 user cinder-volume[84696]: DEBUG cinder.utils [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 11.704(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:49:31 user cinder-volume[84696]: DEBUG cinder.utils [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:49:31 user cinder-volume[84696]: DEBUG cinder.utils [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.087(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:49:31 user cinder-volume[84696]: DEBUG cinder.utils [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_ab5889f8-b6c7-4673-a666-f4b4e5ff5fd5/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:49:31 user cinder-volume[84696]: DEBUG cinder.utils [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.098(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:49:31 user cinder-volume[84696]: DEBUG cinder.utils [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_13cc73df-7780-4934-8cbd-7f372b1cf57a/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:49:31 user cinder-volume[84696]: DEBUG cinder.utils [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 11.780(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:49:31 user cinder-volume[84696]: DEBUG cinder.utils [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:49:31 user cinder-volume[84696]: DEBUG cinder.utils [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 69.878(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:49:31 user cinder-volume[84696]: DEBUG cinder.utils [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_5c16bb82-43c5-4d91-9320-0e1f2bbec574/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:49:31 user cinder-volume[84696]: DEBUG cinder.utils [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 69.705(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:49:31 user cinder-volume[84696]: DEBUG cinder.utils [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:49:34 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=37,cinder:UPDATE=8,cinder:INSERT=7 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:49:34 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=25,cinder:INSERT=5,cinder:UPDATE=13 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:49:34 user cinder-volume[84696]: DEBUG cinder.utils [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 11.918(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:49:34 user cinder-volume[84696]: DEBUG cinder.utils [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:49:34 user cinder-volume[84696]: DEBUG cinder.utils [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.102(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:49:34 user cinder-volume[84696]: DEBUG cinder.utils [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_68f814b0-1291-4a12-ae65-dc0892203d36/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:49:34 user cinder-volume[84696]: DEBUG cinder.utils [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 19.190(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:49:34 user cinder-volume[84696]: DEBUG cinder.utils [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:49:34 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 139.155(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:49:34 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:49:34 user cinder-volume[84696]: ERROR oslo_service.periodic_task [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Error during VolumeManager.publish_service_capabilities: TypeError: 'NoneType' object is not subscriptable Apr 17 15:49:34 user cinder-volume[84696]: ERROR oslo_service.periodic_task Traceback (most recent call last): Apr 17 15:49:34 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/usr/local/lib/python3.10/dist-packages/oslo_service/periodic_task.py", line 216, in run_periodic_tasks Apr 17 15:49:34 user cinder-volume[84696]: ERROR oslo_service.periodic_task task(self, context) Apr 17 15:49:34 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/manager.py", line 2876, in publish_service_capabilities Apr 17 15:49:34 user cinder-volume[84696]: ERROR oslo_service.periodic_task self._report_driver_status(context) Apr 17 15:49:34 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/manager.py", line 2755, in _report_driver_status Apr 17 15:49:34 user cinder-volume[84696]: ERROR oslo_service.periodic_task volume_stats = get_stats() Apr 17 15:49:34 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/usr/local/lib/python3.10/dist-packages/oslo_utils/timeutils.py", line 319, in wrapper Apr 17 15:49:34 user cinder-volume[84696]: ERROR oslo_service.periodic_task result = func(*args, **kwargs) Apr 17 15:49:34 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/manager.py", line 2753, in get_stats Apr 17 15:49:34 user cinder-volume[84696]: ERROR oslo_service.periodic_task return self.driver.get_volume_stats(refresh=True) Apr 17 15:49:34 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/driver.py", line 666, in get_volume_stats Apr 17 15:49:34 user cinder-volume[84696]: ERROR oslo_service.periodic_task self._update_volume_stats() Apr 17 15:49:34 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 726, in _update_volume_stats Apr 17 15:49:34 user cinder-volume[84696]: ERROR oslo_service.periodic_task pool_stats = self.ra.get_pool_stats() Apr 17 15:49:34 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 897, in get_pool_stats Apr 17 15:49:34 user cinder-volume[84696]: ERROR oslo_service.periodic_task if not resp["error"] and resp["code"] == 200: Apr 17 15:49:34 user cinder-volume[84696]: ERROR oslo_service.periodic_task TypeError: 'NoneType' object is not subscriptable Apr 17 15:49:34 user cinder-volume[84696]: ERROR oslo_service.periodic_task Apr 17 15:49:34 user cinder-volume[84696]: WARNING oslo.service.loopingcall [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Function 'cinder.service.Service.periodic_tasks' run outlasted interval by 79.16 sec Apr 17 15:49:34 user cinder-volume[84696]: DEBUG oslo_service.periodic_task [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Running periodic task VolumeManager.publish_service_capabilities {{(pid=84696) run_periodic_tasks /usr/local/lib/python3.10/dist-packages/oslo_service/periodic_task.py:210}} Apr 17 15:49:34 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Updating volume stats {{(pid=84696) _update_volume_stats /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:724}} Apr 17 15:49:34 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Get pool Pool-0 fsprops {{(pid=84696) get_pool_stats /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:894}} Apr 17 15:49:34 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:49:34 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:49:37 user cinder-volume[84696]: DEBUG cinder.utils [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 20.859(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:49:37 user cinder-volume[84696]: DEBUG cinder.utils [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:49:37 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 3.069(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:49:37 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:49:37 user cinder-volume[84696]: DEBUG cinder.utils [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 20.821(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:49:37 user cinder-volume[84696]: DEBUG cinder.utils [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:49:37 user cinder-volume[84696]: DEBUG cinder.utils [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.216(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:49:37 user cinder-volume[84696]: DEBUG cinder.utils [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_aea45c70-7c05-4146-847e-cbb777d3dfbe/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:49:37 user cinder-volume[84696]: DEBUG cinder.utils [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.198(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:49:37 user cinder-volume[84696]: DEBUG cinder.utils [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_dfd745ed-b0c4-4e35-b77c-a911c90ce33e/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:49:37 user cinder-volume[84696]: DEBUG cinder.utils [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.191(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:49:37 user cinder-volume[84696]: DEBUG cinder.utils [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_fbbadf16-651a-47d5-b308-a37e56aa07f2/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:49:37 user cinder-volume[84696]: DEBUG cinder.utils [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 20.975(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:49:37 user cinder-volume[84696]: DEBUG cinder.utils [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:49:40 user cinder-volume[84696]: DEBUG cinder.utils [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 20.920(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:49:40 user cinder-volume[84696]: DEBUG cinder.utils [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:49:40 user cinder-volume[84696]: DEBUG cinder.utils [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.302(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:49:40 user cinder-volume[84696]: DEBUG cinder.utils [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_ab5889f8-b6c7-4673-a666-f4b4e5ff5fd5/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:49:40 user cinder-volume[84696]: DEBUG cinder.utils [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.314(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:49:40 user cinder-volume[84696]: DEBUG cinder.utils [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_13cc73df-7780-4934-8cbd-7f372b1cf57a/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:49:40 user cinder-volume[84696]: DEBUG cinder.utils [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 20.995(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:49:40 user cinder-volume[84696]: DEBUG cinder.utils [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:49:40 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.139(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:49:40 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:49:43 user cinder-volume[84696]: DEBUG cinder.utils [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.134(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:49:43 user cinder-volume[84696]: DEBUG cinder.utils [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:49:43 user cinder-volume[84696]: DEBUG cinder.utils [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.318(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:49:43 user cinder-volume[84696]: DEBUG cinder.utils [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_68f814b0-1291-4a12-ae65-dc0892203d36/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:49:47 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.279(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:49:47 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:49:50 user cinder-volume[84696]: DEBUG cinder.utils [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 72.951(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:49:50 user cinder-volume[84696]: DEBUG cinder.utils [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_2c9714de-f8ea-4d1d-99f9-beb85126ad91/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:49:50 user cinder-volume[84696]: DEBUG cinder.utils [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 72.768(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:49:50 user cinder-volume[84696]: DEBUG cinder.utils [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:49:53 user cinder-volume[84696]: DEBUG cinder.utils [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 37.619(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:49:53 user cinder-volume[84696]: DEBUG cinder.utils [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:49:54 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=1 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:49:56 user cinder-volume[84696]: DEBUG cinder.utils [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.246(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:49:56 user cinder-volume[84696]: DEBUG cinder.utils [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:49:56 user cinder-volume[84696]: DEBUG cinder.utils [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.288(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:49:56 user cinder-volume[84696]: DEBUG cinder.utils [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:49:56 user cinder-volume[84696]: DEBUG cinder.utils [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.616(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:49:56 user cinder-volume[84696]: DEBUG cinder.utils [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_fbbadf16-651a-47d5-b308-a37e56aa07f2/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:49:56 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.498(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:49:56 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:49:56 user cinder-volume[84696]: DEBUG cinder.utils [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.645(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:49:56 user cinder-volume[84696]: DEBUG cinder.utils [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_aea45c70-7c05-4146-847e-cbb777d3dfbe/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:49:56 user cinder-volume[84696]: DEBUG cinder.utils [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.626(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:49:56 user cinder-volume[84696]: DEBUG cinder.utils [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_dfd745ed-b0c4-4e35-b77c-a911c90ce33e/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:49:56 user cinder-volume[84696]: DEBUG cinder.utils [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.402(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:49:56 user cinder-volume[84696]: DEBUG cinder.utils [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:49:59 user cinder-volume[84696]: DEBUG cinder.utils [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.352(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:49:59 user cinder-volume[84696]: DEBUG cinder.utils [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:49:59 user cinder-volume[84696]: DEBUG cinder.utils [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.734(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:49:59 user cinder-volume[84696]: DEBUG cinder.utils [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_ab5889f8-b6c7-4673-a666-f4b4e5ff5fd5/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:49:59 user cinder-volume[84696]: DEBUG cinder.utils [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.746(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:49:59 user cinder-volume[84696]: DEBUG cinder.utils [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_13cc73df-7780-4934-8cbd-7f372b1cf57a/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:49:59 user cinder-volume[84696]: DEBUG cinder.utils [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.427(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:49:59 user cinder-volume[84696]: DEBUG cinder.utils [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:50:02 user cinder-volume[84696]: DEBUG cinder.utils [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.567(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:50:02 user cinder-volume[84696]: DEBUG cinder.utils [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:50:02 user cinder-volume[84696]: DEBUG cinder.utils [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.751(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:50:02 user cinder-volume[84696]: DEBUG cinder.utils [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_68f814b0-1291-4a12-ae65-dc0892203d36/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:50:04 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=11,cinder:UPDATE=3 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:50:04 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Volume reschedule parameters: True retry: {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 15:50:04 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Flow 'volume_create_manager' (0d0c5f0a-7eb0-439f-a14a-41128c0344ae) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:50:04 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (4df56b16-6f65-4f9b-a7d8-91eeae6b6537) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:50:04 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (4df56b16-6f65-4f9b-a7d8-91eeae6b6537) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:50:05Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-ServerActionsTestJSON-volume-872354177',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=2ed5b8aa-fb48-4255-b6d5-f0f84f733834,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='cb5f6b5ed2fc43b69c47e1fc18646545',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:50:05Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:50:05Z,use_quota=True,user_id='638a4f535e9740fc9d810639a5eb24d7',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:50:04 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (f7be3d54-6bd4-45b9-8f47-361f0f9ad395) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:50:04 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (f7be3d54-6bd4-45b9-8f47-361f0f9ad395) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:50:04 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (b1762999-b383-4b45-8781-6d9eb6df7102) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:50:04 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (b1762999-b383-4b45-8781-6d9eb6df7102) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': '2ed5b8aa-fb48-4255-b6d5-f0f84f733834', 'volume_name': 'volume-2ed5b8aa-fb48-4255-b6d5-f0f84f733834', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:50:04 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (00c1c7e6-3784-47b2-8d33-6f62020a5634) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:50:04 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (00c1c7e6-3784-47b2-8d33-6f62020a5634) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:50:04 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (2c26f43a-c175-4bba-aeca-e78a016c2cf2) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:50:04 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Volume 2ed5b8aa-fb48-4255-b6d5-f0f84f733834: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-2ed5b8aa-fb48-4255-b6d5-f0f84f733834', 'volume_size': 1} Apr 17 15:50:04 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] creating volume v_2ed5b8aa-fb48-4255-b6d5-f0f84f733834. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 15:50:04 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] create volume {'name': 'v_2ed5b8aa-fb48-4255-b6d5-f0f84f733834', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 15:50:04 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:50:04 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:50:05 user cinder-volume[84696]: DEBUG cinder.utils [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 0.635(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:50:05 user cinder-volume[84696]: DEBUG cinder.utils [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:50:09 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-778c62be-8963-4e6d-97f1-f456b759ce16 req-8dba5389-38e6-4a46-b382-2d4356cad7db tempest-AttachVolumeShelveTestJSON-1745617049 None] terminate connection for 13bf568b-585c-4556-ab78-e9930ad50214 {{(pid=84696) terminate_connection /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:1032}} Apr 17 15:50:09 user cinder-volume[84696]: INFO cinder.volume.manager [req-778c62be-8963-4e6d-97f1-f456b759ce16 req-8dba5389-38e6-4a46-b382-2d4356cad7db tempest-AttachVolumeShelveTestJSON-1745617049 None] Terminate volume connection completed successfully. Apr 17 15:50:09 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-778c62be-8963-4e6d-97f1-f456b759ce16 req-8dba5389-38e6-4a46-b382-2d4356cad7db tempest-AttachVolumeShelveTestJSON-1745617049 None] Deleting attachment 1634a7d0-3313-448e-9888-8de015f268af. {{(pid=84696) attachment_delete /opt/stack/cinder/cinder/volume/manager.py:5006}} Apr 17 15:50:09 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-778c62be-8963-4e6d-97f1-f456b759ce16 req-8dba5389-38e6-4a46-b382-2d4356cad7db tempest-AttachVolumeShelveTestJSON-1745617049 None] remove_export for volume: 13bf568b-585c-4556-ab78-e9930ad50214. {{(pid=84696) remove_export /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:718}} Apr 17 15:50:09 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-778c62be-8963-4e6d-97f1-f456b759ce16 req-8dba5389-38e6-4a46-b382-2d4356cad7db tempest-AttachVolumeShelveTestJSON-1745617049 None] remove export {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:912}} Apr 17 15:50:09 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-778c62be-8963-4e6d-97f1-f456b759ce16 req-8dba5389-38e6-4a46-b382-2d4356cad7db tempest-AttachVolumeShelveTestJSON-1745617049 None] detach volume:Volume(_name_id=None,admin_metadata=,attach_status='attached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:25:53Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-AttachVolumeShelveTestJSON-volume-1457837636',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=13bf568b-585c-4556-ab78-e9930ad50214,launched_at=2023-04-17T15:25:55Z,metadata=,migration_status=None,multiattach=False,previous_status=None,project_id='214a0f233f114d7aa22e64048a837c0b',provider_auth='CHAP cajgFQvq cTCKdDR0kw38ze',provider_geometry=None,provider_id=None,provider_location='172.16.0.220:3260,1 iqn.2010-10.org.openstack:13bf568b-585c-4556-ab78-e9930ad50214 0',replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:25:53Z,service_uuid='e628352e-c4cf-40eb-8389-cce40663026a',shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='in-use',terminated_at=None,updated_at=2023-04-17T15:31:43Z,use_quota=True,user_id='83b9d77de46143d8b9277a55f5ad4d72',volume_attachment=,volume_type=,volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c) from target:iqn.2010-10.org.openstack:13bf568b-585c-4556-ab78-e9930ad50214. {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:913}} Apr 17 15:50:09 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [req-778c62be-8963-4e6d-97f1-f456b759ce16 req-8dba5389-38e6-4a46-b382-2d4356cad7db tempest-AttachVolumeShelveTestJSON-1745617049 None] detach volume v_13bf568b-585c-4556-ab78-e9930ad50214 from target iqn.2010-10.org.openstack:13bf568b-585c-4556-ab78-e9930ad50214 {{(pid=84696) detach_target_vol /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:606}} Apr 17 15:50:09 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-778c62be-8963-4e6d-97f1-f456b759ce16 req-8dba5389-38e6-4a46-b382-2d4356cad7db tempest-AttachVolumeShelveTestJSON-1745617049 None] Sending pool request DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:13bf568b-585c-4556-ab78-e9930ad50214/luns/v_13bf568b-585c-4556-ab78-e9930ad50214 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:50:09 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-778c62be-8963-4e6d-97f1-f456b759ce16 req-8dba5389-38e6-4a46-b382-2d4356cad7db tempest-AttachVolumeShelveTestJSON-1745617049 None] Sending DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:13bf568b-585c-4556-ab78-e9930ad50214/luns/v_13bf568b-585c-4556-ab78-e9930ad50214 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:50:09 user cinder-volume[84696]: DEBUG cinder.utils [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 4.474(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:50:09 user cinder-volume[84696]: DEBUG cinder.utils [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:50:09 user cinder-volume[84696]: DEBUG cinder.utils [req-778c62be-8963-4e6d-97f1-f456b759ce16 req-8dba5389-38e6-4a46-b382-2d4356cad7db tempest-AttachVolumeShelveTestJSON-1745617049 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 0.272(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:50:09 user cinder-volume[84696]: DEBUG cinder.utils [req-778c62be-8963-4e6d-97f1-f456b759ce16 req-8dba5389-38e6-4a46-b382-2d4356cad7db tempest-AttachVolumeShelveTestJSON-1745617049 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:13bf568b-585c-4556-ab78-e9930ad50214/luns/v_13bf568b-585c-4556-ab78-e9930ad50214 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:50:12 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 37.627(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:50:12 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:50:12 user cinder-volume[84696]: DEBUG cinder.utils [req-778c62be-8963-4e6d-97f1-f456b759ce16 req-8dba5389-38e6-4a46-b382-2d4356cad7db tempest-AttachVolumeShelveTestJSON-1745617049 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 3.349(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:50:12 user cinder-volume[84696]: DEBUG cinder.utils [req-778c62be-8963-4e6d-97f1-f456b759ce16 req-8dba5389-38e6-4a46-b382-2d4356cad7db tempest-AttachVolumeShelveTestJSON-1745617049 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:13bf568b-585c-4556-ab78-e9930ad50214/luns/v_13bf568b-585c-4556-ab78-e9930ad50214 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:50:15 user cinder-volume[84696]: DEBUG cinder.utils [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 10.618(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:50:15 user cinder-volume[84696]: DEBUG cinder.utils [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:50:15 user cinder-volume[84696]: DEBUG cinder.utils [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 136.076(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:50:15 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:50:15 user cinder-volume[84696]: INFO cinder.message.api [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Creating message record for request_id = req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 Apr 17 15:50:15 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 15:50:15 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (9d5a63ca-cfb7-4210-880b-f0307fc50723) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 15:50:15 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 15:50:15 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:47:59Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-ServerRescueNegativeTestJSON-volume-7015736',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=cdedb820-e1c7-4b86-8286-43b3640b3a46,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='9324c99bd4d040bdb5a7f1db782b2783',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:47:59Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:47:59Z,use_quota=True,user_id='8e3487d6b32d46beac556617050fd664',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 15:50:15 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:47:59Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-ServerRescueNegativeTestJSON-volume-7015736',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=cdedb820-e1c7-4b86-8286-43b3640b3a46,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='9324c99bd4d040bdb5a7f1db782b2783',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:47:59Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:47:59Z,use_quota=True,user_id='8e3487d6b32d46beac556617050fd664',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(cdedb820-e1c7-4b86-8286-43b3640b3a46),volume_id=cdedb820-e1c7-4b86-8286-43b3640b3a46,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': 'cdedb820-e1c7-4b86-8286-43b3640b3a46', 'volume_name': 'volume-cdedb820-e1c7-4b86-8286-43b3640b3a46', 'volume_size': 1}} Apr 17 15:50:15 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'retry': {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']}, 'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': 'cdedb820-e1c7-4b86-8286-43b3640b3a46', 'volume': {'id': 'cdedb820-e1c7-4b86-8286-43b3640b3a46', '_name_id': None, 'ec2_id': None, 'user_id': '8e3487d6b32d46beac556617050fd664', 'project_id': '9324c99bd4d040bdb5a7f1db782b2783', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-ServerRescueNegativeTestJSON-volume-7015736', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T15:47:59.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-cdedb820-e1c7-4b86-8286-43b3640b3a46', 'name_id': 'cdedb820-e1c7-4b86-8286-43b3640b3a46', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-ServerRescueNegativeTestJSON-volume-7015736', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '9324c99bd4d040bdb5a7f1db782b2783', 'qos_specs': None, 'reservations': ['f9dad618-bdce-4da3-89f6-14cc03e04ddc', 'f52bc1e3-5f13-4f5b-b4df-610d006660f1', '9b6eb134-10fa-4f48-8b07-6e9470fc5aae', 'f45d6340-19c8-4636-9836-b94ea2be5f46'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': '8e3487d6b32d46beac556617050fd664', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-ServerRescueNegativeTestJSON-volume-7015736', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '9324c99bd4d040bdb5a7f1db782b2783', 'qos_specs': None, 'reservations': ['f9dad618-bdce-4da3-89f6-14cc03e04ddc', 'f52bc1e3-5f13-4f5b-b4df-610d006660f1', '9b6eb134-10fa-4f48-8b07-6e9470fc5aae', 'f45d6340-19c8-4636-9836-b94ea2be5f46'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': '8e3487d6b32d46beac556617050fd664', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': '8e3487d6b32d46beac556617050fd664', 'metadata': {}, 'qos_specs': None}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(cdedb820-e1c7-4b86-8286-43b3640b3a46),volume_id=cdedb820-e1c7-4b86-8286-43b3640b3a46,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:47:59Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-ServerRescueNegativeTestJSON-volume-7015736',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=cdedb820-e1c7-4b86-8286-43b3640b3a46,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='9324c99bd4d040bdb5a7f1db782b2783',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:47:59Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:47:59Z,use_quota=True,user_id='8e3487d6b32d46beac556617050fd664',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 15:50:15 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:47:59Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-ServerRescueNegativeTestJSON-volume-7015736',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=cdedb820-e1c7-4b86-8286-43b3640b3a46,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='9324c99bd4d040bdb5a7f1db782b2783',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:47:59Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:47:59Z,use_quota=True,user_id='8e3487d6b32d46beac556617050fd664',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:47:59Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-ServerRescueNegativeTestJSON-volume-7015736',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=cdedb820-e1c7-4b86-8286-43b3640b3a46,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='9324c99bd4d040bdb5a7f1db782b2783',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:47:59Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:47:59Z,use_quota=True,user_id='8e3487d6b32d46beac556617050fd664',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 15:50:15 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 15:50:15 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 15:50:15 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:50:15 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 15:50:15 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 15:50:15 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 15:50:15 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 15:50:15 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 15:50:15 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 15:50:15 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 15:50:15 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 15:50:15 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 15:50:15 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 15:50:15 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 15:50:15 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 15:50:15 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 15:50:15 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 15:50:15 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 15:50:15 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 15:50:15 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 15:50:15 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (9d5a63ca-cfb7-4210-880b-f0307fc50723) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:50:15 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (9d5a63ca-cfb7-4210-880b-f0307fc50723) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:50:15 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (dec98a2e-2088-4a24-9523-f2350094d5a9) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:50:15 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (dec98a2e-2088-4a24-9523-f2350094d5a9) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:50:15 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (873c476f-2a43-474c-96b7-18cbcbead4a4) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:50:15 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (873c476f-2a43-474c-96b7-18cbcbead4a4) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:50:15 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (427e4daa-48b1-40f1-b755-ad1fe82a7c04) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:50:15 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Updating volume cdedb820-e1c7-4b86-8286-43b3640b3a46 with {'scheduled_at': datetime.datetime(2023, 4, 17, 15, 50, 15, 553492), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 15:50:15 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Volume cdedb820-e1c7-4b86-8286-43b3640b3a46: re-scheduling SchedulerAPI.create_volume attempt 1 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 15:50:15 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Volume cdedb820-e1c7-4b86-8286-43b3640b3a46: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 15:50:15 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] deleating volume v_cdedb820-e1c7-4b86-8286-43b3640b3a46 {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 15:50:15 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] get snapshots for volume v_cdedb820-e1c7-4b86-8286-43b3640b3a46 {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 15:50:15 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_cdedb820-e1c7-4b86-8286-43b3640b3a46/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:50:15 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_cdedb820-e1c7-4b86-8286-43b3640b3a46/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:50:15 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Volume reschedule parameters: True retry: {'num_attempts': 2, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 15:50:15 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Flow 'volume_create_manager' (ffc87cd6-e876-4e83-a0cb-672da84ca274) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:50:15 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (8ef09eb6-13d3-4f99-a9fe-67d1e1f58f0c) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:50:15 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (8ef09eb6-13d3-4f99-a9fe-67d1e1f58f0c) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:47:59Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-ServerRescueNegativeTestJSON-volume-7015736',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=cdedb820-e1c7-4b86-8286-43b3640b3a46,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='9324c99bd4d040bdb5a7f1db782b2783',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:50:16Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:50:16Z,use_quota=True,user_id='8e3487d6b32d46beac556617050fd664',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:50:15 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (18b1c95e-08dd-44a9-9e90-087b0effba30) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:50:15 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (18b1c95e-08dd-44a9-9e90-087b0effba30) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:50:15 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (0adabb07-eec7-46a0-a0e6-d669b71fdef4) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:50:15 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (0adabb07-eec7-46a0-a0e6-d669b71fdef4) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': 'cdedb820-e1c7-4b86-8286-43b3640b3a46', 'volume_name': 'volume-cdedb820-e1c7-4b86-8286-43b3640b3a46', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:50:15 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (41b3a978-1d00-451f-8f78-d69707c91ee9) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:50:15 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (41b3a978-1d00-451f-8f78-d69707c91ee9) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:50:15 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (33c814c5-0243-4381-8577-703ee4293c85) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:50:15 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Volume cdedb820-e1c7-4b86-8286-43b3640b3a46: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-cdedb820-e1c7-4b86-8286-43b3640b3a46', 'volume_size': 1} Apr 17 15:50:15 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] creating volume v_cdedb820-e1c7-4b86-8286-43b3640b3a46. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 15:50:15 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] create volume {'name': 'v_cdedb820-e1c7-4b86-8286-43b3640b3a46', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 15:50:15 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:50:15 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:50:18 user cinder-volume[84696]: DEBUG cinder.utils [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.964(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:50:18 user cinder-volume[84696]: DEBUG cinder.utils [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_cdedb820-e1c7-4b86-8286-43b3640b3a46/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:50:18 user cinder-volume[84696]: DEBUG cinder.utils [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.798(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:50:18 user cinder-volume[84696]: DEBUG cinder.utils [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:50:18 user cinder-volume[84696]: DEBUG cinder.utils [req-778c62be-8963-4e6d-97f1-f456b759ce16 req-8dba5389-38e6-4a46-b382-2d4356cad7db tempest-AttachVolumeShelveTestJSON-1745617049 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 9.492(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:50:18 user cinder-volume[84696]: DEBUG cinder.utils [req-778c62be-8963-4e6d-97f1-f456b759ce16 req-8dba5389-38e6-4a46-b382-2d4356cad7db tempest-AttachVolumeShelveTestJSON-1745617049 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:13bf568b-585c-4556-ab78-e9930ad50214/luns/v_13bf568b-585c-4556-ab78-e9930ad50214 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:50:21 user cinder-volume[84696]: DEBUG cinder.utils [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.868(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:50:21 user cinder-volume[84696]: DEBUG cinder.utils [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:50:21 user cinder-volume[84696]: DEBUG cinder.utils [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.038(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:50:21 user cinder-volume[84696]: DEBUG cinder.utils [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_cdedb820-e1c7-4b86-8286-43b3640b3a46/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:50:24 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=13,cinder:UPDATE=4,cinder:INSERT=2 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:50:24 user cinder-volume[84696]: DEBUG cinder.utils [req-778c62be-8963-4e6d-97f1-f456b759ce16 req-d9cf97f7-129f-4cb2-9bc2-f51b1339159a tempest-AttachVolumeShelveTestJSON-1745617049 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 135.689(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:50:24 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-778c62be-8963-4e6d-97f1-f456b759ce16 req-d9cf97f7-129f-4cb2-9bc2-f51b1339159a tempest-AttachVolumeShelveTestJSON-1745617049 None] Geting None from DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:13bf568b-585c-4556-ab78-e9930ad50214/luns/v_13bf568b-585c-4556-ab78-e9930ad50214 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:50:24 user cinder-volume[84696]: WARNING cinder.volume.manager [req-778c62be-8963-4e6d-97f1-f456b759ce16 req-d9cf97f7-129f-4cb2-9bc2-f51b1339159a tempest-AttachVolumeShelveTestJSON-1745617049 None] Failed to detach volume on the backend, ignoring failure 'NoneType' object is not subscriptable Apr 17 15:50:24 user cinder-volume[84696]: DEBUG cinder.utils [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 138.258(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:50:24 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_93a32a7c-4c6a-472a-8f91-9d34558ba232/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:50:24 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (fd6372ff-955f-45b3-a7c4-551f96bdd404) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 15:50:24 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (53e578f2-e99b-4739-a718-da9597addaf0) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:50:24 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (53e578f2-e99b-4739-a718-da9597addaf0) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:50:24 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Flow 'volume_create_manager' (09111813-7745-4322-aaca-b836df46177b) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 15:50:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 15:50:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 15:50:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 15:50:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 15:50:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 15:50:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 15:50:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 15:50:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 15:50:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 15:50:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 15:50:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 15:50:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 15:50:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 15:50:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 15:50:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 15:50:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 15:50:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 15:50:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 15:50:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 15:50:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 15:50:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 15:50:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 15:50:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 15:50:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 15:50:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:50:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 15:50:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1237, in execute Apr 17 15:50:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_from_image(context, Apr 17 15:50:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/utils.py", line 670, in _wrapper Apr 17 15:50:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return r(f, *args, **kwargs) Apr 17 15:50:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/tenacity/__init__.py", line 423, in __call__ Apr 17 15:50:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server do = self.iter(retry_state=retry_state) Apr 17 15:50:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/tenacity/__init__.py", line 360, in iter Apr 17 15:50:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return fut.result() Apr 17 15:50:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/lib/python3.10/concurrent/futures/_base.py", line 451, in result Apr 17 15:50:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self.__get_result() Apr 17 15:50:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result Apr 17 15:50:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self._exception Apr 17 15:50:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/tenacity/__init__.py", line 426, in __call__ Apr 17 15:50:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = fn(*args, **kwargs) Apr 17 15:50:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1125, in _create_from_image Apr 17 15:50:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_from_image_cache_or_download( Apr 17 15:50:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 944, in _create_from_image_cache_or_download Apr 17 15:50:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update, cloned = self._create_from_image_cache( Apr 17 15:50:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 843, in _create_from_image_cache Apr 17 15:50:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_from_source_volume( Apr 17 15:50:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 659, in _create_from_source_volume Apr 17 15:50:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self.driver.create_cloned_volume(volume, srcvol_ref) Apr 17 15:50:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 525, in create_cloned_volume Apr 17 15:50:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self._clone_object(vname, cvname) Apr 17 15:50:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 461, in _clone_object Apr 17 15:50:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_snapshot(oname, coname) Apr 17 15:50:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 639, in create_snapshot Apr 17 15:50:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201, 204): Apr 17 15:50:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 15:50:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 15:50:24 user cinder-volume[84696]: DEBUG cinder.utils [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 137.960(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:50:24 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:50:24 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (ffd4194b-0692-4180-b2f2-0e5ec1eae1ed) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 15:50:24 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 15:50:24 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:45:47Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TestVolumeBootPattern-volume-origin-14805124',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=93a32a7c-4c6a-472a-8f91-9d34558ba232,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='c8a836b47d954cb9a7c1cd7c0f790426',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:48:07Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:48:07Z,use_quota=True,user_id='a80e753322224ec1a897598b5b9ed6ea',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 15:50:24 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:45:47Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TestVolumeBootPattern-volume-origin-14805124',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=93a32a7c-4c6a-472a-8f91-9d34558ba232,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='c8a836b47d954cb9a7c1cd7c0f790426',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:48:07Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:48:07Z,use_quota=True,user_id='a80e753322224ec1a897598b5b9ed6ea',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=b3f0dec1-7758-44a8-82e0-76ada8837428,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(93a32a7c-4c6a-472a-8f91-9d34558ba232),volume_id=93a32a7c-4c6a-472a-8f91-9d34558ba232,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'image', 'volume_id': '93a32a7c-4c6a-472a-8f91-9d34558ba232', 'volume_name': 'volume-93a32a7c-4c6a-472a-8f91-9d34558ba232', 'volume_size': 1, 'image_id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'image_location': (None, None), 'image_meta': {'name': 'cirros-0.5.2-x86_64-disk', 'disk_format': 'qcow2', 'container_format': 'bare', 'visibility': 'public', 'size': 16300544, 'virtual_size': 117440512, 'status': 'active', 'checksum': 'b874c39491a2377b8490f5f1e89761a4', 'protected': False, 'min_ram': 0, 'min_disk': 0, 'owner': 'a43c211b4ed542538799349489b1d884', 'os_hidden': False, 'os_hash_algo': 'sha512', 'os_hash_value': '6b813aa46bb90b4da216a4d19376593fa3f4fc7e617f03a92b7fe11e9a3981cbe8f0959dbebe36225e5f53dc4492341a4863cac4ed1ee0909f3fc78ef9c3e869', 'id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'created_at': datetime.datetime(2023, 4, 17, 15, 19, 53, tzinfo=datetime.timezone.utc), 'updated_at': datetime.datetime(2023, 4, 17, 15, 19, 55, tzinfo=datetime.timezone.utc), 'tags': [], 'file': '/v2/images/b3f0dec1-7758-44a8-82e0-76ada8837428/file', 'properties': {'hw_rng_model': 'virtio', 'owner_specified.openstack.md5': '', 'owner_specified.openstack.object': 'images/cirros-0.5.2-x86_64-disk', 'owner_specified.openstack.sha256': ''}}, 'image_service': }} Apr 17 15:50:24 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'snapshot_id': None, 'source_volid': None, 'volume_id': '93a32a7c-4c6a-472a-8f91-9d34558ba232', 'volume': {'id': '93a32a7c-4c6a-472a-8f91-9d34558ba232', '_name_id': None, 'ec2_id': None, 'user_id': 'a80e753322224ec1a897598b5b9ed6ea', 'project_id': 'c8a836b47d954cb9a7c1cd7c0f790426', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-TestVolumeBootPattern-volume-origin-14805124', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'admin_metadata': {}, 'glance_metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T15:45:47.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-93a32a7c-4c6a-472a-8f91-9d34558ba232', 'name_id': '93a32a7c-4c6a-472a-8f91-9d34558ba232', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-TestVolumeBootPattern-volume-origin-14805124', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': 'c8a836b47d954cb9a7c1cd7c0f790426', 'qos_specs': None, 'reservations': ['c48dcd83-2285-48af-a8d0-ff11edc29cb9', 'd1463595-875c-453d-8e19-7c670de227bd', '59caf45d-ddd8-48db-b7c0-c2b483358ca9', 'deb0cce9-9fd9-4e11-8704-901c42ef169b'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'a80e753322224ec1a897598b5b9ed6ea', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-TestVolumeBootPattern-volume-origin-14805124', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': 'c8a836b47d954cb9a7c1cd7c0f790426', 'qos_specs': None, 'reservations': ['c48dcd83-2285-48af-a8d0-ff11edc29cb9', 'd1463595-875c-453d-8e19-7c670de227bd', '59caf45d-ddd8-48db-b7c0-c2b483358ca9', 'deb0cce9-9fd9-4e11-8704-901c42ef169b'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'a80e753322224ec1a897598b5b9ed6ea', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': 'a80e753322224ec1a897598b5b9ed6ea', 'metadata': {}, 'qos_specs': None, 'retry': {'num_attempts': 2, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']}}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=b3f0dec1-7758-44a8-82e0-76ada8837428,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(93a32a7c-4c6a-472a-8f91-9d34558ba232),volume_id=93a32a7c-4c6a-472a-8f91-9d34558ba232,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:45:47Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TestVolumeBootPattern-volume-origin-14805124',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=93a32a7c-4c6a-472a-8f91-9d34558ba232,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='c8a836b47d954cb9a7c1cd7c0f790426',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:48:07Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:48:07Z,use_quota=True,user_id='a80e753322224ec1a897598b5b9ed6ea',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 15:50:24 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:45:47Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TestVolumeBootPattern-volume-origin-14805124',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=93a32a7c-4c6a-472a-8f91-9d34558ba232,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='c8a836b47d954cb9a7c1cd7c0f790426',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:48:07Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:48:07Z,use_quota=True,user_id='a80e753322224ec1a897598b5b9ed6ea',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:45:47Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TestVolumeBootPattern-volume-origin-14805124',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=93a32a7c-4c6a-472a-8f91-9d34558ba232,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='c8a836b47d954cb9a7c1cd7c0f790426',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:48:07Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:48:07Z,use_quota=True,user_id='a80e753322224ec1a897598b5b9ed6ea',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 15:50:24 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 15:50:24 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 15:50:24 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:50:24 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 15:50:24 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1237, in execute Apr 17 15:50:24 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_from_image(context, Apr 17 15:50:24 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/utils.py", line 670, in _wrapper Apr 17 15:50:24 user cinder-volume[84696]: ERROR cinder.volume.manager return r(f, *args, **kwargs) Apr 17 15:50:24 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/tenacity/__init__.py", line 423, in __call__ Apr 17 15:50:24 user cinder-volume[84696]: ERROR cinder.volume.manager do = self.iter(retry_state=retry_state) Apr 17 15:50:24 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/tenacity/__init__.py", line 360, in iter Apr 17 15:50:24 user cinder-volume[84696]: ERROR cinder.volume.manager return fut.result() Apr 17 15:50:24 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/lib/python3.10/concurrent/futures/_base.py", line 451, in result Apr 17 15:50:24 user cinder-volume[84696]: ERROR cinder.volume.manager return self.__get_result() Apr 17 15:50:24 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result Apr 17 15:50:24 user cinder-volume[84696]: ERROR cinder.volume.manager raise self._exception Apr 17 15:50:24 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/tenacity/__init__.py", line 426, in __call__ Apr 17 15:50:24 user cinder-volume[84696]: ERROR cinder.volume.manager result = fn(*args, **kwargs) Apr 17 15:50:24 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1125, in _create_from_image Apr 17 15:50:24 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_from_image_cache_or_download( Apr 17 15:50:24 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 944, in _create_from_image_cache_or_download Apr 17 15:50:24 user cinder-volume[84696]: ERROR cinder.volume.manager model_update, cloned = self._create_from_image_cache( Apr 17 15:50:24 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 843, in _create_from_image_cache Apr 17 15:50:24 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_from_source_volume( Apr 17 15:50:24 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 659, in _create_from_source_volume Apr 17 15:50:24 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self.driver.create_cloned_volume(volume, srcvol_ref) Apr 17 15:50:24 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 525, in create_cloned_volume Apr 17 15:50:24 user cinder-volume[84696]: ERROR cinder.volume.manager self._clone_object(vname, cvname) Apr 17 15:50:24 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 461, in _clone_object Apr 17 15:50:24 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_snapshot(oname, coname) Apr 17 15:50:24 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 639, in create_snapshot Apr 17 15:50:24 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201, 204): Apr 17 15:50:24 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 15:50:24 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 15:50:24 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (ffd4194b-0692-4180-b2f2-0e5ec1eae1ed) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:50:24 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (ffd4194b-0692-4180-b2f2-0e5ec1eae1ed) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:50:24 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (81e5f561-0a54-4025-b482-e64744649b65) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:50:24 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (81e5f561-0a54-4025-b482-e64744649b65) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:50:24 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (e8421914-d534-4f63-acac-2d12132d3230) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:50:24 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (e8421914-d534-4f63-acac-2d12132d3230) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:50:24 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (5a647616-587e-451f-ad41-772bf631bc61) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:50:24 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Updating volume 93a32a7c-4c6a-472a-8f91-9d34558ba232 with {'scheduled_at': datetime.datetime(2023, 4, 17, 15, 50, 24, 767308), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 15:50:24 user cinder-volume[84696]: DEBUG cinder.utils [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 19.905(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:50:24 user cinder-volume[84696]: DEBUG cinder.utils [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:50:24 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Volume 93a32a7c-4c6a-472a-8f91-9d34558ba232: re-scheduling SchedulerAPI.create_volume attempt 2 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 15:50:24 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Volume 93a32a7c-4c6a-472a-8f91-9d34558ba232: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 15:50:24 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] deleating volume v_93a32a7c-4c6a-472a-8f91-9d34558ba232 {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 15:50:24 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] get snapshots for volume v_93a32a7c-4c6a-472a-8f91-9d34558ba232 {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 15:50:24 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_93a32a7c-4c6a-472a-8f91-9d34558ba232/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:50:24 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_93a32a7c-4c6a-472a-8f91-9d34558ba232/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:50:24 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Volume reschedule parameters: True retry: {'num_attempts': 3, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 15:50:24 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Flow 'volume_create_manager' (f2fa196b-83f5-47ca-a1b7-e5232894da6d) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:50:24 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (055bd2a4-2fc8-496f-9b11-dea155072372) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:50:24 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (055bd2a4-2fc8-496f-9b11-dea155072372) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:45:47Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TestVolumeBootPattern-volume-origin-14805124',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=93a32a7c-4c6a-472a-8f91-9d34558ba232,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='c8a836b47d954cb9a7c1cd7c0f790426',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:50:25Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:50:25Z,use_quota=True,user_id='a80e753322224ec1a897598b5b9ed6ea',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:50:24 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (6caa0046-7037-45da-bf8d-6687d5b2e814) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:50:24 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (6caa0046-7037-45da-bf8d-6687d5b2e814) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:50:24 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (2600e76a-0786-4380-9fe6-194c0445d2ab) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:50:25 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (2600e76a-0786-4380-9fe6-194c0445d2ab) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'image', 'volume_id': '93a32a7c-4c6a-472a-8f91-9d34558ba232', 'volume_name': 'volume-93a32a7c-4c6a-472a-8f91-9d34558ba232', 'volume_size': 1, 'image_id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'image_location': (None, None), 'image_meta': {'name': 'cirros-0.5.2-x86_64-disk', 'disk_format': 'qcow2', 'container_format': 'bare', 'visibility': 'public', 'size': 16300544, 'virtual_size': 117440512, 'status': 'active', 'checksum': 'b874c39491a2377b8490f5f1e89761a4', 'protected': False, 'min_ram': 0, 'min_disk': 0, 'owner': 'a43c211b4ed542538799349489b1d884', 'os_hidden': False, 'os_hash_algo': 'sha512', 'os_hash_value': '6b813aa46bb90b4da216a4d19376593fa3f4fc7e617f03a92b7fe11e9a3981cbe8f0959dbebe36225e5f53dc4492341a4863cac4ed1ee0909f3fc78ef9c3e869', 'id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'created_at': datetime.datetime(2023, 4, 17, 15, 19, 53, tzinfo=datetime.timezone.utc), 'updated_at': datetime.datetime(2023, 4, 17, 15, 19, 55, tzinfo=datetime.timezone.utc), 'tags': [], 'file': '/v2/images/b3f0dec1-7758-44a8-82e0-76ada8837428/file', 'properties': {'hw_rng_model': 'virtio', 'owner_specified.openstack.md5': '', 'owner_specified.openstack.object': 'images/cirros-0.5.2-x86_64-disk', 'owner_specified.openstack.sha256': ''}}, 'image_service': }' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:50:25 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (231e66e0-36a5-41c2-a1be-8f99a877675f) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:50:25 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (231e66e0-36a5-41c2-a1be-8f99a877675f) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:50:25 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (c700abaf-a7c0-4e50-bd3e-01d84520168d) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:50:25 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Volume 93a32a7c-4c6a-472a-8f91-9d34558ba232: being created as image with specification: {'status': 'creating', 'volume_name': 'volume-93a32a7c-4c6a-472a-8f91-9d34558ba232', 'volume_size': 1, 'image_id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'image_location': (None, None), 'image_meta': {'name': 'cirros-0.5.2-x86_64-disk', 'disk_format': 'qcow2', 'container_format': 'bare', 'visibility': 'public', 'size': 16300544, 'virtual_size': 117440512, 'status': 'active', 'checksum': 'b874c39491a2377b8490f5f1e89761a4', 'protected': False, 'min_ram': 0, 'min_disk': 0, 'owner': 'a43c211b4ed542538799349489b1d884', 'os_hidden': False, 'os_hash_algo': 'sha512', 'os_hash_value': '6b813aa46bb90b4da216a4d19376593fa3f4fc7e617f03a92b7fe11e9a3981cbe8f0959dbebe36225e5f53dc4492341a4863cac4ed1ee0909f3fc78ef9c3e869', 'id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'created_at': datetime.datetime(2023, 4, 17, 15, 19, 53, tzinfo=datetime.timezone.utc), 'updated_at': datetime.datetime(2023, 4, 17, 15, 19, 55, tzinfo=datetime.timezone.utc), 'tags': [], 'file': '/v2/images/b3f0dec1-7758-44a8-82e0-76ada8837428/file', 'properties': {'hw_rng_model': 'virtio', 'owner_specified.openstack.md5': '', 'owner_specified.openstack.object': 'images/cirros-0.5.2-x86_64-disk', 'owner_specified.openstack.sha256': ''}}, 'image_service': } Apr 17 15:50:25 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Cloning 93a32a7c-4c6a-472a-8f91-9d34558ba232 from image b3f0dec1-7758-44a8-82e0-76ada8837428 at location (None, None). {{(pid=84696) _create_from_image /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1071}} Apr 17 15:50:25 user cinder-volume[84696]: DEBUG cinder.coordination [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Acquiring lock "cinder-b3f0dec1-7758-44a8-82e0-76ada8837428" by "_prepare_image_cache_entry" {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:192}} Apr 17 15:50:25 user cinder-volume[84696]: DEBUG cinder.coordination [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Lock "cinder-b3f0dec1-7758-44a8-82e0-76ada8837428" acquired by "_prepare_image_cache_entry" :: waited 0.008s {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:196}} Apr 17 15:50:25 user cinder-volume[84696]: DEBUG cinder.image.cache [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Found image-volume cache entry: {'id': 1, 'image_id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'volume_id': '006f20ae-5306-44fe-afdf-e433f79284e8', 'host': 'user@joviandss-2#joviandss-2', 'size': 1, 'image_updated_at': datetime.datetime(2023, 4, 17, 15, 19, 55), 'last_used': datetime.datetime(2023, 4, 17, 15, 50, 25, 118575)}. {{(pid=84696) get_entry /opt/stack/cinder/cinder/image/cache.py:76}} Apr 17 15:50:25 user cinder-volume[84696]: DEBUG cinder.image.cache [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Image-volume cache entry image_update_at = 2023-04-17 15:19:55+00:00, requested image updated_at = 2023-04-17 15:19:55+00:00. {{(pid=84696) _should_update_entry /opt/stack/cinder/cinder/image/cache.py:259}} Apr 17 15:50:25 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Found cache entry for image = b3f0dec1-7758-44a8-82e0-76ada8837428 on host user@joviandss-2#joviandss-2. {{(pid=84696) _prepare_image_cache_entry /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:886}} Apr 17 15:50:25 user cinder-volume[84696]: DEBUG cinder.coordination [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Lock "cinder-b3f0dec1-7758-44a8-82e0-76ada8837428" released by "_prepare_image_cache_entry" :: held 0.014s {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:207}} Apr 17 15:50:25 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Attempting to retrieve cache entry for image = b3f0dec1-7758-44a8-82e0-76ada8837428 on host user@joviandss-2#joviandss-2. {{(pid=84696) _create_from_image_cache /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:827}} Apr 17 15:50:25 user cinder-volume[84696]: DEBUG cinder.image.cache [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Found image-volume cache entry: {'id': 1, 'image_id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'volume_id': '006f20ae-5306-44fe-afdf-e433f79284e8', 'host': 'user@joviandss-2#joviandss-2', 'size': 1, 'image_updated_at': datetime.datetime(2023, 4, 17, 15, 19, 55), 'last_used': datetime.datetime(2023, 4, 17, 15, 50, 25, 131740)}. {{(pid=84696) get_entry /opt/stack/cinder/cinder/image/cache.py:76}} Apr 17 15:50:25 user cinder-volume[84696]: DEBUG cinder.image.cache [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Image-volume cache entry image_update_at = 2023-04-17 15:19:55+00:00, requested image updated_at = 2023-04-17 15:19:55+00:00. {{(pid=84696) _should_update_entry /opt/stack/cinder/cinder/image/cache.py:259}} Apr 17 15:50:25 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Creating from source image-volume 006f20ae-5306-44fe-afdf-e433f79284e8 {{(pid=84696) _create_from_image_cache /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:841}} Apr 17 15:50:25 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] cloned volume 006f20ae-5306-44fe-afdf-e433f79284e8 to 93a32a7c-4c6a-472a-8f91-9d34558ba232 {{(pid=84696) create_cloned_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:521}} Apr 17 15:50:25 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] cloning v_006f20ae-5306-44fe-afdf-e433f79284e8 to v_93a32a7c-4c6a-472a-8f91-9d34558ba232 {{(pid=84696) _clone_object /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:456}} Apr 17 15:50:25 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] create snapshot v_93a32a7c-4c6a-472a-8f91-9d34558ba232 {{(pid=84696) create_snapshot /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:635}} Apr 17 15:50:25 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:50:25 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:50:27 user cinder-volume[84696]: DEBUG cinder.utils [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.016(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:50:27 user cinder-volume[84696]: DEBUG cinder.utils [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:50:27 user cinder-volume[84696]: DEBUG cinder.utils [req-778c62be-8963-4e6d-97f1-f456b759ce16 req-8dba5389-38e6-4a46-b382-2d4356cad7db tempest-AttachVolumeShelveTestJSON-1745617049 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 18.709(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:50:27 user cinder-volume[84696]: DEBUG cinder.utils [req-778c62be-8963-4e6d-97f1-f456b759ce16 req-8dba5389-38e6-4a46-b382-2d4356cad7db tempest-AttachVolumeShelveTestJSON-1745617049 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:13bf568b-585c-4556-ab78-e9930ad50214/luns/v_13bf568b-585c-4556-ab78-e9930ad50214 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:50:27 user cinder-volume[84696]: DEBUG cinder.utils [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.631(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:50:27 user cinder-volume[84696]: DEBUG cinder.utils [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:50:27 user cinder-volume[84696]: DEBUG cinder.utils [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.189(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:50:27 user cinder-volume[84696]: DEBUG cinder.utils [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_cdedb820-e1c7-4b86-8286-43b3640b3a46/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:50:27 user cinder-volume[84696]: DEBUG cinder.utils [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.988(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:50:27 user cinder-volume[84696]: DEBUG cinder.utils [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_93a32a7c-4c6a-472a-8f91-9d34558ba232/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:50:27 user cinder-volume[84696]: DEBUG cinder.utils [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 72.191(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:50:27 user cinder-volume[84696]: DEBUG cinder.utils [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:50:30 user cinder-volume[84696]: DEBUG cinder.utils [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 74.174(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:50:30 user cinder-volume[84696]: DEBUG cinder.utils [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_fbbadf16-651a-47d5-b308-a37e56aa07f2/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:50:30 user cinder-volume[84696]: DEBUG cinder.utils [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 73.807(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:50:30 user cinder-volume[84696]: DEBUG cinder.utils [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:50:30 user cinder-volume[84696]: DEBUG cinder.utils [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 73.849(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:50:30 user cinder-volume[84696]: DEBUG cinder.utils [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:50:30 user cinder-volume[84696]: DEBUG cinder.utils [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 74.185(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:50:30 user cinder-volume[84696]: DEBUG cinder.utils [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_dfd745ed-b0c4-4e35-b77c-a911c90ce33e/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:50:30 user cinder-volume[84696]: DEBUG cinder.utils [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 74.205(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:50:30 user cinder-volume[84696]: DEBUG cinder.utils [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_aea45c70-7c05-4146-847e-cbb777d3dfbe/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:50:30 user cinder-volume[84696]: DEBUG cinder.utils [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.054(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:50:30 user cinder-volume[84696]: DEBUG cinder.utils [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_93a32a7c-4c6a-472a-8f91-9d34558ba232/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:50:30 user cinder-volume[84696]: DEBUG cinder.utils [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.701(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:50:30 user cinder-volume[84696]: DEBUG cinder.utils [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:50:30 user cinder-volume[84696]: DEBUG cinder.utils [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 73.963(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:50:30 user cinder-volume[84696]: DEBUG cinder.utils [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:50:33 user cinder-volume[84696]: DEBUG cinder.utils [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 138.219(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:50:33 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_441910f8-d29c-4c9e-bc5b-780ad228c85d/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:50:33 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (282c69f7-04a6-489b-b737-2ce95b429beb) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 15:50:33 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (f5a689b7-5e75-4896-96de-bb13400ad29d) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:50:33 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (f5a689b7-5e75-4896-96de-bb13400ad29d) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:50:33 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Flow 'volume_create_manager' (9fa28ced-2781-4f50-a11f-28ca658d5dc7) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 15:50:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-51924a9e-5094-404b-8828-f76cf8bc761a tempest-VolumesActionsTest-707453245 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 15:50:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 15:50:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 15:50:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 15:50:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 15:50:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 15:50:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 15:50:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 15:50:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 15:50:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 15:50:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 15:50:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 15:50:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 15:50:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 15:50:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 15:50:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 15:50:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 15:50:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 15:50:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 15:50:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 15:50:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 15:50:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 15:50:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 15:50:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 15:50:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:50:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 15:50:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 15:50:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 15:50:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 15:50:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 15:50:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 15:50:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 15:50:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 15:50:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 15:50:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 15:50:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 15:50:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 15:50:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 15:50:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 15:50:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 15:50:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 15:50:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 15:50:33 user cinder-volume[84696]: DEBUG cinder.utils [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 74.315(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:50:33 user cinder-volume[84696]: DEBUG cinder.utils [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_ab5889f8-b6c7-4673-a666-f4b4e5ff5fd5/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:50:33 user cinder-volume[84696]: DEBUG cinder.utils [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 74.006(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:50:33 user cinder-volume[84696]: DEBUG cinder.utils [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:50:33 user cinder-volume[84696]: DEBUG cinder.utils [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 73.935(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:50:33 user cinder-volume[84696]: DEBUG cinder.utils [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:50:33 user cinder-volume[84696]: DEBUG cinder.utils [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 74.327(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:50:33 user cinder-volume[84696]: DEBUG cinder.utils [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_13cc73df-7780-4934-8cbd-7f372b1cf57a/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:50:36 user cinder-volume[84696]: DEBUG cinder.utils [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 11.840(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:50:36 user cinder-volume[84696]: DEBUG cinder.utils [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:50:36 user cinder-volume[84696]: DEBUG cinder.utils [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 74.132(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:50:36 user cinder-volume[84696]: DEBUG cinder.utils [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:50:36 user cinder-volume[84696]: DEBUG cinder.utils [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 135.160(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:50:36 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_5c16bb82-43c5-4d91-9320-0e1f2bbec574/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:50:37 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (23135497-50b1-444f-9668-b80977e69ae3) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 15:50:37 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (edd27c77-19b6-4c8d-a123-8b0c447218e5) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:50:37 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (edd27c77-19b6-4c8d-a123-8b0c447218e5) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:50:37 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Flow 'volume_create_manager' (feb9fcc8-fc74-4994-8e14-fb8d6e4a7172) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 15:50:37 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 15:50:37 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 15:50:37 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 15:50:37 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 15:50:37 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 15:50:37 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 15:50:37 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 15:50:37 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 15:50:37 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 15:50:37 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 15:50:37 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 15:50:37 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 15:50:37 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 15:50:37 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 15:50:37 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 15:50:37 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 15:50:37 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 15:50:37 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 15:50:37 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 15:50:37 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 15:50:37 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 15:50:37 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 15:50:37 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 15:50:37 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 15:50:37 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:50:37 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 15:50:37 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 15:50:37 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 15:50:37 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 15:50:37 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 15:50:37 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 15:50:37 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 15:50:37 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 15:50:37 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 15:50:37 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 15:50:37 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 15:50:37 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 15:50:37 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 15:50:37 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 15:50:37 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 15:50:37 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 15:50:37 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 15:50:37 user cinder-volume[84696]: DEBUG cinder.utils [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 74.339(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:50:37 user cinder-volume[84696]: DEBUG cinder.utils [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_68f814b0-1291-4a12-ae65-dc0892203d36/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:50:37 user cinder-volume[84696]: DEBUG cinder.utils [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.225(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:50:37 user cinder-volume[84696]: DEBUG cinder.utils [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_93a32a7c-4c6a-472a-8f91-9d34558ba232/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:50:37 user cinder-volume[84696]: DEBUG cinder.utils [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 135.011(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:50:37 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:50:37 user cinder-volume[84696]: INFO cinder.message.api [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Creating message record for request_id = req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b Apr 17 15:50:37 user cinder-volume[84696]: DEBUG cinder.utils [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.433(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:50:37 user cinder-volume[84696]: DEBUG cinder.utils [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_cdedb820-e1c7-4b86-8286-43b3640b3a46/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:50:37 user cinder-volume[84696]: DEBUG cinder.utils [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.265(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:50:37 user cinder-volume[84696]: DEBUG cinder.utils [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:50:37 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 15:50:37 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=25,cinder:UPDATE=3,cinder:INSERT=1 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:50:37 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (f40cf3e8-b8e2-46c1-bf46-336c8a2ab7bf) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 15:50:37 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 15:50:37 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:43:43Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TestMinimumBasicScenario-volume-1067752463',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=5c16bb82-43c5-4d91-9320-0e1f2bbec574,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='17d1dae0ad7149bbbd160525d64c0c20',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:48:22Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:48:22Z,use_quota=True,user_id='3c407d9d494d485c9c95dc3981284fba',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 15:50:37 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:43:43Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TestMinimumBasicScenario-volume-1067752463',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=5c16bb82-43c5-4d91-9320-0e1f2bbec574,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='17d1dae0ad7149bbbd160525d64c0c20',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:48:22Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:48:22Z,use_quota=True,user_id='3c407d9d494d485c9c95dc3981284fba',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(5c16bb82-43c5-4d91-9320-0e1f2bbec574),volume_id=5c16bb82-43c5-4d91-9320-0e1f2bbec574,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': '5c16bb82-43c5-4d91-9320-0e1f2bbec574', 'volume_name': 'volume-5c16bb82-43c5-4d91-9320-0e1f2bbec574', 'volume_size': 1}} Apr 17 15:50:37 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': '5c16bb82-43c5-4d91-9320-0e1f2bbec574', 'volume': {'id': '5c16bb82-43c5-4d91-9320-0e1f2bbec574', '_name_id': None, 'ec2_id': None, 'user_id': '3c407d9d494d485c9c95dc3981284fba', 'project_id': '17d1dae0ad7149bbbd160525d64c0c20', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-TestMinimumBasicScenario-volume-1067752463', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'admin_metadata': {}, 'glance_metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T15:43:43.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-5c16bb82-43c5-4d91-9320-0e1f2bbec574', 'name_id': '5c16bb82-43c5-4d91-9320-0e1f2bbec574', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-TestMinimumBasicScenario-volume-1067752463', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '17d1dae0ad7149bbbd160525d64c0c20', 'qos_specs': None, 'reservations': ['dc14bf14-c68a-4eb7-b82a-f5cc6f1cd0d8', 'bc18e142-51f5-4393-aeb8-9697eb8e95e4', '3bbb7ba4-5edd-410f-beff-181109d36e4a', 'd2dbd02b-21f5-42b4-a139-2855ceadc14e'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': '3c407d9d494d485c9c95dc3981284fba', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-TestMinimumBasicScenario-volume-1067752463', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '17d1dae0ad7149bbbd160525d64c0c20', 'qos_specs': None, 'reservations': ['dc14bf14-c68a-4eb7-b82a-f5cc6f1cd0d8', 'bc18e142-51f5-4393-aeb8-9697eb8e95e4', '3bbb7ba4-5edd-410f-beff-181109d36e4a', 'd2dbd02b-21f5-42b4-a139-2855ceadc14e'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': '3c407d9d494d485c9c95dc3981284fba', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': '3c407d9d494d485c9c95dc3981284fba', 'metadata': {}, 'qos_specs': None, 'retry': {'num_attempts': 3, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']}}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(5c16bb82-43c5-4d91-9320-0e1f2bbec574),volume_id=5c16bb82-43c5-4d91-9320-0e1f2bbec574,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:43:43Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TestMinimumBasicScenario-volume-1067752463',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=5c16bb82-43c5-4d91-9320-0e1f2bbec574,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='17d1dae0ad7149bbbd160525d64c0c20',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:48:22Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:48:22Z,use_quota=True,user_id='3c407d9d494d485c9c95dc3981284fba',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 15:50:37 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:43:43Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TestMinimumBasicScenario-volume-1067752463',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=5c16bb82-43c5-4d91-9320-0e1f2bbec574,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='17d1dae0ad7149bbbd160525d64c0c20',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:48:22Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:48:22Z,use_quota=True,user_id='3c407d9d494d485c9c95dc3981284fba',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:43:43Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TestMinimumBasicScenario-volume-1067752463',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=5c16bb82-43c5-4d91-9320-0e1f2bbec574,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='17d1dae0ad7149bbbd160525d64c0c20',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:48:22Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:48:22Z,use_quota=True,user_id='3c407d9d494d485c9c95dc3981284fba',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 15:50:37 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 15:50:37 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 15:50:37 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:50:37 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 15:50:37 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 15:50:37 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 15:50:37 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 15:50:37 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 15:50:37 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 15:50:37 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 15:50:37 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 15:50:37 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 15:50:37 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 15:50:37 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 15:50:37 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 15:50:37 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 15:50:37 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 15:50:37 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 15:50:37 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 15:50:37 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 15:50:37 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (f40cf3e8-b8e2-46c1-bf46-336c8a2ab7bf) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:50:37 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (f40cf3e8-b8e2-46c1-bf46-336c8a2ab7bf) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:50:37 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (194a7a34-0f2c-4b9b-a785-ee7c4c4bb522) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:50:37 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (194a7a34-0f2c-4b9b-a785-ee7c4c4bb522) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:50:37 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (ec3d1ee5-01d1-4a00-b734-8a021097f480) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:50:37 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (ec3d1ee5-01d1-4a00-b734-8a021097f480) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:50:37 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (e09b31ab-cc40-4bfc-95b8-19c7269b9325) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:50:37 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Updating volume 5c16bb82-43c5-4d91-9320-0e1f2bbec574 with {'scheduled_at': datetime.datetime(2023, 4, 17, 15, 50, 37, 90975), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 15:50:37 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Volume 5c16bb82-43c5-4d91-9320-0e1f2bbec574: re-scheduling SchedulerAPI.create_volume attempt 3 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 15:50:37 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Volume 5c16bb82-43c5-4d91-9320-0e1f2bbec574: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 15:50:37 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] deleating volume v_5c16bb82-43c5-4d91-9320-0e1f2bbec574 {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 15:50:37 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] get snapshots for volume v_5c16bb82-43c5-4d91-9320-0e1f2bbec574 {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 15:50:37 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_5c16bb82-43c5-4d91-9320-0e1f2bbec574/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:50:37 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_5c16bb82-43c5-4d91-9320-0e1f2bbec574/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:50:40 user cinder-volume[84696]: DEBUG cinder.utils [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.881(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:50:40 user cinder-volume[84696]: DEBUG cinder.utils [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_5c16bb82-43c5-4d91-9320-0e1f2bbec574/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:50:43 user cinder-volume[84696]: DEBUG cinder.utils [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 38.266(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:50:43 user cinder-volume[84696]: DEBUG cinder.utils [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:50:43 user cinder-volume[84696]: DEBUG cinder.utils [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.954(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:50:43 user cinder-volume[84696]: DEBUG cinder.utils [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_5c16bb82-43c5-4d91-9320-0e1f2bbec574/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:50:46 user cinder-volume[84696]: DEBUG cinder.utils [req-778c62be-8963-4e6d-97f1-f456b759ce16 req-8dba5389-38e6-4a46-b382-2d4356cad7db tempest-AttachVolumeShelveTestJSON-1745617049 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 37.135(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:50:46 user cinder-volume[84696]: DEBUG cinder.utils [req-778c62be-8963-4e6d-97f1-f456b759ce16 req-8dba5389-38e6-4a46-b382-2d4356cad7db tempest-AttachVolumeShelveTestJSON-1745617049 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:13bf568b-585c-4556-ab78-e9930ad50214/luns/v_13bf568b-585c-4556-ab78-e9930ad50214 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:50:46 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 71.419(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:50:46 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:50:46 user cinder-volume[84696]: DEBUG cinder.utils [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.059(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:50:46 user cinder-volume[84696]: DEBUG cinder.utils [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:50:46 user cinder-volume[84696]: DEBUG cinder.utils [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.415(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:50:46 user cinder-volume[84696]: DEBUG cinder.utils [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_93a32a7c-4c6a-472a-8f91-9d34558ba232/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:50:49 user cinder-volume[84696]: DEBUG cinder.utils [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.097(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:50:49 user cinder-volume[84696]: DEBUG cinder.utils [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_5c16bb82-43c5-4d91-9320-0e1f2bbec574/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:50:54 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:UPDATE=9,cinder:SELECT=9,cinder:INSERT=2 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:50:55 user cinder-volume[84696]: DEBUG cinder.utils [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.832(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:50:55 user cinder-volume[84696]: DEBUG cinder.utils [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_cdedb820-e1c7-4b86-8286-43b3640b3a46/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:50:55 user cinder-volume[84696]: DEBUG cinder.utils [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.665(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:50:55 user cinder-volume[84696]: DEBUG cinder.utils [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:50:55 user cinder-volume[84696]: DEBUG cinder.utils [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 138.051(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:50:55 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:50:55 user cinder-volume[84696]: INFO cinder.message.api [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Creating message record for request_id = req-636827e5-ca50-4d75-a738-9b54469bf4fe Apr 17 15:50:55 user cinder-volume[84696]: DEBUG cinder.utils [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 138.240(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:50:55 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_2c9714de-f8ea-4d1d-99f9-beb85126ad91/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:50:55 user cinder-volume[84696]: WARNING cinder.volume.manager [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (6d6245bf-af9c-4f2e-b7ff-5d36fbf9d72d) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 15:50:55 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (ce5b4d58-ae31-4588-b654-7c512790cdca) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:50:55 user cinder-volume[84696]: WARNING cinder.volume.manager [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (ce5b4d58-ae31-4588-b654-7c512790cdca) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:50:55 user cinder-volume[84696]: WARNING cinder.volume.manager [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Flow 'volume_create_manager' (26920333-f21d-422c-8a95-071d6214fa61) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 15:50:55 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 15:50:55 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 15:50:55 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 15:50:55 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 15:50:55 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 15:50:55 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 15:50:55 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 15:50:55 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 15:50:55 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 15:50:55 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 15:50:55 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 15:50:55 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 15:50:55 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 15:50:55 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 15:50:55 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 15:50:55 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 15:50:55 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 15:50:55 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 15:50:55 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 15:50:55 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 15:50:55 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 15:50:55 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 15:50:55 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 15:50:55 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 15:50:55 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:50:55 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 15:50:55 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 15:50:55 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 15:50:55 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 15:50:55 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 15:50:55 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 15:50:55 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 15:50:55 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 15:50:55 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 15:50:55 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 15:50:55 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 15:50:55 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 15:50:55 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 15:50:55 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 15:50:55 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 15:50:55 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 15:50:55 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 15:50:55 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 15:50:55 user cinder-volume[84696]: WARNING cinder.volume.manager [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (4efd3e1a-02eb-469a-a755-003065d9b7d2) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 15:50:55 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 15:50:55 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:44:03Z,deleted=False,deleted_at=None,display_description='',display_name='05e08eea-cc2c-4332-9e97-07cb9b466af6-blank-vol',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=2c9714de-f8ea-4d1d-99f9-beb85126ad91,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='540826be4baf46dba9617b2fd47821a3',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:48:37Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:48:37Z,use_quota=True,user_id='d819e3f51368413c99691d1b45bbcfb1',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 15:50:55 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:44:03Z,deleted=False,deleted_at=None,display_description='',display_name='05e08eea-cc2c-4332-9e97-07cb9b466af6-blank-vol',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=2c9714de-f8ea-4d1d-99f9-beb85126ad91,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='540826be4baf46dba9617b2fd47821a3',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:48:37Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:48:37Z,use_quota=True,user_id='d819e3f51368413c99691d1b45bbcfb1',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(2c9714de-f8ea-4d1d-99f9-beb85126ad91),volume_id=2c9714de-f8ea-4d1d-99f9-beb85126ad91,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': '2c9714de-f8ea-4d1d-99f9-beb85126ad91', 'volume_name': 'volume-2c9714de-f8ea-4d1d-99f9-beb85126ad91', 'volume_size': 1}} Apr 17 15:50:55 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': '2c9714de-f8ea-4d1d-99f9-beb85126ad91', 'volume': {'id': '2c9714de-f8ea-4d1d-99f9-beb85126ad91', '_name_id': None, 'ec2_id': None, 'user_id': 'd819e3f51368413c99691d1b45bbcfb1', 'project_id': '540826be4baf46dba9617b2fd47821a3', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': '05e08eea-cc2c-4332-9e97-07cb9b466af6-blank-vol', 'display_description': '', 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'admin_metadata': {}, 'glance_metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T15:44:02.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-2c9714de-f8ea-4d1d-99f9-beb85126ad91', 'name_id': '2c9714de-f8ea-4d1d-99f9-beb85126ad91', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': '', 'display_name': '05e08eea-cc2c-4332-9e97-07cb9b466af6-blank-vol', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '540826be4baf46dba9617b2fd47821a3', 'qos_specs': None, 'reservations': ['e20dc1ca-0a9e-4413-be1e-c7f19faae9e1', '81c416d5-515b-4265-888a-740c186d6f45', 'da7244e3-def8-4c3b-b96c-0088e02b1914', 'afef167b-9548-4f1a-a499-b206d13c826a'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'd819e3f51368413c99691d1b45bbcfb1', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': '', 'display_name': '05e08eea-cc2c-4332-9e97-07cb9b466af6-blank-vol', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '540826be4baf46dba9617b2fd47821a3', 'qos_specs': None, 'reservations': ['e20dc1ca-0a9e-4413-be1e-c7f19faae9e1', '81c416d5-515b-4265-888a-740c186d6f45', 'da7244e3-def8-4c3b-b96c-0088e02b1914', 'afef167b-9548-4f1a-a499-b206d13c826a'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'd819e3f51368413c99691d1b45bbcfb1', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': 'd819e3f51368413c99691d1b45bbcfb1', 'metadata': {}, 'qos_specs': None, 'retry': {'num_attempts': 3, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']}}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(2c9714de-f8ea-4d1d-99f9-beb85126ad91),volume_id=2c9714de-f8ea-4d1d-99f9-beb85126ad91,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:44:03Z,deleted=False,deleted_at=None,display_description='',display_name='05e08eea-cc2c-4332-9e97-07cb9b466af6-blank-vol',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=2c9714de-f8ea-4d1d-99f9-beb85126ad91,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='540826be4baf46dba9617b2fd47821a3',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:48:37Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:48:37Z,use_quota=True,user_id='d819e3f51368413c99691d1b45bbcfb1',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 15:50:55 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:44:03Z,deleted=False,deleted_at=None,display_description='',display_name='05e08eea-cc2c-4332-9e97-07cb9b466af6-blank-vol',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=2c9714de-f8ea-4d1d-99f9-beb85126ad91,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='540826be4baf46dba9617b2fd47821a3',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:48:37Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:48:37Z,use_quota=True,user_id='d819e3f51368413c99691d1b45bbcfb1',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:44:03Z,deleted=False,deleted_at=None,display_description='',display_name='05e08eea-cc2c-4332-9e97-07cb9b466af6-blank-vol',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=2c9714de-f8ea-4d1d-99f9-beb85126ad91,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='540826be4baf46dba9617b2fd47821a3',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:48:37Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:48:37Z,use_quota=True,user_id='d819e3f51368413c99691d1b45bbcfb1',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 15:50:55 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 15:50:55 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 15:50:55 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:50:55 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 15:50:55 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 15:50:55 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 15:50:55 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 15:50:55 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 15:50:55 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 15:50:55 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 15:50:55 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 15:50:55 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 15:50:55 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 15:50:55 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 15:50:55 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 15:50:55 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 15:50:55 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 15:50:55 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 15:50:55 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 15:50:55 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 15:50:55 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (4efd3e1a-02eb-469a-a755-003065d9b7d2) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:50:55 user cinder-volume[84696]: WARNING cinder.volume.manager [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (4efd3e1a-02eb-469a-a755-003065d9b7d2) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:50:55 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (b9052dc1-84b3-4996-9500-5074d44e289a) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:50:55 user cinder-volume[84696]: WARNING cinder.volume.manager [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (b9052dc1-84b3-4996-9500-5074d44e289a) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:50:55 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (8f68d0d2-69e4-463b-a42e-eb323b7280a2) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:50:55 user cinder-volume[84696]: WARNING cinder.volume.manager [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (8f68d0d2-69e4-463b-a42e-eb323b7280a2) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:50:55 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (2e236988-b8a4-435d-9f21-a584f5eac774) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:50:55 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Updating volume 2c9714de-f8ea-4d1d-99f9-beb85126ad91 with {'scheduled_at': datetime.datetime(2023, 4, 17, 15, 50, 55, 505783), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 15:50:55 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Volume 2c9714de-f8ea-4d1d-99f9-beb85126ad91: re-scheduling SchedulerAPI.create_volume attempt 3 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 15:50:55 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Volume 2c9714de-f8ea-4d1d-99f9-beb85126ad91: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 15:50:55 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] deleating volume v_2c9714de-f8ea-4d1d-99f9-beb85126ad91 {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 15:50:55 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] get snapshots for volume v_2c9714de-f8ea-4d1d-99f9-beb85126ad91 {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 15:50:55 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_2c9714de-f8ea-4d1d-99f9-beb85126ad91/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:50:55 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_2c9714de-f8ea-4d1d-99f9-beb85126ad91/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:50:58 user cinder-volume[84696]: DEBUG cinder.utils [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.960(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:50:58 user cinder-volume[84696]: DEBUG cinder.utils [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_2c9714de-f8ea-4d1d-99f9-beb85126ad91/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:50:58 user cinder-volume[84696]: DEBUG cinder.utils [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.315(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:50:58 user cinder-volume[84696]: DEBUG cinder.utils [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_5c16bb82-43c5-4d91-9320-0e1f2bbec574/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:51:01 user cinder-volume[84696]: DEBUG cinder.utils [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.032(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:51:01 user cinder-volume[84696]: DEBUG cinder.utils [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_2c9714de-f8ea-4d1d-99f9-beb85126ad91/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:51:04 user cinder-volume[84696]: DEBUG cinder.utils [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.487(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:51:04 user cinder-volume[84696]: DEBUG cinder.utils [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:51:04 user cinder-volume[84696]: DEBUG cinder.utils [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.843(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:51:04 user cinder-volume[84696]: DEBUG cinder.utils [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_93a32a7c-4c6a-472a-8f91-9d34558ba232/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:51:07 user cinder-volume[84696]: DEBUG cinder.utils [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.176(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:51:07 user cinder-volume[84696]: DEBUG cinder.utils [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_2c9714de-f8ea-4d1d-99f9-beb85126ad91/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:51:14 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=9,cinder:UPDATE=2,cinder:INSERT=1 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:51:17 user cinder-volume[84696]: DEBUG cinder.utils [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 40.193(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:51:17 user cinder-volume[84696]: DEBUG cinder.utils [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_5c16bb82-43c5-4d91-9320-0e1f2bbec574/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:51:17 user cinder-volume[84696]: DEBUG cinder.utils [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 72.508(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:51:17 user cinder-volume[84696]: DEBUG cinder.utils [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:51:17 user cinder-volume[84696]: DEBUG cinder.utils [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.843(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:51:17 user cinder-volume[84696]: DEBUG cinder.utils [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_2c9714de-f8ea-4d1d-99f9-beb85126ad91/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:51:20 user cinder-volume[84696]: DEBUG cinder.utils [req-778c62be-8963-4e6d-97f1-f456b759ce16 req-8dba5389-38e6-4a46-b382-2d4356cad7db tempest-AttachVolumeShelveTestJSON-1745617049 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 71.375(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:51:20 user cinder-volume[84696]: DEBUG cinder.utils [req-778c62be-8963-4e6d-97f1-f456b759ce16 req-8dba5389-38e6-4a46-b382-2d4356cad7db tempest-AttachVolumeShelveTestJSON-1745617049 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:13bf568b-585c-4556-ab78-e9930ad50214/luns/v_13bf568b-585c-4556-ab78-e9930ad50214 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:51:24 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=6,cinder:INSERT=1,cinder:UPDATE=3 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:51:30 user cinder-volume[84696]: DEBUG cinder.utils [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 74.804(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:51:30 user cinder-volume[84696]: DEBUG cinder.utils [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_cdedb820-e1c7-4b86-8286-43b3640b3a46/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:51:30 user cinder-volume[84696]: DEBUG cinder.utils [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 74.637(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:51:30 user cinder-volume[84696]: DEBUG cinder.utils [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:51:33 user cinder-volume[84696]: DEBUG cinder.utils [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 137.875(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:51:33 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:51:33 user cinder-volume[84696]: INFO cinder.message.api [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Creating message record for request_id = req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a Apr 17 15:51:33 user cinder-volume[84696]: DEBUG cinder.utils [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 37.940(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:51:33 user cinder-volume[84696]: DEBUG cinder.utils [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_2c9714de-f8ea-4d1d-99f9-beb85126ad91/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:51:33 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 15:51:33 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (65a59a4e-335e-48a1-a7c5-bc52c1c31f2c) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 15:51:33 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 15:51:33 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:49:15Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-AttachVolumeNegativeTest-volume-1650632398',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=4a2d1c00-a134-4f73-8508-86f35b58acf7,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='8586cc10142d4743874f43cdb0f37aba',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:49:15Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:49:15Z,use_quota=True,user_id='9fdc7c80b4654692a52c8a4c4149324f',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 15:51:33 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:49:15Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-AttachVolumeNegativeTest-volume-1650632398',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=4a2d1c00-a134-4f73-8508-86f35b58acf7,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='8586cc10142d4743874f43cdb0f37aba',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:49:15Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:49:15Z,use_quota=True,user_id='9fdc7c80b4654692a52c8a4c4149324f',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(4a2d1c00-a134-4f73-8508-86f35b58acf7),volume_id=4a2d1c00-a134-4f73-8508-86f35b58acf7,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': '4a2d1c00-a134-4f73-8508-86f35b58acf7', 'volume_name': 'volume-4a2d1c00-a134-4f73-8508-86f35b58acf7', 'volume_size': 1}} Apr 17 15:51:33 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'retry': {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']}, 'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': '4a2d1c00-a134-4f73-8508-86f35b58acf7', 'volume': {'id': '4a2d1c00-a134-4f73-8508-86f35b58acf7', '_name_id': None, 'ec2_id': None, 'user_id': '9fdc7c80b4654692a52c8a4c4149324f', 'project_id': '8586cc10142d4743874f43cdb0f37aba', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-AttachVolumeNegativeTest-volume-1650632398', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T15:49:15.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-4a2d1c00-a134-4f73-8508-86f35b58acf7', 'name_id': '4a2d1c00-a134-4f73-8508-86f35b58acf7', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-AttachVolumeNegativeTest-volume-1650632398', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '8586cc10142d4743874f43cdb0f37aba', 'qos_specs': None, 'reservations': ['ff09af43-7592-43c3-b526-fbe9f6237e58', '0c1bba75-a6c1-4cba-ad55-d6fa91e1347c', 'a45604df-7ba9-46e4-9aff-b47f55715a32', 'eb18d99b-bfb4-4076-a464-297431791417'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': '9fdc7c80b4654692a52c8a4c4149324f', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-AttachVolumeNegativeTest-volume-1650632398', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '8586cc10142d4743874f43cdb0f37aba', 'qos_specs': None, 'reservations': ['ff09af43-7592-43c3-b526-fbe9f6237e58', '0c1bba75-a6c1-4cba-ad55-d6fa91e1347c', 'a45604df-7ba9-46e4-9aff-b47f55715a32', 'eb18d99b-bfb4-4076-a464-297431791417'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': '9fdc7c80b4654692a52c8a4c4149324f', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': '9fdc7c80b4654692a52c8a4c4149324f', 'metadata': {}, 'qos_specs': None}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(4a2d1c00-a134-4f73-8508-86f35b58acf7),volume_id=4a2d1c00-a134-4f73-8508-86f35b58acf7,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:49:15Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-AttachVolumeNegativeTest-volume-1650632398',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=4a2d1c00-a134-4f73-8508-86f35b58acf7,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='8586cc10142d4743874f43cdb0f37aba',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:49:15Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:49:15Z,use_quota=True,user_id='9fdc7c80b4654692a52c8a4c4149324f',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 15:51:33 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:49:15Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-AttachVolumeNegativeTest-volume-1650632398',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=4a2d1c00-a134-4f73-8508-86f35b58acf7,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='8586cc10142d4743874f43cdb0f37aba',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:49:15Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:49:15Z,use_quota=True,user_id='9fdc7c80b4654692a52c8a4c4149324f',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:49:15Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-AttachVolumeNegativeTest-volume-1650632398',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=4a2d1c00-a134-4f73-8508-86f35b58acf7,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='8586cc10142d4743874f43cdb0f37aba',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:49:15Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:49:15Z,use_quota=True,user_id='9fdc7c80b4654692a52c8a4c4149324f',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 15:51:33 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 15:51:33 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 15:51:33 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:51:33 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 15:51:33 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 15:51:33 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 15:51:33 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 15:51:33 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 15:51:33 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 15:51:33 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 15:51:33 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 15:51:33 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 15:51:33 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 15:51:33 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 15:51:33 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 15:51:33 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 15:51:33 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 15:51:33 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 15:51:33 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 15:51:33 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 15:51:33 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (65a59a4e-335e-48a1-a7c5-bc52c1c31f2c) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:51:33 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (65a59a4e-335e-48a1-a7c5-bc52c1c31f2c) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:51:33 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (18626c41-c279-460e-b48f-ac0f1e256fdc) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:51:33 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (18626c41-c279-460e-b48f-ac0f1e256fdc) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:51:33 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (cc3875b0-1df3-4f2c-9e20-c10a31bc194c) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:51:33 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (cc3875b0-1df3-4f2c-9e20-c10a31bc194c) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:51:33 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (057c958c-d875-42c6-8087-149e232298b2) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:51:33 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Updating volume 4a2d1c00-a134-4f73-8508-86f35b58acf7 with {'scheduled_at': datetime.datetime(2023, 4, 17, 15, 51, 33, 525561), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 15:51:33 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Volume 4a2d1c00-a134-4f73-8508-86f35b58acf7: re-scheduling SchedulerAPI.create_volume attempt 1 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 15:51:33 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Volume 4a2d1c00-a134-4f73-8508-86f35b58acf7: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 15:51:33 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] deleating volume v_4a2d1c00-a134-4f73-8508-86f35b58acf7 {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 15:51:33 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] get snapshots for volume v_4a2d1c00-a134-4f73-8508-86f35b58acf7 {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 15:51:33 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_4a2d1c00-a134-4f73-8508-86f35b58acf7/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:51:33 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_4a2d1c00-a134-4f73-8508-86f35b58acf7/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:51:33 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Volume reschedule parameters: True retry: {'num_attempts': 2, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 15:51:33 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Flow 'volume_create_manager' (d2e0780c-751d-4669-a6a1-af8229f83db7) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:51:33 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (27632b54-03ef-4088-958f-dcf41f8a56ad) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:51:33 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (27632b54-03ef-4088-958f-dcf41f8a56ad) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:49:15Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-AttachVolumeNegativeTest-volume-1650632398',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=4a2d1c00-a134-4f73-8508-86f35b58acf7,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='8586cc10142d4743874f43cdb0f37aba',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:51:34Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:51:34Z,use_quota=True,user_id='9fdc7c80b4654692a52c8a4c4149324f',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:51:33 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (b7923afd-b289-4b11-9a5b-ae1e0e60e247) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:51:33 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (b7923afd-b289-4b11-9a5b-ae1e0e60e247) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:51:33 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (8a486fa4-d880-4c67-82ef-b013d259fe67) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:51:33 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (8a486fa4-d880-4c67-82ef-b013d259fe67) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': '4a2d1c00-a134-4f73-8508-86f35b58acf7', 'volume_name': 'volume-4a2d1c00-a134-4f73-8508-86f35b58acf7', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:51:33 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (75dce16c-0230-4a17-abfe-429811260ecb) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:51:33 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (75dce16c-0230-4a17-abfe-429811260ecb) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:51:33 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (501412f3-e67a-416e-8e57-176fdd23dbee) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:51:33 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Volume 4a2d1c00-a134-4f73-8508-86f35b58acf7: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-4a2d1c00-a134-4f73-8508-86f35b58acf7', 'volume_size': 1} Apr 17 15:51:33 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] creating volume v_4a2d1c00-a134-4f73-8508-86f35b58acf7. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 15:51:33 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] create volume {'name': 'v_4a2d1c00-a134-4f73-8508-86f35b58acf7', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 15:51:33 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:51:33 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:51:36 user cinder-volume[84696]: DEBUG cinder.utils [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.994(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:51:36 user cinder-volume[84696]: DEBUG cinder.utils [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_4a2d1c00-a134-4f73-8508-86f35b58acf7/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:51:36 user cinder-volume[84696]: DEBUG cinder.utils [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.832(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:51:36 user cinder-volume[84696]: DEBUG cinder.utils [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:51:36 user cinder-volume[84696]: DEBUG cinder.utils [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 139.885(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:51:36 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_dfd745ed-b0c4-4e35-b77c-a911c90ce33e/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:51:36 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (0da1616d-8a2c-46bb-aea1-8287b7af25cf) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 15:51:36 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (ef2fc1f0-7641-4986-a834-18c3b09a924e) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:51:36 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (ef2fc1f0-7641-4986-a834-18c3b09a924e) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:51:36 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Flow 'volume_create_manager' (b81145b7-9801-4b0e-a85a-a429119190e0) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 15:51:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 15:51:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 15:51:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 15:51:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 15:51:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 15:51:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 15:51:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 15:51:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 15:51:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 15:51:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 15:51:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 15:51:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 15:51:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 15:51:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 15:51:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 15:51:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 15:51:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 15:51:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 15:51:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 15:51:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 15:51:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 15:51:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 15:51:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 15:51:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 15:51:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:51:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 15:51:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 15:51:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 15:51:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 15:51:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 15:51:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 15:51:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 15:51:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 15:51:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 15:51:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 15:51:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 15:51:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 15:51:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 15:51:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 15:51:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 15:51:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 15:51:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 15:51:36 user cinder-volume[84696]: DEBUG cinder.utils [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 139.568(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:51:36 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:51:36 user cinder-volume[84696]: INFO cinder.message.api [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Creating message record for request_id = req-be98cecd-d344-4ab9-b5f6-c268a7041adb Apr 17 15:51:36 user cinder-volume[84696]: DEBUG cinder.utils [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 139.534(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:51:36 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:51:36 user cinder-volume[84696]: INFO cinder.message.api [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Creating message record for request_id = req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 Apr 17 15:51:36 user cinder-volume[84696]: DEBUG cinder.utils [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 139.931(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:51:36 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_aea45c70-7c05-4146-847e-cbb777d3dfbe/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:51:36 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (b5bdf986-2fce-4bd2-a19a-0e363a52af33) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 15:51:36 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (4a96466f-ae69-4442-b1ee-51acc9db10b5) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:51:36 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (4a96466f-ae69-4442-b1ee-51acc9db10b5) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:51:36 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Flow 'volume_create_manager' (e02c3aee-6db0-49bb-9cf8-8f921800fe6f) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 15:51:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 15:51:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 15:51:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 15:51:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 15:51:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 15:51:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 15:51:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 15:51:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 15:51:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 15:51:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 15:51:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 15:51:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 15:51:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 15:51:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 15:51:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 15:51:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 15:51:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 15:51:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 15:51:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 15:51:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 15:51:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 15:51:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 15:51:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 15:51:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 15:51:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:51:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 15:51:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 15:51:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 15:51:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 15:51:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 15:51:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 15:51:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 15:51:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 15:51:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 15:51:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 15:51:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 15:51:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 15:51:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 15:51:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 15:51:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 15:51:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 15:51:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 15:51:36 user cinder-volume[84696]: DEBUG cinder.utils [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 139.704(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:51:36 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:51:36 user cinder-volume[84696]: INFO cinder.message.api [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Creating message record for request_id = req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f Apr 17 15:51:36 user cinder-volume[84696]: DEBUG cinder.utils [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 139.924(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:51:36 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_fbbadf16-651a-47d5-b308-a37e56aa07f2/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:51:36 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (6af14be6-309b-47d8-9e72-4d46f866188d) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 15:51:36 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (c0efca2b-d4b5-442d-a5ed-4f1fef3b0734) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:51:36 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (c0efca2b-d4b5-442d-a5ed-4f1fef3b0734) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:51:36 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Flow 'volume_create_manager' (15d67a7b-0986-468a-b44b-c6ffaf68d3fa) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 15:51:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 15:51:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 15:51:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 15:51:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 15:51:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 15:51:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 15:51:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 15:51:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 15:51:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 15:51:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 15:51:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 15:51:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 15:51:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 15:51:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 15:51:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 15:51:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 15:51:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 15:51:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 15:51:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 15:51:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 15:51:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 15:51:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 15:51:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 15:51:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 15:51:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:51:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 15:51:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 15:51:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 15:51:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 15:51:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 15:51:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 15:51:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 15:51:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 15:51:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 15:51:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 15:51:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 15:51:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 15:51:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 15:51:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 15:51:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 15:51:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 15:51:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 15:51:36 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 15:51:36 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 15:51:36 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 15:51:36 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (46a7e832-4174-4a49-9cc7-2aa79369b69a) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 15:51:36 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 15:51:36 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:44:38Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-CreateVolumesFromImageTest-Volume-1463431787',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=dfd745ed-b0c4-4e35-b77c-a911c90ce33e,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='75babbbca67f4035a48ee8ee816b18db',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:49:17Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:49:17Z,use_quota=True,user_id='d5299006144f4ee6817349348a489017',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 15:51:36 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:44:38Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-CreateVolumesFromImageTest-Volume-1463431787',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=dfd745ed-b0c4-4e35-b77c-a911c90ce33e,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='75babbbca67f4035a48ee8ee816b18db',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:49:17Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:49:17Z,use_quota=True,user_id='d5299006144f4ee6817349348a489017',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(dfd745ed-b0c4-4e35-b77c-a911c90ce33e),volume_id=dfd745ed-b0c4-4e35-b77c-a911c90ce33e,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': 'dfd745ed-b0c4-4e35-b77c-a911c90ce33e', 'volume_name': 'volume-dfd745ed-b0c4-4e35-b77c-a911c90ce33e', 'volume_size': 1}} Apr 17 15:51:36 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': 'dfd745ed-b0c4-4e35-b77c-a911c90ce33e', 'volume': {'id': 'dfd745ed-b0c4-4e35-b77c-a911c90ce33e', '_name_id': None, 'ec2_id': None, 'user_id': 'd5299006144f4ee6817349348a489017', 'project_id': '75babbbca67f4035a48ee8ee816b18db', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-CreateVolumesFromImageTest-Volume-1463431787', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'admin_metadata': {}, 'glance_metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T15:44:38.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-dfd745ed-b0c4-4e35-b77c-a911c90ce33e', 'name_id': 'dfd745ed-b0c4-4e35-b77c-a911c90ce33e', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-CreateVolumesFromImageTest-Volume-1463431787', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '75babbbca67f4035a48ee8ee816b18db', 'qos_specs': None, 'reservations': ['24de6078-218c-412e-8c05-3e1cd2d51a18', '25d60ea2-888f-44b7-a809-be58574b0515', '8d3c3175-5e14-45c1-aac7-a222d3122bbb', '7c2bd4f1-736b-4589-98c3-db4b057ca99b'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'd5299006144f4ee6817349348a489017', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-CreateVolumesFromImageTest-Volume-1463431787', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '75babbbca67f4035a48ee8ee816b18db', 'qos_specs': None, 'reservations': ['24de6078-218c-412e-8c05-3e1cd2d51a18', '25d60ea2-888f-44b7-a809-be58574b0515', '8d3c3175-5e14-45c1-aac7-a222d3122bbb', '7c2bd4f1-736b-4589-98c3-db4b057ca99b'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'd5299006144f4ee6817349348a489017', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': 'd5299006144f4ee6817349348a489017', 'metadata': {}, 'qos_specs': None, 'retry': {'num_attempts': 3, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']}}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(dfd745ed-b0c4-4e35-b77c-a911c90ce33e),volume_id=dfd745ed-b0c4-4e35-b77c-a911c90ce33e,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:44:38Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-CreateVolumesFromImageTest-Volume-1463431787',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=dfd745ed-b0c4-4e35-b77c-a911c90ce33e,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='75babbbca67f4035a48ee8ee816b18db',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:49:17Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:49:17Z,use_quota=True,user_id='d5299006144f4ee6817349348a489017',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 15:51:36 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:44:38Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-CreateVolumesFromImageTest-Volume-1463431787',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=dfd745ed-b0c4-4e35-b77c-a911c90ce33e,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='75babbbca67f4035a48ee8ee816b18db',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:49:17Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:49:17Z,use_quota=True,user_id='d5299006144f4ee6817349348a489017',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:44:38Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-CreateVolumesFromImageTest-Volume-1463431787',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=dfd745ed-b0c4-4e35-b77c-a911c90ce33e,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='75babbbca67f4035a48ee8ee816b18db',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:49:17Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:49:17Z,use_quota=True,user_id='d5299006144f4ee6817349348a489017',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 15:51:36 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 15:51:36 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 15:51:36 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:51:36 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 15:51:36 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 15:51:36 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 15:51:36 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 15:51:36 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 15:51:36 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 15:51:36 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 15:51:36 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 15:51:36 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 15:51:36 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 15:51:36 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 15:51:36 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 15:51:36 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 15:51:36 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 15:51:36 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 15:51:36 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 15:51:36 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 15:51:36 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (46a7e832-4174-4a49-9cc7-2aa79369b69a) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:51:36 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (46a7e832-4174-4a49-9cc7-2aa79369b69a) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:51:36 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (7418534d-6ff2-442f-ac11-83607405a06c) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:51:36 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (7418534d-6ff2-442f-ac11-83607405a06c) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:51:36 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (a4663525-c05a-4433-b540-b8941b417aca) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:51:36 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (a4663525-c05a-4433-b540-b8941b417aca) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:51:36 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (68d65eb2-d76e-4966-8ef0-9d655bd4d1a2) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:51:36 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Updating volume dfd745ed-b0c4-4e35-b77c-a911c90ce33e with {'scheduled_at': datetime.datetime(2023, 4, 17, 15, 51, 36, 668575), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 15:51:36 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (4681bef3-39fb-4da6-8f26-0a7615c4b427) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 15:51:36 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 15:51:36 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:44:39Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-CreateVolumesFromImageTest-Volume-800604710',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=aea45c70-7c05-4146-847e-cbb777d3dfbe,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='75babbbca67f4035a48ee8ee816b18db',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:49:17Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:49:17Z,use_quota=True,user_id='d5299006144f4ee6817349348a489017',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 15:51:36 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:44:39Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-CreateVolumesFromImageTest-Volume-800604710',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=aea45c70-7c05-4146-847e-cbb777d3dfbe,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='75babbbca67f4035a48ee8ee816b18db',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:49:17Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:49:17Z,use_quota=True,user_id='d5299006144f4ee6817349348a489017',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(aea45c70-7c05-4146-847e-cbb777d3dfbe),volume_id=aea45c70-7c05-4146-847e-cbb777d3dfbe,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': 'aea45c70-7c05-4146-847e-cbb777d3dfbe', 'volume_name': 'volume-aea45c70-7c05-4146-847e-cbb777d3dfbe', 'volume_size': 1}} Apr 17 15:51:36 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': 'aea45c70-7c05-4146-847e-cbb777d3dfbe', 'volume': {'id': 'aea45c70-7c05-4146-847e-cbb777d3dfbe', '_name_id': None, 'ec2_id': None, 'user_id': 'd5299006144f4ee6817349348a489017', 'project_id': '75babbbca67f4035a48ee8ee816b18db', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-CreateVolumesFromImageTest-Volume-800604710', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'admin_metadata': {}, 'glance_metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T15:44:38.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-aea45c70-7c05-4146-847e-cbb777d3dfbe', 'name_id': 'aea45c70-7c05-4146-847e-cbb777d3dfbe', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-CreateVolumesFromImageTest-Volume-800604710', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '75babbbca67f4035a48ee8ee816b18db', 'qos_specs': None, 'reservations': ['0a40a0f0-8514-4f1f-ac7e-cbc2d47cbe5f', 'b703b541-7376-4c99-8896-74ebf0e7fb9f', '4c0d25ea-51e7-41c5-8f64-d45e42a842b1', '32031e87-40eb-4f2d-aa17-2ea553f6a406'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'd5299006144f4ee6817349348a489017', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-CreateVolumesFromImageTest-Volume-800604710', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '75babbbca67f4035a48ee8ee816b18db', 'qos_specs': None, 'reservations': ['0a40a0f0-8514-4f1f-ac7e-cbc2d47cbe5f', 'b703b541-7376-4c99-8896-74ebf0e7fb9f', '4c0d25ea-51e7-41c5-8f64-d45e42a842b1', '32031e87-40eb-4f2d-aa17-2ea553f6a406'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'd5299006144f4ee6817349348a489017', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': 'd5299006144f4ee6817349348a489017', 'metadata': {}, 'qos_specs': None, 'retry': {'num_attempts': 3, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']}}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(aea45c70-7c05-4146-847e-cbb777d3dfbe),volume_id=aea45c70-7c05-4146-847e-cbb777d3dfbe,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:44:39Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-CreateVolumesFromImageTest-Volume-800604710',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=aea45c70-7c05-4146-847e-cbb777d3dfbe,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='75babbbca67f4035a48ee8ee816b18db',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:49:17Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:49:17Z,use_quota=True,user_id='d5299006144f4ee6817349348a489017',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 15:51:36 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:44:39Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-CreateVolumesFromImageTest-Volume-800604710',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=aea45c70-7c05-4146-847e-cbb777d3dfbe,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='75babbbca67f4035a48ee8ee816b18db',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:49:17Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:49:17Z,use_quota=True,user_id='d5299006144f4ee6817349348a489017',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:44:39Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-CreateVolumesFromImageTest-Volume-800604710',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=aea45c70-7c05-4146-847e-cbb777d3dfbe,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='75babbbca67f4035a48ee8ee816b18db',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:49:17Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:49:17Z,use_quota=True,user_id='d5299006144f4ee6817349348a489017',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 15:51:36 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 15:51:36 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 15:51:36 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:51:36 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 15:51:36 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 15:51:36 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 15:51:36 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 15:51:36 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 15:51:36 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 15:51:36 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 15:51:36 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 15:51:36 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 15:51:36 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 15:51:36 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 15:51:36 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 15:51:36 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 15:51:36 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 15:51:36 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 15:51:36 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 15:51:36 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 15:51:36 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (4681bef3-39fb-4da6-8f26-0a7615c4b427) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:51:36 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (4681bef3-39fb-4da6-8f26-0a7615c4b427) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:51:36 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (cbe2b0c6-a93e-4dab-a0a2-6ae209ccea00) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:51:36 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (cbe2b0c6-a93e-4dab-a0a2-6ae209ccea00) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:51:36 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (8b8e51f7-7191-4940-b14c-7b43492c1509) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:51:36 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (8b8e51f7-7191-4940-b14c-7b43492c1509) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:51:36 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (86606587-6131-4612-b937-e2c0d95449ef) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:51:36 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Updating volume aea45c70-7c05-4146-847e-cbb777d3dfbe with {'scheduled_at': datetime.datetime(2023, 4, 17, 15, 51, 36, 716737), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 15:51:36 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (56e4cca7-6494-499c-96e7-627fee9eb451) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 15:51:36 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 15:51:36 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:44:39Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-CreateVolumesFromImageTest-Volume-2146530532',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=fbbadf16-651a-47d5-b308-a37e56aa07f2,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='75babbbca67f4035a48ee8ee816b18db',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:49:17Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:49:17Z,use_quota=True,user_id='d5299006144f4ee6817349348a489017',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 15:51:36 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:44:39Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-CreateVolumesFromImageTest-Volume-2146530532',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=fbbadf16-651a-47d5-b308-a37e56aa07f2,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='75babbbca67f4035a48ee8ee816b18db',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:49:17Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:49:17Z,use_quota=True,user_id='d5299006144f4ee6817349348a489017',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(fbbadf16-651a-47d5-b308-a37e56aa07f2),volume_id=fbbadf16-651a-47d5-b308-a37e56aa07f2,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': 'fbbadf16-651a-47d5-b308-a37e56aa07f2', 'volume_name': 'volume-fbbadf16-651a-47d5-b308-a37e56aa07f2', 'volume_size': 1}} Apr 17 15:51:36 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': 'fbbadf16-651a-47d5-b308-a37e56aa07f2', 'volume': {'id': 'fbbadf16-651a-47d5-b308-a37e56aa07f2', '_name_id': None, 'ec2_id': None, 'user_id': 'd5299006144f4ee6817349348a489017', 'project_id': '75babbbca67f4035a48ee8ee816b18db', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-CreateVolumesFromImageTest-Volume-2146530532', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'admin_metadata': {}, 'glance_metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T15:44:38.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-fbbadf16-651a-47d5-b308-a37e56aa07f2', 'name_id': 'fbbadf16-651a-47d5-b308-a37e56aa07f2', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-CreateVolumesFromImageTest-Volume-2146530532', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '75babbbca67f4035a48ee8ee816b18db', 'qos_specs': None, 'reservations': ['ebe2c0e3-e7c3-4d72-b0ff-d7918e651f9b', '11cb75cb-f9b7-46b2-bc19-f76f314b9643', '9657a0be-fa65-4315-b170-50972324b109', 'b1d14f83-c9a6-448d-acae-a1bf415819e5'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'd5299006144f4ee6817349348a489017', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-CreateVolumesFromImageTest-Volume-2146530532', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '75babbbca67f4035a48ee8ee816b18db', 'qos_specs': None, 'reservations': ['ebe2c0e3-e7c3-4d72-b0ff-d7918e651f9b', '11cb75cb-f9b7-46b2-bc19-f76f314b9643', '9657a0be-fa65-4315-b170-50972324b109', 'b1d14f83-c9a6-448d-acae-a1bf415819e5'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'd5299006144f4ee6817349348a489017', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': 'd5299006144f4ee6817349348a489017', 'metadata': {}, 'qos_specs': None, 'retry': {'num_attempts': 3, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']}}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(fbbadf16-651a-47d5-b308-a37e56aa07f2),volume_id=fbbadf16-651a-47d5-b308-a37e56aa07f2,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:44:39Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-CreateVolumesFromImageTest-Volume-2146530532',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=fbbadf16-651a-47d5-b308-a37e56aa07f2,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='75babbbca67f4035a48ee8ee816b18db',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:49:17Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:49:17Z,use_quota=True,user_id='d5299006144f4ee6817349348a489017',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 15:51:36 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:44:39Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-CreateVolumesFromImageTest-Volume-2146530532',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=fbbadf16-651a-47d5-b308-a37e56aa07f2,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='75babbbca67f4035a48ee8ee816b18db',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:49:17Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:49:17Z,use_quota=True,user_id='d5299006144f4ee6817349348a489017',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:44:39Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-CreateVolumesFromImageTest-Volume-2146530532',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=fbbadf16-651a-47d5-b308-a37e56aa07f2,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='75babbbca67f4035a48ee8ee816b18db',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:49:17Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:49:17Z,use_quota=True,user_id='d5299006144f4ee6817349348a489017',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 15:51:36 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 15:51:36 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 15:51:36 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:51:36 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 15:51:36 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 15:51:36 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 15:51:36 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 15:51:36 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 15:51:36 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 15:51:36 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 15:51:36 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 15:51:36 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 15:51:36 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 15:51:36 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 15:51:36 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 15:51:36 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 15:51:36 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 15:51:36 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 15:51:36 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 15:51:36 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 15:51:36 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (56e4cca7-6494-499c-96e7-627fee9eb451) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:51:36 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (56e4cca7-6494-499c-96e7-627fee9eb451) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:51:36 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (b9897d71-e285-4657-a1e8-85f152afdcf0) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:51:36 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (b9897d71-e285-4657-a1e8-85f152afdcf0) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:51:36 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (5fd6737e-92c8-44b2-9cf9-05fd83bdd49d) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:51:36 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (5fd6737e-92c8-44b2-9cf9-05fd83bdd49d) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:51:36 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (0a72d5cb-ba2d-4dde-a6c4-034bae01c38e) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:51:36 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Updating volume fbbadf16-651a-47d5-b308-a37e56aa07f2 with {'scheduled_at': datetime.datetime(2023, 4, 17, 15, 51, 36, 769136), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 15:51:36 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Volume aea45c70-7c05-4146-847e-cbb777d3dfbe: re-scheduling SchedulerAPI.create_volume attempt 3 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 15:51:36 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Volume dfd745ed-b0c4-4e35-b77c-a911c90ce33e: re-scheduling SchedulerAPI.create_volume attempt 3 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 15:51:36 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Volume fbbadf16-651a-47d5-b308-a37e56aa07f2: re-scheduling SchedulerAPI.create_volume attempt 3 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 15:51:36 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Volume aea45c70-7c05-4146-847e-cbb777d3dfbe: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 15:51:36 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] deleating volume v_aea45c70-7c05-4146-847e-cbb777d3dfbe {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 15:51:36 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] get snapshots for volume v_aea45c70-7c05-4146-847e-cbb777d3dfbe {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 15:51:36 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_aea45c70-7c05-4146-847e-cbb777d3dfbe/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:51:36 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_aea45c70-7c05-4146-847e-cbb777d3dfbe/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:51:36 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Volume dfd745ed-b0c4-4e35-b77c-a911c90ce33e: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 15:51:36 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] deleating volume v_dfd745ed-b0c4-4e35-b77c-a911c90ce33e {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 15:51:36 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] get snapshots for volume v_dfd745ed-b0c4-4e35-b77c-a911c90ce33e {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 15:51:36 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_dfd745ed-b0c4-4e35-b77c-a911c90ce33e/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:51:36 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_dfd745ed-b0c4-4e35-b77c-a911c90ce33e/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:51:36 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Volume fbbadf16-651a-47d5-b308-a37e56aa07f2: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 15:51:36 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] deleating volume v_fbbadf16-651a-47d5-b308-a37e56aa07f2 {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 15:51:36 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] get snapshots for volume v_fbbadf16-651a-47d5-b308-a37e56aa07f2 {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 15:51:36 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_fbbadf16-651a-47d5-b308-a37e56aa07f2/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:51:36 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_fbbadf16-651a-47d5-b308-a37e56aa07f2/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:51:39 user cinder-volume[84696]: DEBUG cinder.utils [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 74.464(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:51:39 user cinder-volume[84696]: DEBUG cinder.utils [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:51:39 user cinder-volume[84696]: DEBUG cinder.utils [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.788(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:51:39 user cinder-volume[84696]: DEBUG cinder.utils [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_fbbadf16-651a-47d5-b308-a37e56aa07f2/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:51:39 user cinder-volume[84696]: DEBUG cinder.utils [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.815(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:51:39 user cinder-volume[84696]: DEBUG cinder.utils [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_aea45c70-7c05-4146-847e-cbb777d3dfbe/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:51:39 user cinder-volume[84696]: DEBUG cinder.utils [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 139.994(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:51:39 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_ab5889f8-b6c7-4673-a666-f4b4e5ff5fd5/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:51:39 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (ed61cc6c-4259-46de-8e73-3b5f80096571) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 15:51:39 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (f5a58638-0fab-4fbe-83df-d20129322641) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:51:39 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (f5a58638-0fab-4fbe-83df-d20129322641) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:51:39 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Flow 'volume_create_manager' (36004948-f6dd-4fbc-80e8-70e49699879b) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 15:51:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 15:51:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 15:51:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 15:51:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 15:51:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 15:51:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 15:51:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 15:51:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 15:51:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 15:51:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 15:51:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 15:51:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 15:51:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 15:51:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 15:51:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 15:51:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 15:51:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 15:51:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 15:51:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 15:51:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 15:51:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 15:51:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 15:51:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 15:51:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 15:51:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:51:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 15:51:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 15:51:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 15:51:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 15:51:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 15:51:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 15:51:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 15:51:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 15:51:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 15:51:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 15:51:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 15:51:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 15:51:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 15:51:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 15:51:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 15:51:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 15:51:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 15:51:39 user cinder-volume[84696]: DEBUG cinder.utils [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.816(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:51:39 user cinder-volume[84696]: DEBUG cinder.utils [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_dfd745ed-b0c4-4e35-b77c-a911c90ce33e/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:51:39 user cinder-volume[84696]: DEBUG cinder.utils [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 74.841(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:51:39 user cinder-volume[84696]: DEBUG cinder.utils [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_93a32a7c-4c6a-472a-8f91-9d34558ba232/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:51:39 user cinder-volume[84696]: DEBUG cinder.utils [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 140.024(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:51:39 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_13cc73df-7780-4934-8cbd-7f372b1cf57a/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:51:39 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (f9ed0951-88b0-4fb9-aaab-6be33f2e9366) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 15:51:39 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (b14fdc3c-2a4d-4b8c-bde6-41ea57565678) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:51:39 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (b14fdc3c-2a4d-4b8c-bde6-41ea57565678) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:51:39 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Flow 'volume_create_manager' (93b410ca-1864-4f6c-9a57-32151032e466) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 15:51:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 15:51:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 15:51:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 15:51:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 15:51:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 15:51:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 15:51:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 15:51:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 15:51:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 15:51:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 15:51:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 15:51:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 15:51:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 15:51:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 15:51:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 15:51:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 15:51:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 15:51:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 15:51:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 15:51:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 15:51:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 15:51:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 15:51:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 15:51:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 15:51:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:51:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 15:51:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 15:51:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 15:51:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 15:51:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 15:51:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 15:51:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 15:51:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 15:51:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 15:51:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 15:51:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 15:51:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 15:51:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 15:51:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 15:51:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 15:51:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 15:51:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 15:51:39 user cinder-volume[84696]: DEBUG cinder.utils [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.938(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:51:39 user cinder-volume[84696]: DEBUG cinder.utils [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:51:39 user cinder-volume[84696]: DEBUG cinder.utils [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.103(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:51:39 user cinder-volume[84696]: DEBUG cinder.utils [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_4a2d1c00-a134-4f73-8508-86f35b58acf7/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:51:39 user cinder-volume[84696]: DEBUG cinder.utils [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 139.651(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:51:39 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:51:39 user cinder-volume[84696]: INFO cinder.message.api [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Creating message record for request_id = req-9b871d6d-58e3-4523-b328-b6ffa7aca14e Apr 17 15:51:39 user cinder-volume[84696]: WARNING urllib3.connectionpool [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Connection pool is full, discarding connection: 172.16.0.220. Connection pool size: 10: queue.Full Apr 17 15:51:39 user cinder-volume[84696]: DEBUG cinder.utils [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 139.727(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:51:39 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:51:39 user cinder-volume[84696]: INFO cinder.message.api [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Creating message record for request_id = req-27d7b7b3-4675-424c-af48-cc2422ffc813 Apr 17 15:51:39 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 15:51:39 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 15:51:39 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (671165ad-ebcd-4131-93e4-5e514a6552b3) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 15:51:39 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 15:51:39 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:44:39Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-CreateVolumesFromImageTest-Volume-1478171265',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=ab5889f8-b6c7-4673-a666-f4b4e5ff5fd5,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='75babbbca67f4035a48ee8ee816b18db',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:49:20Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:49:20Z,use_quota=True,user_id='d5299006144f4ee6817349348a489017',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 15:51:39 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:44:39Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-CreateVolumesFromImageTest-Volume-1478171265',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=ab5889f8-b6c7-4673-a666-f4b4e5ff5fd5,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='75babbbca67f4035a48ee8ee816b18db',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:49:20Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:49:20Z,use_quota=True,user_id='d5299006144f4ee6817349348a489017',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(ab5889f8-b6c7-4673-a666-f4b4e5ff5fd5),volume_id=ab5889f8-b6c7-4673-a666-f4b4e5ff5fd5,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': 'ab5889f8-b6c7-4673-a666-f4b4e5ff5fd5', 'volume_name': 'volume-ab5889f8-b6c7-4673-a666-f4b4e5ff5fd5', 'volume_size': 1}} Apr 17 15:51:39 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': 'ab5889f8-b6c7-4673-a666-f4b4e5ff5fd5', 'volume': {'id': 'ab5889f8-b6c7-4673-a666-f4b4e5ff5fd5', '_name_id': None, 'ec2_id': None, 'user_id': 'd5299006144f4ee6817349348a489017', 'project_id': '75babbbca67f4035a48ee8ee816b18db', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-CreateVolumesFromImageTest-Volume-1478171265', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'admin_metadata': {}, 'glance_metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T15:44:39.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-ab5889f8-b6c7-4673-a666-f4b4e5ff5fd5', 'name_id': 'ab5889f8-b6c7-4673-a666-f4b4e5ff5fd5', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-CreateVolumesFromImageTest-Volume-1478171265', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '75babbbca67f4035a48ee8ee816b18db', 'qos_specs': None, 'reservations': ['a64639ba-df1a-40e6-bded-21cad01b745f', '4323d77e-0f64-483e-8433-0f03b9e38ddd', '7b2c692c-9fb9-4f7b-9b9c-94383941c4df', '45cf8435-1a25-41db-aac5-ff8bf8698729'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'd5299006144f4ee6817349348a489017', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-CreateVolumesFromImageTest-Volume-1478171265', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '75babbbca67f4035a48ee8ee816b18db', 'qos_specs': None, 'reservations': ['a64639ba-df1a-40e6-bded-21cad01b745f', '4323d77e-0f64-483e-8433-0f03b9e38ddd', '7b2c692c-9fb9-4f7b-9b9c-94383941c4df', '45cf8435-1a25-41db-aac5-ff8bf8698729'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'd5299006144f4ee6817349348a489017', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': 'd5299006144f4ee6817349348a489017', 'metadata': {}, 'qos_specs': None, 'retry': {'num_attempts': 3, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']}}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(ab5889f8-b6c7-4673-a666-f4b4e5ff5fd5),volume_id=ab5889f8-b6c7-4673-a666-f4b4e5ff5fd5,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:44:39Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-CreateVolumesFromImageTest-Volume-1478171265',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=ab5889f8-b6c7-4673-a666-f4b4e5ff5fd5,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='75babbbca67f4035a48ee8ee816b18db',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:49:20Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:49:20Z,use_quota=True,user_id='d5299006144f4ee6817349348a489017',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 15:51:39 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:44:39Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-CreateVolumesFromImageTest-Volume-1478171265',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=ab5889f8-b6c7-4673-a666-f4b4e5ff5fd5,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='75babbbca67f4035a48ee8ee816b18db',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:49:20Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:49:20Z,use_quota=True,user_id='d5299006144f4ee6817349348a489017',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:44:39Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-CreateVolumesFromImageTest-Volume-1478171265',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=ab5889f8-b6c7-4673-a666-f4b4e5ff5fd5,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='75babbbca67f4035a48ee8ee816b18db',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:49:20Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:49:20Z,use_quota=True,user_id='d5299006144f4ee6817349348a489017',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 15:51:39 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 15:51:39 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 15:51:39 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:51:39 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 15:51:39 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 15:51:39 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 15:51:39 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 15:51:39 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 15:51:39 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 15:51:39 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 15:51:39 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 15:51:39 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 15:51:39 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 15:51:39 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 15:51:39 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 15:51:39 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 15:51:39 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 15:51:39 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 15:51:39 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 15:51:39 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 15:51:39 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (671165ad-ebcd-4131-93e4-5e514a6552b3) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:51:39 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (671165ad-ebcd-4131-93e4-5e514a6552b3) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:51:39 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (25c63d92-de74-424e-a068-d6dca2250db4) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:51:39 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (25c63d92-de74-424e-a068-d6dca2250db4) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:51:39 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (ffd36bbb-4c10-4144-b23b-1a77d8c93778) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:51:39 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (ffd36bbb-4c10-4144-b23b-1a77d8c93778) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:51:39 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (d63d7778-b87e-4d8f-b05d-c0bc4e4f2689) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:51:39 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Updating volume ab5889f8-b6c7-4673-a666-f4b4e5ff5fd5 with {'scheduled_at': datetime.datetime(2023, 4, 17, 15, 51, 39, 718524), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 15:51:39 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Volume ab5889f8-b6c7-4673-a666-f4b4e5ff5fd5: re-scheduling SchedulerAPI.create_volume attempt 3 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 15:51:39 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (36177b7f-40f9-4c25-9f43-6e183241c50c) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 15:51:39 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 15:51:39 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:44:39Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-CreateVolumesFromImageTest-Volume-1776307206',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=13cc73df-7780-4934-8cbd-7f372b1cf57a,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='75babbbca67f4035a48ee8ee816b18db',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:49:20Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:49:20Z,use_quota=True,user_id='d5299006144f4ee6817349348a489017',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 15:51:39 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:44:39Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-CreateVolumesFromImageTest-Volume-1776307206',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=13cc73df-7780-4934-8cbd-7f372b1cf57a,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='75babbbca67f4035a48ee8ee816b18db',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:49:20Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:49:20Z,use_quota=True,user_id='d5299006144f4ee6817349348a489017',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(13cc73df-7780-4934-8cbd-7f372b1cf57a),volume_id=13cc73df-7780-4934-8cbd-7f372b1cf57a,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': '13cc73df-7780-4934-8cbd-7f372b1cf57a', 'volume_name': 'volume-13cc73df-7780-4934-8cbd-7f372b1cf57a', 'volume_size': 1}} Apr 17 15:51:39 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': '13cc73df-7780-4934-8cbd-7f372b1cf57a', 'volume': {'id': '13cc73df-7780-4934-8cbd-7f372b1cf57a', '_name_id': None, 'ec2_id': None, 'user_id': 'd5299006144f4ee6817349348a489017', 'project_id': '75babbbca67f4035a48ee8ee816b18db', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-CreateVolumesFromImageTest-Volume-1776307206', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'admin_metadata': {}, 'glance_metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T15:44:39.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-13cc73df-7780-4934-8cbd-7f372b1cf57a', 'name_id': '13cc73df-7780-4934-8cbd-7f372b1cf57a', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-CreateVolumesFromImageTest-Volume-1776307206', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '75babbbca67f4035a48ee8ee816b18db', 'qos_specs': None, 'reservations': ['a8c0981a-3509-44ee-887e-5a4c15f5fc31', 'be48fc25-1b54-4ebf-aef6-6988c7432c47', '45f69ba8-0c40-4500-b132-878446677716', '5143bd10-b0c0-4336-a02c-d9f5b822c37b'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'd5299006144f4ee6817349348a489017', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-CreateVolumesFromImageTest-Volume-1776307206', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '75babbbca67f4035a48ee8ee816b18db', 'qos_specs': None, 'reservations': ['a8c0981a-3509-44ee-887e-5a4c15f5fc31', 'be48fc25-1b54-4ebf-aef6-6988c7432c47', '45f69ba8-0c40-4500-b132-878446677716', '5143bd10-b0c0-4336-a02c-d9f5b822c37b'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'd5299006144f4ee6817349348a489017', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': 'd5299006144f4ee6817349348a489017', 'metadata': {}, 'qos_specs': None, 'retry': {'num_attempts': 3, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']}}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(13cc73df-7780-4934-8cbd-7f372b1cf57a),volume_id=13cc73df-7780-4934-8cbd-7f372b1cf57a,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:44:39Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-CreateVolumesFromImageTest-Volume-1776307206',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=13cc73df-7780-4934-8cbd-7f372b1cf57a,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='75babbbca67f4035a48ee8ee816b18db',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:49:20Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:49:20Z,use_quota=True,user_id='d5299006144f4ee6817349348a489017',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 15:51:39 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:44:39Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-CreateVolumesFromImageTest-Volume-1776307206',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=13cc73df-7780-4934-8cbd-7f372b1cf57a,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='75babbbca67f4035a48ee8ee816b18db',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:49:20Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:49:20Z,use_quota=True,user_id='d5299006144f4ee6817349348a489017',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:44:39Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-CreateVolumesFromImageTest-Volume-1776307206',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=13cc73df-7780-4934-8cbd-7f372b1cf57a,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='75babbbca67f4035a48ee8ee816b18db',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:49:20Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:49:20Z,use_quota=True,user_id='d5299006144f4ee6817349348a489017',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 15:51:39 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 15:51:39 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 15:51:39 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:51:39 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 15:51:39 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 15:51:39 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 15:51:39 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 15:51:39 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 15:51:39 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 15:51:39 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 15:51:39 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 15:51:39 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 15:51:39 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 15:51:39 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 15:51:39 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 15:51:39 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 15:51:39 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 15:51:39 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 15:51:39 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 15:51:39 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 15:51:39 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (36177b7f-40f9-4c25-9f43-6e183241c50c) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:51:39 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (36177b7f-40f9-4c25-9f43-6e183241c50c) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:51:39 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (2d570206-ca36-4c07-9215-efee2f5bb1ae) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:51:39 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (2d570206-ca36-4c07-9215-efee2f5bb1ae) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:51:39 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (f5b299f1-d02a-4c3e-93b7-2ca59c3e291e) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:51:39 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (f5b299f1-d02a-4c3e-93b7-2ca59c3e291e) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:51:39 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (f20c37de-84d6-484b-8000-206b81409724) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:51:39 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Updating volume 13cc73df-7780-4934-8cbd-7f372b1cf57a with {'scheduled_at': datetime.datetime(2023, 4, 17, 15, 51, 39, 782272), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 15:51:39 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Volume 13cc73df-7780-4934-8cbd-7f372b1cf57a: re-scheduling SchedulerAPI.create_volume attempt 3 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 15:51:39 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Volume ab5889f8-b6c7-4673-a666-f4b4e5ff5fd5: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 15:51:39 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] deleating volume v_ab5889f8-b6c7-4673-a666-f4b4e5ff5fd5 {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 15:51:39 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] get snapshots for volume v_ab5889f8-b6c7-4673-a666-f4b4e5ff5fd5 {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 15:51:39 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_ab5889f8-b6c7-4673-a666-f4b4e5ff5fd5/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:51:39 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_ab5889f8-b6c7-4673-a666-f4b4e5ff5fd5/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:51:39 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Volume 13cc73df-7780-4934-8cbd-7f372b1cf57a: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 15:51:39 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] deleating volume v_13cc73df-7780-4934-8cbd-7f372b1cf57a {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 15:51:39 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] get snapshots for volume v_13cc73df-7780-4934-8cbd-7f372b1cf57a {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 15:51:39 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_13cc73df-7780-4934-8cbd-7f372b1cf57a/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:51:39 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_13cc73df-7780-4934-8cbd-7f372b1cf57a/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:51:42 user cinder-volume[84696]: DEBUG cinder.utils [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.850(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:51:42 user cinder-volume[84696]: DEBUG cinder.utils [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_ab5889f8-b6c7-4673-a666-f4b4e5ff5fd5/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:51:42 user cinder-volume[84696]: DEBUG cinder.utils [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.839(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:51:42 user cinder-volume[84696]: DEBUG cinder.utils [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_13cc73df-7780-4934-8cbd-7f372b1cf57a/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:51:42 user cinder-volume[84696]: DEBUG cinder.utils [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 140.009(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:51:42 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_68f814b0-1291-4a12-ae65-dc0892203d36/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:51:42 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (48c4ff00-053c-4a8b-8d01-bf8c90352d82) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 15:51:42 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (762d4ee0-8819-457a-b6c8-bad3a12c96c6) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:51:42 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (762d4ee0-8819-457a-b6c8-bad3a12c96c6) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:51:42 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Flow 'volume_create_manager' (ddf23f63-d343-4c4e-ab15-b09de6fa5ae4) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 15:51:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 15:51:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 15:51:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 15:51:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 15:51:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 15:51:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 15:51:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 15:51:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 15:51:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 15:51:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 15:51:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 15:51:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 15:51:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 15:51:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 15:51:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 15:51:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 15:51:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 15:51:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 15:51:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 15:51:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 15:51:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 15:51:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 15:51:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 15:51:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 15:51:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:51:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 15:51:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 15:51:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 15:51:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 15:51:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 15:51:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 15:51:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 15:51:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 15:51:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 15:51:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 15:51:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 15:51:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 15:51:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 15:51:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 15:51:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 15:51:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 15:51:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 15:51:42 user cinder-volume[84696]: DEBUG cinder.utils [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 139.846(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:51:42 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:51:42 user cinder-volume[84696]: INFO cinder.message.api [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Creating message record for request_id = req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 Apr 17 15:51:42 user cinder-volume[84696]: DEBUG cinder.utils [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.911(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:51:42 user cinder-volume[84696]: DEBUG cinder.utils [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_aea45c70-7c05-4146-847e-cbb777d3dfbe/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:51:42 user cinder-volume[84696]: DEBUG cinder.utils [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.887(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:51:42 user cinder-volume[84696]: DEBUG cinder.utils [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_fbbadf16-651a-47d5-b308-a37e56aa07f2/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:51:42 user cinder-volume[84696]: DEBUG cinder.utils [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.896(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:51:42 user cinder-volume[84696]: DEBUG cinder.utils [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_dfd745ed-b0c4-4e35-b77c-a911c90ce33e/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:51:42 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 15:51:42 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (dbc74882-420b-4d9e-9ff9-82fe72629ce4) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 15:51:42 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 15:51:42 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:44:42Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-ServerRescueNegativeTestJSON-volume-1878278176',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=68f814b0-1291-4a12-ae65-dc0892203d36,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='9324c99bd4d040bdb5a7f1db782b2783',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:49:23Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:49:23Z,use_quota=True,user_id='8e3487d6b32d46beac556617050fd664',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 15:51:42 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:44:42Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-ServerRescueNegativeTestJSON-volume-1878278176',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=68f814b0-1291-4a12-ae65-dc0892203d36,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='9324c99bd4d040bdb5a7f1db782b2783',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:49:23Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:49:23Z,use_quota=True,user_id='8e3487d6b32d46beac556617050fd664',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(68f814b0-1291-4a12-ae65-dc0892203d36),volume_id=68f814b0-1291-4a12-ae65-dc0892203d36,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': '68f814b0-1291-4a12-ae65-dc0892203d36', 'volume_name': 'volume-68f814b0-1291-4a12-ae65-dc0892203d36', 'volume_size': 1}} Apr 17 15:51:42 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': '68f814b0-1291-4a12-ae65-dc0892203d36', 'volume': {'id': '68f814b0-1291-4a12-ae65-dc0892203d36', '_name_id': None, 'ec2_id': None, 'user_id': '8e3487d6b32d46beac556617050fd664', 'project_id': '9324c99bd4d040bdb5a7f1db782b2783', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-ServerRescueNegativeTestJSON-volume-1878278176', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'admin_metadata': {}, 'glance_metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T15:44:42.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-68f814b0-1291-4a12-ae65-dc0892203d36', 'name_id': '68f814b0-1291-4a12-ae65-dc0892203d36', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-ServerRescueNegativeTestJSON-volume-1878278176', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '9324c99bd4d040bdb5a7f1db782b2783', 'qos_specs': None, 'reservations': ['2f994b27-d987-48a4-ad61-f3ddbb20a152', 'b47baa6b-4689-4254-948c-f073028527b3', '59e249a4-01b6-44aa-a2e4-029e54867201', '148031dd-8a91-4041-9ef3-af4b5a029444'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': '8e3487d6b32d46beac556617050fd664', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-ServerRescueNegativeTestJSON-volume-1878278176', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '9324c99bd4d040bdb5a7f1db782b2783', 'qos_specs': None, 'reservations': ['2f994b27-d987-48a4-ad61-f3ddbb20a152', 'b47baa6b-4689-4254-948c-f073028527b3', '59e249a4-01b6-44aa-a2e4-029e54867201', '148031dd-8a91-4041-9ef3-af4b5a029444'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': '8e3487d6b32d46beac556617050fd664', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': '8e3487d6b32d46beac556617050fd664', 'metadata': {}, 'qos_specs': None, 'retry': {'num_attempts': 3, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']}}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(68f814b0-1291-4a12-ae65-dc0892203d36),volume_id=68f814b0-1291-4a12-ae65-dc0892203d36,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:44:42Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-ServerRescueNegativeTestJSON-volume-1878278176',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=68f814b0-1291-4a12-ae65-dc0892203d36,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='9324c99bd4d040bdb5a7f1db782b2783',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:49:23Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:49:23Z,use_quota=True,user_id='8e3487d6b32d46beac556617050fd664',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 15:51:42 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:44:42Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-ServerRescueNegativeTestJSON-volume-1878278176',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=68f814b0-1291-4a12-ae65-dc0892203d36,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='9324c99bd4d040bdb5a7f1db782b2783',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:49:23Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:49:23Z,use_quota=True,user_id='8e3487d6b32d46beac556617050fd664',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:44:42Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-ServerRescueNegativeTestJSON-volume-1878278176',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=68f814b0-1291-4a12-ae65-dc0892203d36,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='9324c99bd4d040bdb5a7f1db782b2783',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:49:23Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:49:23Z,use_quota=True,user_id='8e3487d6b32d46beac556617050fd664',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 15:51:42 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 15:51:42 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 15:51:42 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:51:42 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 15:51:42 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 15:51:42 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 15:51:42 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 15:51:42 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 15:51:42 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 15:51:42 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 15:51:42 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 15:51:42 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 15:51:42 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 15:51:42 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 15:51:42 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 15:51:42 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 15:51:42 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 15:51:42 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 15:51:42 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 15:51:42 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 15:51:42 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (dbc74882-420b-4d9e-9ff9-82fe72629ce4) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:51:42 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (dbc74882-420b-4d9e-9ff9-82fe72629ce4) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:51:42 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (f9cd7bb4-0e3a-4851-b457-4e61c3a0f1e4) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:51:42 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (f9cd7bb4-0e3a-4851-b457-4e61c3a0f1e4) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:51:42 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (2d709152-3d0a-456c-baa9-43704a23d108) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:51:42 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (2d709152-3d0a-456c-baa9-43704a23d108) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:51:42 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (d8132ee2-3794-49bc-8f00-ba6d46ed7cac) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:51:42 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Updating volume 68f814b0-1291-4a12-ae65-dc0892203d36 with {'scheduled_at': datetime.datetime(2023, 4, 17, 15, 51, 42, 769154), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 15:51:42 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Volume 68f814b0-1291-4a12-ae65-dc0892203d36: re-scheduling SchedulerAPI.create_volume attempt 3 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 15:51:42 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Volume 68f814b0-1291-4a12-ae65-dc0892203d36: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 15:51:42 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] deleating volume v_68f814b0-1291-4a12-ae65-dc0892203d36 {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 15:51:42 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] get snapshots for volume v_68f814b0-1291-4a12-ae65-dc0892203d36 {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 15:51:42 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_68f814b0-1291-4a12-ae65-dc0892203d36/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:51:42 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_68f814b0-1291-4a12-ae65-dc0892203d36/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:51:45 user cinder-volume[84696]: DEBUG cinder.utils [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.042(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:51:45 user cinder-volume[84696]: DEBUG cinder.utils [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:51:45 user cinder-volume[84696]: DEBUG cinder.utils [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.207(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:51:45 user cinder-volume[84696]: DEBUG cinder.utils [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_4a2d1c00-a134-4f73-8508-86f35b58acf7/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:51:45 user cinder-volume[84696]: DEBUG cinder.utils [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.964(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:51:45 user cinder-volume[84696]: DEBUG cinder.utils [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_68f814b0-1291-4a12-ae65-dc0892203d36/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:51:45 user cinder-volume[84696]: DEBUG cinder.utils [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.912(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:51:45 user cinder-volume[84696]: DEBUG cinder.utils [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_13cc73df-7780-4934-8cbd-7f372b1cf57a/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:51:45 user cinder-volume[84696]: DEBUG cinder.utils [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.927(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:51:45 user cinder-volume[84696]: DEBUG cinder.utils [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_ab5889f8-b6c7-4673-a666-f4b4e5ff5fd5/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:51:48 user cinder-volume[84696]: DEBUG cinder.utils [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.028(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:51:48 user cinder-volume[84696]: DEBUG cinder.utils [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_aea45c70-7c05-4146-847e-cbb777d3dfbe/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:51:48 user cinder-volume[84696]: DEBUG cinder.utils [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.003(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:51:48 user cinder-volume[84696]: DEBUG cinder.utils [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_fbbadf16-651a-47d5-b308-a37e56aa07f2/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:51:48 user cinder-volume[84696]: DEBUG cinder.utils [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.036(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:51:48 user cinder-volume[84696]: DEBUG cinder.utils [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_68f814b0-1291-4a12-ae65-dc0892203d36/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:51:48 user cinder-volume[84696]: DEBUG cinder.utils [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.013(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:51:48 user cinder-volume[84696]: DEBUG cinder.utils [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_dfd745ed-b0c4-4e35-b77c-a911c90ce33e/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:51:51 user cinder-volume[84696]: DEBUG cinder.utils [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 74.721(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:51:51 user cinder-volume[84696]: DEBUG cinder.utils [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_5c16bb82-43c5-4d91-9320-0e1f2bbec574/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:51:51 user cinder-volume[84696]: DEBUG cinder.utils [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.053(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:51:51 user cinder-volume[84696]: DEBUG cinder.utils [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_13cc73df-7780-4934-8cbd-7f372b1cf57a/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:51:51 user cinder-volume[84696]: DEBUG cinder.utils [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.069(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:51:51 user cinder-volume[84696]: DEBUG cinder.utils [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_ab5889f8-b6c7-4673-a666-f4b4e5ff5fd5/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:51:51 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 137.116(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:51:51 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:51:51 user cinder-volume[84696]: ERROR oslo_service.periodic_task [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Error during VolumeManager.publish_service_capabilities: TypeError: 'NoneType' object is not subscriptable Apr 17 15:51:51 user cinder-volume[84696]: ERROR oslo_service.periodic_task Traceback (most recent call last): Apr 17 15:51:51 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/usr/local/lib/python3.10/dist-packages/oslo_service/periodic_task.py", line 216, in run_periodic_tasks Apr 17 15:51:51 user cinder-volume[84696]: ERROR oslo_service.periodic_task task(self, context) Apr 17 15:51:51 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/manager.py", line 2876, in publish_service_capabilities Apr 17 15:51:51 user cinder-volume[84696]: ERROR oslo_service.periodic_task self._report_driver_status(context) Apr 17 15:51:51 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/manager.py", line 2755, in _report_driver_status Apr 17 15:51:51 user cinder-volume[84696]: ERROR oslo_service.periodic_task volume_stats = get_stats() Apr 17 15:51:51 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/usr/local/lib/python3.10/dist-packages/oslo_utils/timeutils.py", line 319, in wrapper Apr 17 15:51:51 user cinder-volume[84696]: ERROR oslo_service.periodic_task result = func(*args, **kwargs) Apr 17 15:51:51 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/manager.py", line 2753, in get_stats Apr 17 15:51:51 user cinder-volume[84696]: ERROR oslo_service.periodic_task return self.driver.get_volume_stats(refresh=True) Apr 17 15:51:51 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/driver.py", line 666, in get_volume_stats Apr 17 15:51:51 user cinder-volume[84696]: ERROR oslo_service.periodic_task self._update_volume_stats() Apr 17 15:51:51 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 726, in _update_volume_stats Apr 17 15:51:51 user cinder-volume[84696]: ERROR oslo_service.periodic_task pool_stats = self.ra.get_pool_stats() Apr 17 15:51:51 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 897, in get_pool_stats Apr 17 15:51:51 user cinder-volume[84696]: ERROR oslo_service.periodic_task if not resp["error"] and resp["code"] == 200: Apr 17 15:51:51 user cinder-volume[84696]: ERROR oslo_service.periodic_task TypeError: 'NoneType' object is not subscriptable Apr 17 15:51:51 user cinder-volume[84696]: ERROR oslo_service.periodic_task Apr 17 15:51:51 user cinder-volume[84696]: WARNING oslo.service.loopingcall [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Function 'cinder.service.Service.periodic_tasks' run outlasted interval by 77.12 sec Apr 17 15:51:51 user cinder-volume[84696]: DEBUG oslo_service.periodic_task [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Running periodic task VolumeManager.publish_service_capabilities {{(pid=84696) run_periodic_tasks /usr/local/lib/python3.10/dist-packages/oslo_service/periodic_task.py:210}} Apr 17 15:51:51 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Updating volume stats {{(pid=84696) _update_volume_stats /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:724}} Apr 17 15:51:51 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Get pool Pool-0 fsprops {{(pid=84696) get_pool_stats /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:894}} Apr 17 15:51:51 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:51:51 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:51:54 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=25,cinder:UPDATE=8,cinder:INSERT=9 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:51:54 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:INSERT=5,cinder:UPDATE=10,cinder:SELECT=17 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:51:54 user cinder-volume[84696]: DEBUG cinder.utils [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.258(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:51:54 user cinder-volume[84696]: DEBUG cinder.utils [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:51:54 user cinder-volume[84696]: DEBUG cinder.utils [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.180(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:51:54 user cinder-volume[84696]: DEBUG cinder.utils [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_68f814b0-1291-4a12-ae65-dc0892203d36/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:51:54 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 3.066(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:51:54 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:51:54 user cinder-volume[84696]: DEBUG cinder.utils [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.427(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:51:54 user cinder-volume[84696]: DEBUG cinder.utils [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_4a2d1c00-a134-4f73-8508-86f35b58acf7/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:51:58 user cinder-volume[84696]: DEBUG cinder.utils [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.227(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:51:58 user cinder-volume[84696]: DEBUG cinder.utils [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_dfd745ed-b0c4-4e35-b77c-a911c90ce33e/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:51:58 user cinder-volume[84696]: DEBUG cinder.utils [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.222(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:51:58 user cinder-volume[84696]: DEBUG cinder.utils [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_fbbadf16-651a-47d5-b308-a37e56aa07f2/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:51:58 user cinder-volume[84696]: DEBUG cinder.utils [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.250(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:51:58 user cinder-volume[84696]: DEBUG cinder.utils [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_aea45c70-7c05-4146-847e-cbb777d3dfbe/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:51:58 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.143(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:51:58 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:52:01 user cinder-volume[84696]: DEBUG cinder.utils [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.282(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:52:01 user cinder-volume[84696]: DEBUG cinder.utils [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_ab5889f8-b6c7-4673-a666-f4b4e5ff5fd5/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:52:01 user cinder-volume[84696]: DEBUG cinder.utils [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.270(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:52:01 user cinder-volume[84696]: DEBUG cinder.utils [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_13cc73df-7780-4934-8cbd-7f372b1cf57a/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:52:04 user cinder-volume[84696]: DEBUG cinder.utils [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.394(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:52:04 user cinder-volume[84696]: DEBUG cinder.utils [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_68f814b0-1291-4a12-ae65-dc0892203d36/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:52:04 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.280(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:52:04 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:52:07 user cinder-volume[84696]: DEBUG cinder.utils [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 71.728(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:52:07 user cinder-volume[84696]: DEBUG cinder.utils [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_2c9714de-f8ea-4d1d-99f9-beb85126ad91/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:52:09 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-778c62be-8963-4e6d-97f1-f456b759ce16 req-00547a1c-f80b-48db-8093-70f53a331fbe tempest-AttachVolumeShelveTestJSON-1745617049 None] terminate connection for 13bf568b-585c-4556-ab78-e9930ad50214 {{(pid=84696) terminate_connection /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:1032}} Apr 17 15:52:09 user cinder-volume[84696]: INFO cinder.volume.manager [req-778c62be-8963-4e6d-97f1-f456b759ce16 req-00547a1c-f80b-48db-8093-70f53a331fbe tempest-AttachVolumeShelveTestJSON-1745617049 None] Terminate volume connection completed successfully. Apr 17 15:52:09 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-778c62be-8963-4e6d-97f1-f456b759ce16 req-00547a1c-f80b-48db-8093-70f53a331fbe tempest-AttachVolumeShelveTestJSON-1745617049 None] Deleting attachment 1634a7d0-3313-448e-9888-8de015f268af. {{(pid=84696) attachment_delete /opt/stack/cinder/cinder/volume/manager.py:5006}} Apr 17 15:52:09 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-778c62be-8963-4e6d-97f1-f456b759ce16 req-00547a1c-f80b-48db-8093-70f53a331fbe tempest-AttachVolumeShelveTestJSON-1745617049 None] remove_export for volume: 13bf568b-585c-4556-ab78-e9930ad50214. {{(pid=84696) remove_export /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:718}} Apr 17 15:52:09 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-778c62be-8963-4e6d-97f1-f456b759ce16 req-00547a1c-f80b-48db-8093-70f53a331fbe tempest-AttachVolumeShelveTestJSON-1745617049 None] remove export {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:912}} Apr 17 15:52:09 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-778c62be-8963-4e6d-97f1-f456b759ce16 req-00547a1c-f80b-48db-8093-70f53a331fbe tempest-AttachVolumeShelveTestJSON-1745617049 None] detach volume:Volume(_name_id=None,admin_metadata=,attach_status='attached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:25:53Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-AttachVolumeShelveTestJSON-volume-1457837636',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=13bf568b-585c-4556-ab78-e9930ad50214,launched_at=2023-04-17T15:25:55Z,metadata=,migration_status=None,multiattach=False,previous_status=None,project_id='214a0f233f114d7aa22e64048a837c0b',provider_auth='CHAP cajgFQvq cTCKdDR0kw38ze',provider_geometry=None,provider_id=None,provider_location='172.16.0.220:3260,1 iqn.2010-10.org.openstack:13bf568b-585c-4556-ab78-e9930ad50214 0',replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:25:53Z,service_uuid='e628352e-c4cf-40eb-8389-cce40663026a',shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='in-use',terminated_at=None,updated_at=2023-04-17T15:31:43Z,use_quota=True,user_id='83b9d77de46143d8b9277a55f5ad4d72',volume_attachment=,volume_type=,volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c) from target:iqn.2010-10.org.openstack:13bf568b-585c-4556-ab78-e9930ad50214. {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:913}} Apr 17 15:52:09 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [req-778c62be-8963-4e6d-97f1-f456b759ce16 req-00547a1c-f80b-48db-8093-70f53a331fbe tempest-AttachVolumeShelveTestJSON-1745617049 None] detach volume v_13bf568b-585c-4556-ab78-e9930ad50214 from target iqn.2010-10.org.openstack:13bf568b-585c-4556-ab78-e9930ad50214 {{(pid=84696) detach_target_vol /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:606}} Apr 17 15:52:09 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-778c62be-8963-4e6d-97f1-f456b759ce16 req-00547a1c-f80b-48db-8093-70f53a331fbe tempest-AttachVolumeShelveTestJSON-1745617049 None] Sending pool request DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:13bf568b-585c-4556-ab78-e9930ad50214/luns/v_13bf568b-585c-4556-ab78-e9930ad50214 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:52:09 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-778c62be-8963-4e6d-97f1-f456b759ce16 req-00547a1c-f80b-48db-8093-70f53a331fbe tempest-AttachVolumeShelveTestJSON-1745617049 None] Sending DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:13bf568b-585c-4556-ab78-e9930ad50214/luns/v_13bf568b-585c-4556-ab78-e9930ad50214 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:52:10 user cinder-volume[84696]: DEBUG cinder.utils [req-778c62be-8963-4e6d-97f1-f456b759ce16 req-00547a1c-f80b-48db-8093-70f53a331fbe tempest-AttachVolumeShelveTestJSON-1745617049 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 1.221(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:52:10 user cinder-volume[84696]: DEBUG cinder.utils [req-778c62be-8963-4e6d-97f1-f456b759ce16 req-00547a1c-f80b-48db-8093-70f53a331fbe tempest-AttachVolumeShelveTestJSON-1745617049 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:13bf568b-585c-4556-ab78-e9930ad50214/luns/v_13bf568b-585c-4556-ab78-e9930ad50214 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:52:13 user cinder-volume[84696]: DEBUG cinder.utils [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.854(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:52:13 user cinder-volume[84696]: DEBUG cinder.utils [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_4a2d1c00-a134-4f73-8508-86f35b58acf7/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:52:13 user cinder-volume[84696]: DEBUG cinder.utils [req-778c62be-8963-4e6d-97f1-f456b759ce16 req-00547a1c-f80b-48db-8093-70f53a331fbe tempest-AttachVolumeShelveTestJSON-1745617049 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 4.295(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:52:13 user cinder-volume[84696]: DEBUG cinder.utils [req-778c62be-8963-4e6d-97f1-f456b759ce16 req-00547a1c-f80b-48db-8093-70f53a331fbe tempest-AttachVolumeShelveTestJSON-1745617049 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:13bf568b-585c-4556-ab78-e9930ad50214/luns/v_13bf568b-585c-4556-ab78-e9930ad50214 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:52:13 user cinder-volume[84696]: DEBUG cinder.utils [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.694(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:52:13 user cinder-volume[84696]: DEBUG cinder.utils [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:52:13 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.501(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:52:13 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:52:16 user cinder-volume[84696]: DEBUG cinder.utils [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.658(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:52:16 user cinder-volume[84696]: DEBUG cinder.utils [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_dfd745ed-b0c4-4e35-b77c-a911c90ce33e/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:52:16 user cinder-volume[84696]: DEBUG cinder.utils [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.678(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:52:16 user cinder-volume[84696]: DEBUG cinder.utils [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_aea45c70-7c05-4146-847e-cbb777d3dfbe/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:52:16 user cinder-volume[84696]: DEBUG cinder.utils [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.654(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:52:16 user cinder-volume[84696]: DEBUG cinder.utils [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_fbbadf16-651a-47d5-b308-a37e56aa07f2/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:52:19 user cinder-volume[84696]: DEBUG cinder.utils [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.718(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:52:19 user cinder-volume[84696]: DEBUG cinder.utils [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_ab5889f8-b6c7-4673-a666-f4b4e5ff5fd5/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:52:19 user cinder-volume[84696]: DEBUG cinder.utils [req-778c62be-8963-4e6d-97f1-f456b759ce16 req-00547a1c-f80b-48db-8093-70f53a331fbe tempest-AttachVolumeShelveTestJSON-1745617049 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 10.444(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:52:19 user cinder-volume[84696]: DEBUG cinder.utils [req-778c62be-8963-4e6d-97f1-f456b759ce16 req-00547a1c-f80b-48db-8093-70f53a331fbe tempest-AttachVolumeShelveTestJSON-1745617049 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:13bf568b-585c-4556-ab78-e9930ad50214/luns/v_13bf568b-585c-4556-ab78-e9930ad50214 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:52:19 user cinder-volume[84696]: DEBUG cinder.utils [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.708(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:52:19 user cinder-volume[84696]: DEBUG cinder.utils [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_13cc73df-7780-4934-8cbd-7f372b1cf57a/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:52:20 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Volume reschedule parameters: True retry: {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 15:52:20 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Flow 'volume_create_manager' (089d207d-9f34-4926-89a0-81b8785be524) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:52:20 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (82c0f5e5-8513-4e74-89ed-f77b1bbf1d72) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:52:20 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (82c0f5e5-8513-4e74-89ed-f77b1bbf1d72) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:52:20Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TestVolumeBootPattern-volume-origin-952290434',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=60979eba-96b1-4cb3-bae1-8874a86cb411,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='c8a836b47d954cb9a7c1cd7c0f790426',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:52:20Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:52:20Z,use_quota=True,user_id='a80e753322224ec1a897598b5b9ed6ea',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:52:20 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (636fc593-9b63-4d21-9a0e-3c875f1e78d7) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:52:20 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (636fc593-9b63-4d21-9a0e-3c875f1e78d7) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:52:20 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (a80bdf04-688e-45c1-b49a-66e13ffd5cf5) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:52:20 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (a80bdf04-688e-45c1-b49a-66e13ffd5cf5) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'image', 'volume_id': '60979eba-96b1-4cb3-bae1-8874a86cb411', 'volume_name': 'volume-60979eba-96b1-4cb3-bae1-8874a86cb411', 'volume_size': 1, 'image_id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'image_location': (None, None), 'image_meta': {'name': 'cirros-0.5.2-x86_64-disk', 'disk_format': 'qcow2', 'container_format': 'bare', 'visibility': 'public', 'size': 16300544, 'virtual_size': 117440512, 'status': 'active', 'checksum': 'b874c39491a2377b8490f5f1e89761a4', 'protected': False, 'min_ram': 0, 'min_disk': 0, 'owner': 'a43c211b4ed542538799349489b1d884', 'os_hidden': False, 'os_hash_algo': 'sha512', 'os_hash_value': '6b813aa46bb90b4da216a4d19376593fa3f4fc7e617f03a92b7fe11e9a3981cbe8f0959dbebe36225e5f53dc4492341a4863cac4ed1ee0909f3fc78ef9c3e869', 'id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'created_at': datetime.datetime(2023, 4, 17, 15, 19, 53, tzinfo=datetime.timezone.utc), 'updated_at': datetime.datetime(2023, 4, 17, 15, 19, 55, tzinfo=datetime.timezone.utc), 'tags': [], 'file': '/v2/images/b3f0dec1-7758-44a8-82e0-76ada8837428/file', 'properties': {'hw_rng_model': 'virtio', 'owner_specified.openstack.md5': '', 'owner_specified.openstack.object': 'images/cirros-0.5.2-x86_64-disk', 'owner_specified.openstack.sha256': ''}}, 'image_service': }' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:52:20 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (54bd49fa-777c-458d-922b-b6f17f158e01) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:52:20 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (54bd49fa-777c-458d-922b-b6f17f158e01) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:52:20 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (d2e1c41b-94dd-43bf-ad21-607a2d54aabc) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:52:20 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Volume 60979eba-96b1-4cb3-bae1-8874a86cb411: being created as image with specification: {'status': 'creating', 'volume_name': 'volume-60979eba-96b1-4cb3-bae1-8874a86cb411', 'volume_size': 1, 'image_id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'image_location': (None, None), 'image_meta': {'name': 'cirros-0.5.2-x86_64-disk', 'disk_format': 'qcow2', 'container_format': 'bare', 'visibility': 'public', 'size': 16300544, 'virtual_size': 117440512, 'status': 'active', 'checksum': 'b874c39491a2377b8490f5f1e89761a4', 'protected': False, 'min_ram': 0, 'min_disk': 0, 'owner': 'a43c211b4ed542538799349489b1d884', 'os_hidden': False, 'os_hash_algo': 'sha512', 'os_hash_value': '6b813aa46bb90b4da216a4d19376593fa3f4fc7e617f03a92b7fe11e9a3981cbe8f0959dbebe36225e5f53dc4492341a4863cac4ed1ee0909f3fc78ef9c3e869', 'id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'created_at': datetime.datetime(2023, 4, 17, 15, 19, 53, tzinfo=datetime.timezone.utc), 'updated_at': datetime.datetime(2023, 4, 17, 15, 19, 55, tzinfo=datetime.timezone.utc), 'tags': [], 'file': '/v2/images/b3f0dec1-7758-44a8-82e0-76ada8837428/file', 'properties': {'hw_rng_model': 'virtio', 'owner_specified.openstack.md5': '', 'owner_specified.openstack.object': 'images/cirros-0.5.2-x86_64-disk', 'owner_specified.openstack.sha256': ''}}, 'image_service': } Apr 17 15:52:20 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Cloning 60979eba-96b1-4cb3-bae1-8874a86cb411 from image b3f0dec1-7758-44a8-82e0-76ada8837428 at location (None, None). {{(pid=84696) _create_from_image /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1071}} Apr 17 15:52:20 user cinder-volume[84696]: DEBUG cinder.coordination [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Acquiring lock "cinder-b3f0dec1-7758-44a8-82e0-76ada8837428" by "_prepare_image_cache_entry" {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:192}} Apr 17 15:52:20 user cinder-volume[84696]: DEBUG cinder.coordination [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Lock "cinder-b3f0dec1-7758-44a8-82e0-76ada8837428" acquired by "_prepare_image_cache_entry" :: waited 0.009s {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:196}} Apr 17 15:52:20 user cinder-volume[84696]: DEBUG cinder.image.cache [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Found image-volume cache entry: {'id': 1, 'image_id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'volume_id': '006f20ae-5306-44fe-afdf-e433f79284e8', 'host': 'user@joviandss-2#joviandss-2', 'size': 1, 'image_updated_at': datetime.datetime(2023, 4, 17, 15, 19, 55), 'last_used': datetime.datetime(2023, 4, 17, 15, 52, 20, 190549)}. {{(pid=84696) get_entry /opt/stack/cinder/cinder/image/cache.py:76}} Apr 17 15:52:20 user cinder-volume[84696]: DEBUG cinder.image.cache [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Image-volume cache entry image_update_at = 2023-04-17 15:19:55+00:00, requested image updated_at = 2023-04-17 15:19:55+00:00. {{(pid=84696) _should_update_entry /opt/stack/cinder/cinder/image/cache.py:259}} Apr 17 15:52:20 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Found cache entry for image = b3f0dec1-7758-44a8-82e0-76ada8837428 on host user@joviandss-2#joviandss-2. {{(pid=84696) _prepare_image_cache_entry /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:886}} Apr 17 15:52:20 user cinder-volume[84696]: DEBUG cinder.coordination [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Lock "cinder-b3f0dec1-7758-44a8-82e0-76ada8837428" released by "_prepare_image_cache_entry" :: held 0.014s {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:207}} Apr 17 15:52:20 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Attempting to retrieve cache entry for image = b3f0dec1-7758-44a8-82e0-76ada8837428 on host user@joviandss-2#joviandss-2. {{(pid=84696) _create_from_image_cache /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:827}} Apr 17 15:52:20 user cinder-volume[84696]: DEBUG cinder.image.cache [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Found image-volume cache entry: {'id': 1, 'image_id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'volume_id': '006f20ae-5306-44fe-afdf-e433f79284e8', 'host': 'user@joviandss-2#joviandss-2', 'size': 1, 'image_updated_at': datetime.datetime(2023, 4, 17, 15, 19, 55), 'last_used': datetime.datetime(2023, 4, 17, 15, 52, 20, 203743)}. {{(pid=84696) get_entry /opt/stack/cinder/cinder/image/cache.py:76}} Apr 17 15:52:20 user cinder-volume[84696]: DEBUG cinder.image.cache [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Image-volume cache entry image_update_at = 2023-04-17 15:19:55+00:00, requested image updated_at = 2023-04-17 15:19:55+00:00. {{(pid=84696) _should_update_entry /opt/stack/cinder/cinder/image/cache.py:259}} Apr 17 15:52:20 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Creating from source image-volume 006f20ae-5306-44fe-afdf-e433f79284e8 {{(pid=84696) _create_from_image_cache /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:841}} Apr 17 15:52:20 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] cloned volume 006f20ae-5306-44fe-afdf-e433f79284e8 to 60979eba-96b1-4cb3-bae1-8874a86cb411 {{(pid=84696) create_cloned_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:521}} Apr 17 15:52:20 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] cloning v_006f20ae-5306-44fe-afdf-e433f79284e8 to v_60979eba-96b1-4cb3-bae1-8874a86cb411 {{(pid=84696) _clone_object /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:456}} Apr 17 15:52:20 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] create snapshot v_60979eba-96b1-4cb3-bae1-8874a86cb411 {{(pid=84696) create_snapshot /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:635}} Apr 17 15:52:20 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:52:20 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:52:22 user cinder-volume[84696]: DEBUG cinder.utils [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 137.755(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:52:22 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:52:22 user cinder-volume[84696]: INFO cinder.message.api [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Creating message record for request_id = req-4381eeaf-f88c-49c3-a64f-a647fc1596cd Apr 17 15:52:22 user cinder-volume[84696]: DEBUG cinder.utils [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.405(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:52:22 user cinder-volume[84696]: DEBUG cinder.utils [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:52:22 user cinder-volume[84696]: DEBUG cinder.utils [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.836(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:52:22 user cinder-volume[84696]: DEBUG cinder.utils [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_68f814b0-1291-4a12-ae65-dc0892203d36/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:52:22 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 15:52:22 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (2c26f43a-c175-4bba-aeca-e78a016c2cf2) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 15:52:22 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 15:52:22 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:50:05Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-ServerActionsTestJSON-volume-872354177',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=2ed5b8aa-fb48-4255-b6d5-f0f84f733834,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='cb5f6b5ed2fc43b69c47e1fc18646545',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:50:05Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:50:05Z,use_quota=True,user_id='638a4f535e9740fc9d810639a5eb24d7',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 15:52:22 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:50:05Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-ServerActionsTestJSON-volume-872354177',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=2ed5b8aa-fb48-4255-b6d5-f0f84f733834,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='cb5f6b5ed2fc43b69c47e1fc18646545',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:50:05Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:50:05Z,use_quota=True,user_id='638a4f535e9740fc9d810639a5eb24d7',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(2ed5b8aa-fb48-4255-b6d5-f0f84f733834),volume_id=2ed5b8aa-fb48-4255-b6d5-f0f84f733834,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': '2ed5b8aa-fb48-4255-b6d5-f0f84f733834', 'volume_name': 'volume-2ed5b8aa-fb48-4255-b6d5-f0f84f733834', 'volume_size': 1}} Apr 17 15:52:22 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'retry': {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']}, 'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': '2ed5b8aa-fb48-4255-b6d5-f0f84f733834', 'volume': {'id': '2ed5b8aa-fb48-4255-b6d5-f0f84f733834', '_name_id': None, 'ec2_id': None, 'user_id': '638a4f535e9740fc9d810639a5eb24d7', 'project_id': 'cb5f6b5ed2fc43b69c47e1fc18646545', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-ServerActionsTestJSON-volume-872354177', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T15:50:04.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-2ed5b8aa-fb48-4255-b6d5-f0f84f733834', 'name_id': '2ed5b8aa-fb48-4255-b6d5-f0f84f733834', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-ServerActionsTestJSON-volume-872354177', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': 'cb5f6b5ed2fc43b69c47e1fc18646545', 'qos_specs': None, 'reservations': ['6016bd49-cebc-4e3e-a82c-96aaf7390191', '75d52529-e09a-471c-b975-4ba7744d76df', 'f897feca-3a4b-4df1-9b1f-3c8ce372c8c8', 'e5402891-ace1-4d31-9e37-0068e08e9ec2'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': '638a4f535e9740fc9d810639a5eb24d7', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-ServerActionsTestJSON-volume-872354177', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': 'cb5f6b5ed2fc43b69c47e1fc18646545', 'qos_specs': None, 'reservations': ['6016bd49-cebc-4e3e-a82c-96aaf7390191', '75d52529-e09a-471c-b975-4ba7744d76df', 'f897feca-3a4b-4df1-9b1f-3c8ce372c8c8', 'e5402891-ace1-4d31-9e37-0068e08e9ec2'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': '638a4f535e9740fc9d810639a5eb24d7', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': '638a4f535e9740fc9d810639a5eb24d7', 'metadata': {}, 'qos_specs': None}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(2ed5b8aa-fb48-4255-b6d5-f0f84f733834),volume_id=2ed5b8aa-fb48-4255-b6d5-f0f84f733834,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:50:05Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-ServerActionsTestJSON-volume-872354177',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=2ed5b8aa-fb48-4255-b6d5-f0f84f733834,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='cb5f6b5ed2fc43b69c47e1fc18646545',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:50:05Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:50:05Z,use_quota=True,user_id='638a4f535e9740fc9d810639a5eb24d7',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 15:52:22 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:50:05Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-ServerActionsTestJSON-volume-872354177',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=2ed5b8aa-fb48-4255-b6d5-f0f84f733834,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='cb5f6b5ed2fc43b69c47e1fc18646545',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:50:05Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:50:05Z,use_quota=True,user_id='638a4f535e9740fc9d810639a5eb24d7',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:50:05Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-ServerActionsTestJSON-volume-872354177',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=2ed5b8aa-fb48-4255-b6d5-f0f84f733834,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='cb5f6b5ed2fc43b69c47e1fc18646545',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:50:05Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:50:05Z,use_quota=True,user_id='638a4f535e9740fc9d810639a5eb24d7',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 15:52:22 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 15:52:22 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 15:52:22 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:52:22 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 15:52:22 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 15:52:22 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 15:52:22 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 15:52:22 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 15:52:22 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 15:52:22 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 15:52:22 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 15:52:22 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 15:52:22 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 15:52:22 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 15:52:22 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 15:52:22 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 15:52:22 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 15:52:22 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 15:52:22 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 15:52:22 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 15:52:22 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (2c26f43a-c175-4bba-aeca-e78a016c2cf2) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:52:22 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (2c26f43a-c175-4bba-aeca-e78a016c2cf2) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:52:22 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (00c1c7e6-3784-47b2-8d33-6f62020a5634) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:52:22 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (00c1c7e6-3784-47b2-8d33-6f62020a5634) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:52:22 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (b1762999-b383-4b45-8781-6d9eb6df7102) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:52:22 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (b1762999-b383-4b45-8781-6d9eb6df7102) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:52:22 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (f7be3d54-6bd4-45b9-8f47-361f0f9ad395) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:52:22 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Updating volume 2ed5b8aa-fb48-4255-b6d5-f0f84f733834 with {'scheduled_at': datetime.datetime(2023, 4, 17, 15, 52, 22, 692594), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 15:52:22 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Volume 2ed5b8aa-fb48-4255-b6d5-f0f84f733834: re-scheduling SchedulerAPI.create_volume attempt 1 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 15:52:22 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Volume 2ed5b8aa-fb48-4255-b6d5-f0f84f733834: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 15:52:22 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] deleating volume v_2ed5b8aa-fb48-4255-b6d5-f0f84f733834 {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 15:52:22 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] get snapshots for volume v_2ed5b8aa-fb48-4255-b6d5-f0f84f733834 {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 15:52:22 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_2ed5b8aa-fb48-4255-b6d5-f0f84f733834/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:52:22 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_2ed5b8aa-fb48-4255-b6d5-f0f84f733834/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:52:22 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Volume reschedule parameters: True retry: {'num_attempts': 2, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 15:52:22 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Flow 'volume_create_manager' (2ff98574-8f57-4cce-a4ee-2226cedf8965) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:52:22 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (660f6bd8-d8e2-4441-9b4c-f30e552b77bc) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:52:22 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (660f6bd8-d8e2-4441-9b4c-f30e552b77bc) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:50:05Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-ServerActionsTestJSON-volume-872354177',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=2ed5b8aa-fb48-4255-b6d5-f0f84f733834,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='cb5f6b5ed2fc43b69c47e1fc18646545',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:52:23Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:52:23Z,use_quota=True,user_id='638a4f535e9740fc9d810639a5eb24d7',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:52:22 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (7add7189-fa1b-4ede-9dc3-2da161948fbc) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:52:22 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (7add7189-fa1b-4ede-9dc3-2da161948fbc) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:52:22 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (55141689-4da4-41ed-8815-8d05d6cb7467) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:52:22 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (55141689-4da4-41ed-8815-8d05d6cb7467) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': '2ed5b8aa-fb48-4255-b6d5-f0f84f733834', 'volume_name': 'volume-2ed5b8aa-fb48-4255-b6d5-f0f84f733834', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:52:22 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (7c19f377-7dfb-4c80-acfa-ec8228e097b4) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:52:22 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (7c19f377-7dfb-4c80-acfa-ec8228e097b4) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:52:22 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (5f5d5412-4da8-445a-a3ec-11f8659486fe) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:52:22 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Volume 2ed5b8aa-fb48-4255-b6d5-f0f84f733834: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-2ed5b8aa-fb48-4255-b6d5-f0f84f733834', 'volume_size': 1} Apr 17 15:52:22 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] creating volume v_2ed5b8aa-fb48-4255-b6d5-f0f84f733834. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 15:52:22 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] create volume {'name': 'v_2ed5b8aa-fb48-4255-b6d5-f0f84f733834', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 15:52:22 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:52:22 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:52:25 user cinder-volume[84696]: DEBUG cinder.utils [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.975(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:52:25 user cinder-volume[84696]: DEBUG cinder.utils [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_2ed5b8aa-fb48-4255-b6d5-f0f84f733834/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:52:25 user cinder-volume[84696]: DEBUG cinder.utils [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.472(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:52:25 user cinder-volume[84696]: DEBUG cinder.utils [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:52:25 user cinder-volume[84696]: DEBUG cinder.utils [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.813(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:52:25 user cinder-volume[84696]: DEBUG cinder.utils [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:52:25 user cinder-volume[84696]: DEBUG cinder.utils [req-778c62be-8963-4e6d-97f1-f456b759ce16 req-8dba5389-38e6-4a46-b382-2d4356cad7db tempest-AttachVolumeShelveTestJSON-1745617049 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 136.629(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:52:25 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-778c62be-8963-4e6d-97f1-f456b759ce16 req-8dba5389-38e6-4a46-b382-2d4356cad7db tempest-AttachVolumeShelveTestJSON-1745617049 None] Geting None from DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:13bf568b-585c-4556-ab78-e9930ad50214/luns/v_13bf568b-585c-4556-ab78-e9930ad50214 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:52:25 user cinder-volume[84696]: WARNING cinder.volume.manager [req-778c62be-8963-4e6d-97f1-f456b759ce16 req-8dba5389-38e6-4a46-b382-2d4356cad7db tempest-AttachVolumeShelveTestJSON-1745617049 None] Failed to detach volume on the backend, ignoring failure 'NoneType' object is not subscriptable Apr 17 15:52:28 user cinder-volume[84696]: DEBUG cinder.utils [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.880(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:52:28 user cinder-volume[84696]: DEBUG cinder.utils [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:52:28 user cinder-volume[84696]: DEBUG cinder.utils [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.048(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:52:28 user cinder-volume[84696]: DEBUG cinder.utils [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_2ed5b8aa-fb48-4255-b6d5-f0f84f733834/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:52:28 user cinder-volume[84696]: DEBUG cinder.utils [req-778c62be-8963-4e6d-97f1-f456b759ce16 req-00547a1c-f80b-48db-8093-70f53a331fbe tempest-AttachVolumeShelveTestJSON-1745617049 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 19.655(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:52:28 user cinder-volume[84696]: DEBUG cinder.utils [req-778c62be-8963-4e6d-97f1-f456b759ce16 req-00547a1c-f80b-48db-8093-70f53a331fbe tempest-AttachVolumeShelveTestJSON-1745617049 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:13bf568b-585c-4556-ab78-e9930ad50214/luns/v_13bf568b-585c-4556-ab78-e9930ad50214 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:52:31 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.927(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:52:31 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:52:31 user cinder-volume[84696]: DEBUG cinder.utils [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 11.614(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:52:31 user cinder-volume[84696]: DEBUG cinder.utils [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:52:32 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=19,cinder:UPDATE=5,cinder:INSERT=1 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:52:34 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=17,cinder:UPDATE=5,cinder:INSERT=1 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:52:34 user cinder-volume[84696]: DEBUG cinder.utils [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.028(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:52:34 user cinder-volume[84696]: DEBUG cinder.utils [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:52:34 user cinder-volume[84696]: DEBUG cinder.utils [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.197(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:52:34 user cinder-volume[84696]: DEBUG cinder.utils [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_2ed5b8aa-fb48-4255-b6d5-f0f84f733834/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:52:34 user cinder-volume[84696]: DEBUG cinder.utils [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 139.155(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:52:34 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:52:34 user cinder-volume[84696]: INFO cinder.message.api [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Creating message record for request_id = req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 Apr 17 15:52:34 user cinder-volume[84696]: DEBUG cinder.utils [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 139.326(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:52:34 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_cdedb820-e1c7-4b86-8286-43b3640b3a46/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:52:34 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (427e4daa-48b1-40f1-b755-ad1fe82a7c04) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 15:52:34 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (86278ba8-51c9-49d1-ad15-468a700d90e8) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:52:34 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (86278ba8-51c9-49d1-ad15-468a700d90e8) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:52:34 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Flow 'volume_create_manager' (b731ba4a-4006-467b-913f-f8ef3652da50) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 15:52:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 15:52:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 15:52:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 15:52:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 15:52:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 15:52:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 15:52:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 15:52:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 15:52:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 15:52:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 15:52:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 15:52:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 15:52:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 15:52:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 15:52:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 15:52:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 15:52:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 15:52:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 15:52:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 15:52:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 15:52:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 15:52:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 15:52:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 15:52:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 15:52:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:52:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 15:52:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 15:52:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 15:52:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 15:52:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 15:52:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 15:52:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 15:52:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 15:52:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 15:52:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 15:52:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 15:52:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 15:52:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 15:52:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 15:52:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 15:52:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 15:52:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 15:52:34 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 15:52:34 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (33c814c5-0243-4381-8577-703ee4293c85) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 15:52:34 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 15:52:34 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:47:59Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-ServerRescueNegativeTestJSON-volume-7015736',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=cdedb820-e1c7-4b86-8286-43b3640b3a46,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='9324c99bd4d040bdb5a7f1db782b2783',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:50:16Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:50:16Z,use_quota=True,user_id='8e3487d6b32d46beac556617050fd664',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 15:52:34 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:47:59Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-ServerRescueNegativeTestJSON-volume-7015736',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=cdedb820-e1c7-4b86-8286-43b3640b3a46,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='9324c99bd4d040bdb5a7f1db782b2783',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:50:16Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:50:16Z,use_quota=True,user_id='8e3487d6b32d46beac556617050fd664',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(cdedb820-e1c7-4b86-8286-43b3640b3a46),volume_id=cdedb820-e1c7-4b86-8286-43b3640b3a46,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': 'cdedb820-e1c7-4b86-8286-43b3640b3a46', 'volume_name': 'volume-cdedb820-e1c7-4b86-8286-43b3640b3a46', 'volume_size': 1}} Apr 17 15:52:34 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': 'cdedb820-e1c7-4b86-8286-43b3640b3a46', 'volume': {'id': 'cdedb820-e1c7-4b86-8286-43b3640b3a46', '_name_id': None, 'ec2_id': None, 'user_id': '8e3487d6b32d46beac556617050fd664', 'project_id': '9324c99bd4d040bdb5a7f1db782b2783', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-ServerRescueNegativeTestJSON-volume-7015736', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'admin_metadata': {}, 'glance_metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T15:47:59.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-cdedb820-e1c7-4b86-8286-43b3640b3a46', 'name_id': 'cdedb820-e1c7-4b86-8286-43b3640b3a46', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-ServerRescueNegativeTestJSON-volume-7015736', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '9324c99bd4d040bdb5a7f1db782b2783', 'qos_specs': None, 'reservations': ['f9dad618-bdce-4da3-89f6-14cc03e04ddc', 'f52bc1e3-5f13-4f5b-b4df-610d006660f1', '9b6eb134-10fa-4f48-8b07-6e9470fc5aae', 'f45d6340-19c8-4636-9836-b94ea2be5f46'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': '8e3487d6b32d46beac556617050fd664', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-ServerRescueNegativeTestJSON-volume-7015736', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '9324c99bd4d040bdb5a7f1db782b2783', 'qos_specs': None, 'reservations': ['f9dad618-bdce-4da3-89f6-14cc03e04ddc', 'f52bc1e3-5f13-4f5b-b4df-610d006660f1', '9b6eb134-10fa-4f48-8b07-6e9470fc5aae', 'f45d6340-19c8-4636-9836-b94ea2be5f46'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': '8e3487d6b32d46beac556617050fd664', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': '8e3487d6b32d46beac556617050fd664', 'metadata': {}, 'qos_specs': None, 'retry': {'num_attempts': 2, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']}}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(cdedb820-e1c7-4b86-8286-43b3640b3a46),volume_id=cdedb820-e1c7-4b86-8286-43b3640b3a46,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:47:59Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-ServerRescueNegativeTestJSON-volume-7015736',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=cdedb820-e1c7-4b86-8286-43b3640b3a46,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='9324c99bd4d040bdb5a7f1db782b2783',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:50:16Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:50:16Z,use_quota=True,user_id='8e3487d6b32d46beac556617050fd664',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 15:52:34 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:47:59Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-ServerRescueNegativeTestJSON-volume-7015736',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=cdedb820-e1c7-4b86-8286-43b3640b3a46,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='9324c99bd4d040bdb5a7f1db782b2783',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:50:16Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:50:16Z,use_quota=True,user_id='8e3487d6b32d46beac556617050fd664',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:47:59Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-ServerRescueNegativeTestJSON-volume-7015736',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=cdedb820-e1c7-4b86-8286-43b3640b3a46,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='9324c99bd4d040bdb5a7f1db782b2783',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:50:16Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:50:16Z,use_quota=True,user_id='8e3487d6b32d46beac556617050fd664',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 15:52:34 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 15:52:34 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 15:52:34 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:52:34 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 15:52:34 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 15:52:34 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 15:52:34 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 15:52:34 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 15:52:34 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 15:52:34 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 15:52:34 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 15:52:34 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 15:52:34 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 15:52:34 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 15:52:34 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 15:52:34 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 15:52:34 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 15:52:34 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 15:52:34 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 15:52:34 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 15:52:34 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (33c814c5-0243-4381-8577-703ee4293c85) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:52:34 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (33c814c5-0243-4381-8577-703ee4293c85) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:52:34 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (41b3a978-1d00-451f-8f78-d69707c91ee9) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:52:34 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (41b3a978-1d00-451f-8f78-d69707c91ee9) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:52:34 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (0adabb07-eec7-46a0-a0e6-d669b71fdef4) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:52:34 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (0adabb07-eec7-46a0-a0e6-d669b71fdef4) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:52:35 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (18b1c95e-08dd-44a9-9e90-087b0effba30) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:52:35 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Updating volume cdedb820-e1c7-4b86-8286-43b3640b3a46 with {'scheduled_at': datetime.datetime(2023, 4, 17, 15, 52, 35, 1453), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 15:52:35 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Volume cdedb820-e1c7-4b86-8286-43b3640b3a46: re-scheduling SchedulerAPI.create_volume attempt 2 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 15:52:35 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Volume cdedb820-e1c7-4b86-8286-43b3640b3a46: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 15:52:35 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] deleating volume v_cdedb820-e1c7-4b86-8286-43b3640b3a46 {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 15:52:35 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] get snapshots for volume v_cdedb820-e1c7-4b86-8286-43b3640b3a46 {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 15:52:35 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_cdedb820-e1c7-4b86-8286-43b3640b3a46/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:52:35 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_cdedb820-e1c7-4b86-8286-43b3640b3a46/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:52:35 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Volume reschedule parameters: True retry: {'num_attempts': 3, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 15:52:35 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Flow 'volume_create_manager' (f71e7e7d-84a9-4c36-a4dc-68f30540149b) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:52:35 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (268945a7-b980-4e7b-a0ed-3456111a0c53) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:52:35 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (268945a7-b980-4e7b-a0ed-3456111a0c53) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:47:59Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-ServerRescueNegativeTestJSON-volume-7015736',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=cdedb820-e1c7-4b86-8286-43b3640b3a46,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='9324c99bd4d040bdb5a7f1db782b2783',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:52:35Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:52:35Z,use_quota=True,user_id='8e3487d6b32d46beac556617050fd664',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:52:35 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (03f8a8f7-ceef-40f0-93f7-9b4d6ce5a8ac) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:52:35 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (03f8a8f7-ceef-40f0-93f7-9b4d6ce5a8ac) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:52:35 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (9a2a11d0-78c3-4023-880b-4bb2d8f4fd7e) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:52:35 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (9a2a11d0-78c3-4023-880b-4bb2d8f4fd7e) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': 'cdedb820-e1c7-4b86-8286-43b3640b3a46', 'volume_name': 'volume-cdedb820-e1c7-4b86-8286-43b3640b3a46', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:52:35 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (820a4a8e-fefa-4df6-a3a5-d1512e6ec15b) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:52:35 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (820a4a8e-fefa-4df6-a3a5-d1512e6ec15b) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:52:35 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (bf97356d-bfe5-42d1-8d21-73dc45767690) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:52:35 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Volume cdedb820-e1c7-4b86-8286-43b3640b3a46: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-cdedb820-e1c7-4b86-8286-43b3640b3a46', 'volume_size': 1} Apr 17 15:52:35 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] creating volume v_cdedb820-e1c7-4b86-8286-43b3640b3a46. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 15:52:35 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] create volume {'name': 'v_cdedb820-e1c7-4b86-8286-43b3640b3a46', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 15:52:35 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:52:35 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:52:37 user cinder-volume[84696]: DEBUG cinder.utils [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.767(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:52:37 user cinder-volume[84696]: DEBUG cinder.utils [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:52:37 user cinder-volume[84696]: DEBUG cinder.utils [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.956(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:52:37 user cinder-volume[84696]: DEBUG cinder.utils [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_cdedb820-e1c7-4b86-8286-43b3640b3a46/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:52:41 user cinder-volume[84696]: DEBUG cinder.utils [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.026(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:52:41 user cinder-volume[84696]: DEBUG cinder.utils [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_cdedb820-e1c7-4b86-8286-43b3640b3a46/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:52:41 user cinder-volume[84696]: DEBUG cinder.utils [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 20.830(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:52:41 user cinder-volume[84696]: DEBUG cinder.utils [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:52:41 user cinder-volume[84696]: DEBUG cinder.utils [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.841(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:52:41 user cinder-volume[84696]: DEBUG cinder.utils [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:52:44 user cinder-volume[84696]: DEBUG cinder.utils [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 139.330(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:52:44 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_93a32a7c-4c6a-472a-8f91-9d34558ba232/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:52:44 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (5a647616-587e-451f-ad41-772bf631bc61) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 15:52:44 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (66bdcfbf-3b5a-4eeb-8a0e-92299cac60a6) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:52:44 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (66bdcfbf-3b5a-4eeb-8a0e-92299cac60a6) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:52:44 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Flow 'volume_create_manager' (9854c8a1-2cb7-4529-964f-3d4607c1552f) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 15:52:44 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 15:52:44 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 15:52:44 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 15:52:44 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 15:52:44 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 15:52:44 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 15:52:44 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 15:52:44 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 15:52:44 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 15:52:44 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 15:52:44 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 15:52:44 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 15:52:44 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 15:52:44 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 15:52:44 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 15:52:44 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 15:52:44 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 15:52:44 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 15:52:44 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 15:52:44 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 15:52:44 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 15:52:44 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 15:52:44 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 15:52:44 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 15:52:44 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:52:44 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 15:52:44 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1237, in execute Apr 17 15:52:44 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_from_image(context, Apr 17 15:52:44 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/utils.py", line 670, in _wrapper Apr 17 15:52:44 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return r(f, *args, **kwargs) Apr 17 15:52:44 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/tenacity/__init__.py", line 423, in __call__ Apr 17 15:52:44 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server do = self.iter(retry_state=retry_state) Apr 17 15:52:44 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/tenacity/__init__.py", line 360, in iter Apr 17 15:52:44 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return fut.result() Apr 17 15:52:44 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/lib/python3.10/concurrent/futures/_base.py", line 451, in result Apr 17 15:52:44 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self.__get_result() Apr 17 15:52:44 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result Apr 17 15:52:44 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self._exception Apr 17 15:52:44 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/tenacity/__init__.py", line 426, in __call__ Apr 17 15:52:44 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = fn(*args, **kwargs) Apr 17 15:52:44 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1125, in _create_from_image Apr 17 15:52:44 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_from_image_cache_or_download( Apr 17 15:52:44 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 944, in _create_from_image_cache_or_download Apr 17 15:52:44 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update, cloned = self._create_from_image_cache( Apr 17 15:52:44 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 843, in _create_from_image_cache Apr 17 15:52:44 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_from_source_volume( Apr 17 15:52:44 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 659, in _create_from_source_volume Apr 17 15:52:44 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self.driver.create_cloned_volume(volume, srcvol_ref) Apr 17 15:52:44 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 525, in create_cloned_volume Apr 17 15:52:44 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self._clone_object(vname, cvname) Apr 17 15:52:44 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 461, in _clone_object Apr 17 15:52:44 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_snapshot(oname, coname) Apr 17 15:52:44 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 639, in create_snapshot Apr 17 15:52:44 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201, 204): Apr 17 15:52:44 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 15:52:44 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 15:52:44 user cinder-volume[84696]: DEBUG cinder.utils [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.262(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:52:44 user cinder-volume[84696]: DEBUG cinder.utils [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:52:44 user cinder-volume[84696]: DEBUG cinder.utils [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.430(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:52:44 user cinder-volume[84696]: DEBUG cinder.utils [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_2ed5b8aa-fb48-4255-b6d5-f0f84f733834/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:52:44 user cinder-volume[84696]: DEBUG cinder.utils [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 138.999(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:52:44 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:52:44 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (c700abaf-a7c0-4e50-bd3e-01d84520168d) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 15:52:44 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 15:52:44 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:45:47Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TestVolumeBootPattern-volume-origin-14805124',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=93a32a7c-4c6a-472a-8f91-9d34558ba232,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='c8a836b47d954cb9a7c1cd7c0f790426',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:50:25Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:50:25Z,use_quota=True,user_id='a80e753322224ec1a897598b5b9ed6ea',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 15:52:44 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:45:47Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TestVolumeBootPattern-volume-origin-14805124',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=93a32a7c-4c6a-472a-8f91-9d34558ba232,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='c8a836b47d954cb9a7c1cd7c0f790426',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:50:25Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:50:25Z,use_quota=True,user_id='a80e753322224ec1a897598b5b9ed6ea',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=b3f0dec1-7758-44a8-82e0-76ada8837428,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(93a32a7c-4c6a-472a-8f91-9d34558ba232),volume_id=93a32a7c-4c6a-472a-8f91-9d34558ba232,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'image', 'volume_id': '93a32a7c-4c6a-472a-8f91-9d34558ba232', 'volume_name': 'volume-93a32a7c-4c6a-472a-8f91-9d34558ba232', 'volume_size': 1, 'image_id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'image_location': (None, None), 'image_meta': {'name': 'cirros-0.5.2-x86_64-disk', 'disk_format': 'qcow2', 'container_format': 'bare', 'visibility': 'public', 'size': 16300544, 'virtual_size': 117440512, 'status': 'active', 'checksum': 'b874c39491a2377b8490f5f1e89761a4', 'protected': False, 'min_ram': 0, 'min_disk': 0, 'owner': 'a43c211b4ed542538799349489b1d884', 'os_hidden': False, 'os_hash_algo': 'sha512', 'os_hash_value': '6b813aa46bb90b4da216a4d19376593fa3f4fc7e617f03a92b7fe11e9a3981cbe8f0959dbebe36225e5f53dc4492341a4863cac4ed1ee0909f3fc78ef9c3e869', 'id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'created_at': datetime.datetime(2023, 4, 17, 15, 19, 53, tzinfo=datetime.timezone.utc), 'updated_at': datetime.datetime(2023, 4, 17, 15, 19, 55, tzinfo=datetime.timezone.utc), 'tags': [], 'file': '/v2/images/b3f0dec1-7758-44a8-82e0-76ada8837428/file', 'properties': {'hw_rng_model': 'virtio', 'owner_specified.openstack.md5': '', 'owner_specified.openstack.object': 'images/cirros-0.5.2-x86_64-disk', 'owner_specified.openstack.sha256': ''}}, 'image_service': }} Apr 17 15:52:44 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'snapshot_id': None, 'source_volid': None, 'volume_id': '93a32a7c-4c6a-472a-8f91-9d34558ba232', 'volume': {'id': '93a32a7c-4c6a-472a-8f91-9d34558ba232', '_name_id': None, 'ec2_id': None, 'user_id': 'a80e753322224ec1a897598b5b9ed6ea', 'project_id': 'c8a836b47d954cb9a7c1cd7c0f790426', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-TestVolumeBootPattern-volume-origin-14805124', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'admin_metadata': {}, 'glance_metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T15:45:47.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-93a32a7c-4c6a-472a-8f91-9d34558ba232', 'name_id': '93a32a7c-4c6a-472a-8f91-9d34558ba232', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-TestVolumeBootPattern-volume-origin-14805124', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': 'c8a836b47d954cb9a7c1cd7c0f790426', 'qos_specs': None, 'reservations': ['c48dcd83-2285-48af-a8d0-ff11edc29cb9', 'd1463595-875c-453d-8e19-7c670de227bd', '59caf45d-ddd8-48db-b7c0-c2b483358ca9', 'deb0cce9-9fd9-4e11-8704-901c42ef169b'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'a80e753322224ec1a897598b5b9ed6ea', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-TestVolumeBootPattern-volume-origin-14805124', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': 'c8a836b47d954cb9a7c1cd7c0f790426', 'qos_specs': None, 'reservations': ['c48dcd83-2285-48af-a8d0-ff11edc29cb9', 'd1463595-875c-453d-8e19-7c670de227bd', '59caf45d-ddd8-48db-b7c0-c2b483358ca9', 'deb0cce9-9fd9-4e11-8704-901c42ef169b'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'a80e753322224ec1a897598b5b9ed6ea', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': 'a80e753322224ec1a897598b5b9ed6ea', 'metadata': {}, 'qos_specs': None, 'retry': {'num_attempts': 3, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']}}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=b3f0dec1-7758-44a8-82e0-76ada8837428,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(93a32a7c-4c6a-472a-8f91-9d34558ba232),volume_id=93a32a7c-4c6a-472a-8f91-9d34558ba232,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:45:47Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TestVolumeBootPattern-volume-origin-14805124',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=93a32a7c-4c6a-472a-8f91-9d34558ba232,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='c8a836b47d954cb9a7c1cd7c0f790426',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:50:25Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:50:25Z,use_quota=True,user_id='a80e753322224ec1a897598b5b9ed6ea',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 15:52:44 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:45:47Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TestVolumeBootPattern-volume-origin-14805124',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=93a32a7c-4c6a-472a-8f91-9d34558ba232,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='c8a836b47d954cb9a7c1cd7c0f790426',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:50:25Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:50:25Z,use_quota=True,user_id='a80e753322224ec1a897598b5b9ed6ea',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:45:47Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TestVolumeBootPattern-volume-origin-14805124',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=93a32a7c-4c6a-472a-8f91-9d34558ba232,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='c8a836b47d954cb9a7c1cd7c0f790426',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:50:25Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:50:25Z,use_quota=True,user_id='a80e753322224ec1a897598b5b9ed6ea',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 15:52:44 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 15:52:44 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 15:52:44 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:52:44 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 15:52:44 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1237, in execute Apr 17 15:52:44 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_from_image(context, Apr 17 15:52:44 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/utils.py", line 670, in _wrapper Apr 17 15:52:44 user cinder-volume[84696]: ERROR cinder.volume.manager return r(f, *args, **kwargs) Apr 17 15:52:44 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/tenacity/__init__.py", line 423, in __call__ Apr 17 15:52:44 user cinder-volume[84696]: ERROR cinder.volume.manager do = self.iter(retry_state=retry_state) Apr 17 15:52:44 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/tenacity/__init__.py", line 360, in iter Apr 17 15:52:44 user cinder-volume[84696]: ERROR cinder.volume.manager return fut.result() Apr 17 15:52:44 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/lib/python3.10/concurrent/futures/_base.py", line 451, in result Apr 17 15:52:44 user cinder-volume[84696]: ERROR cinder.volume.manager return self.__get_result() Apr 17 15:52:44 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result Apr 17 15:52:44 user cinder-volume[84696]: ERROR cinder.volume.manager raise self._exception Apr 17 15:52:44 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/tenacity/__init__.py", line 426, in __call__ Apr 17 15:52:44 user cinder-volume[84696]: ERROR cinder.volume.manager result = fn(*args, **kwargs) Apr 17 15:52:44 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1125, in _create_from_image Apr 17 15:52:44 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_from_image_cache_or_download( Apr 17 15:52:44 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 944, in _create_from_image_cache_or_download Apr 17 15:52:44 user cinder-volume[84696]: ERROR cinder.volume.manager model_update, cloned = self._create_from_image_cache( Apr 17 15:52:44 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 843, in _create_from_image_cache Apr 17 15:52:44 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_from_source_volume( Apr 17 15:52:44 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 659, in _create_from_source_volume Apr 17 15:52:44 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self.driver.create_cloned_volume(volume, srcvol_ref) Apr 17 15:52:44 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 525, in create_cloned_volume Apr 17 15:52:44 user cinder-volume[84696]: ERROR cinder.volume.manager self._clone_object(vname, cvname) Apr 17 15:52:44 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 461, in _clone_object Apr 17 15:52:44 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_snapshot(oname, coname) Apr 17 15:52:44 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 639, in create_snapshot Apr 17 15:52:44 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201, 204): Apr 17 15:52:44 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 15:52:44 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 15:52:44 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (c700abaf-a7c0-4e50-bd3e-01d84520168d) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:52:44 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (c700abaf-a7c0-4e50-bd3e-01d84520168d) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:52:44 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (231e66e0-36a5-41c2-a1be-8f99a877675f) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:52:44 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (231e66e0-36a5-41c2-a1be-8f99a877675f) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:52:44 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (2600e76a-0786-4380-9fe6-194c0445d2ab) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:52:44 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (2600e76a-0786-4380-9fe6-194c0445d2ab) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:52:44 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (6caa0046-7037-45da-bf8d-6687d5b2e814) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:52:44 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Updating volume 93a32a7c-4c6a-472a-8f91-9d34558ba232 with {'scheduled_at': datetime.datetime(2023, 4, 17, 15, 52, 44, 193634), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 15:52:44 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Volume 93a32a7c-4c6a-472a-8f91-9d34558ba232: re-scheduling SchedulerAPI.create_volume attempt 3 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 15:52:44 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Volume 93a32a7c-4c6a-472a-8f91-9d34558ba232: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 15:52:44 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] deleating volume v_93a32a7c-4c6a-472a-8f91-9d34558ba232 {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 15:52:44 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] get snapshots for volume v_93a32a7c-4c6a-472a-8f91-9d34558ba232 {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 15:52:44 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_93a32a7c-4c6a-472a-8f91-9d34558ba232/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:52:44 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_93a32a7c-4c6a-472a-8f91-9d34558ba232/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:52:47 user cinder-volume[84696]: DEBUG cinder.utils [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.969(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:52:47 user cinder-volume[84696]: DEBUG cinder.utils [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_93a32a7c-4c6a-472a-8f91-9d34558ba232/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:52:47 user cinder-volume[84696]: DEBUG cinder.utils [req-778c62be-8963-4e6d-97f1-f456b759ce16 req-00547a1c-f80b-48db-8093-70f53a331fbe tempest-AttachVolumeShelveTestJSON-1745617049 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 38.092(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:52:47 user cinder-volume[84696]: DEBUG cinder.utils [req-778c62be-8963-4e6d-97f1-f456b759ce16 req-00547a1c-f80b-48db-8093-70f53a331fbe tempest-AttachVolumeShelveTestJSON-1745617049 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:13bf568b-585c-4556-ab78-e9930ad50214/luns/v_13bf568b-585c-4556-ab78-e9930ad50214 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:52:47 user cinder-volume[84696]: DEBUG cinder.utils [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 73.654(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:52:47 user cinder-volume[84696]: DEBUG cinder.utils [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_4a2d1c00-a134-4f73-8508-86f35b58acf7/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:52:47 user cinder-volume[84696]: DEBUG cinder.utils [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.179(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:52:47 user cinder-volume[84696]: DEBUG cinder.utils [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_cdedb820-e1c7-4b86-8286-43b3640b3a46/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:52:47 user cinder-volume[84696]: DEBUG cinder.utils [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 11.993(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:52:47 user cinder-volume[84696]: DEBUG cinder.utils [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:52:47 user cinder-volume[84696]: DEBUG cinder.utils [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 73.494(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:52:47 user cinder-volume[84696]: DEBUG cinder.utils [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:52:50 user cinder-volume[84696]: DEBUG cinder.utils [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 73.468(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:52:50 user cinder-volume[84696]: DEBUG cinder.utils [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_aea45c70-7c05-4146-847e-cbb777d3dfbe/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:52:50 user cinder-volume[84696]: DEBUG cinder.utils [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 73.445(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:52:50 user cinder-volume[84696]: DEBUG cinder.utils [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_fbbadf16-651a-47d5-b308-a37e56aa07f2/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:52:50 user cinder-volume[84696]: DEBUG cinder.utils [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 73.454(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:52:50 user cinder-volume[84696]: DEBUG cinder.utils [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_dfd745ed-b0c4-4e35-b77c-a911c90ce33e/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:52:50 user cinder-volume[84696]: DEBUG cinder.utils [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.043(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:52:50 user cinder-volume[84696]: DEBUG cinder.utils [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_93a32a7c-4c6a-472a-8f91-9d34558ba232/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:52:53 user cinder-volume[84696]: DEBUG cinder.utils [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 73.492(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:52:53 user cinder-volume[84696]: DEBUG cinder.utils [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_13cc73df-7780-4934-8cbd-7f372b1cf57a/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:52:53 user cinder-volume[84696]: DEBUG cinder.utils [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 73.507(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:52:53 user cinder-volume[84696]: DEBUG cinder.utils [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_ab5889f8-b6c7-4673-a666-f4b4e5ff5fd5/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:52:54 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=11,cinder:UPDATE=4,cinder:INSERT=2 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:52:56 user cinder-volume[84696]: DEBUG cinder.utils [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 73.618(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:52:56 user cinder-volume[84696]: DEBUG cinder.utils [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_68f814b0-1291-4a12-ae65-dc0892203d36/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:52:56 user cinder-volume[84696]: DEBUG cinder.utils [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 139.235(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:52:56 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_5c16bb82-43c5-4d91-9320-0e1f2bbec574/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:52:56 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (e09b31ab-cc40-4bfc-95b8-19c7269b9325) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 15:52:56 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (52d382d0-f289-4b52-aacc-943b882e8e39) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:52:56 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (52d382d0-f289-4b52-aacc-943b882e8e39) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:52:56 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Flow 'volume_create_manager' (9decf911-1891-4460-a859-237cd7e861bd) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 15:52:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-17ff3f5b-2f53-46f9-aad4-ca363f7dd09b tempest-TestMinimumBasicScenario-1774469422 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 15:52:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 15:52:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 15:52:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 15:52:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 15:52:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 15:52:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 15:52:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 15:52:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 15:52:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 15:52:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 15:52:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 15:52:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 15:52:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 15:52:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 15:52:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 15:52:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 15:52:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 15:52:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 15:52:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 15:52:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 15:52:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 15:52:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 15:52:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 15:52:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:52:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 15:52:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 15:52:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 15:52:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 15:52:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 15:52:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 15:52:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 15:52:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 15:52:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 15:52:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 15:52:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 15:52:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 15:52:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 15:52:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 15:52:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 15:52:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 15:52:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 15:52:56 user cinder-volume[84696]: DEBUG cinder.utils [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.220(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:52:56 user cinder-volume[84696]: DEBUG cinder.utils [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:52:56 user cinder-volume[84696]: DEBUG cinder.utils [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.409(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:52:56 user cinder-volume[84696]: DEBUG cinder.utils [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_cdedb820-e1c7-4b86-8286-43b3640b3a46/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:52:56 user cinder-volume[84696]: DEBUG cinder.utils [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.205(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:52:56 user cinder-volume[84696]: DEBUG cinder.utils [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_93a32a7c-4c6a-472a-8f91-9d34558ba232/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:52:59 user cinder-volume[84696]: DEBUG cinder.utils [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.261(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:52:59 user cinder-volume[84696]: DEBUG cinder.utils [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:53:02 user cinder-volume[84696]: DEBUG cinder.utils [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.672(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:53:02 user cinder-volume[84696]: DEBUG cinder.utils [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:53:02 user cinder-volume[84696]: DEBUG cinder.utils [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.840(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:53:02 user cinder-volume[84696]: DEBUG cinder.utils [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_2ed5b8aa-fb48-4255-b6d5-f0f84f733834/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:53:02 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Volume reschedule parameters: True retry: {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 15:53:02 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Flow 'volume_create_manager' (a7bdd606-b7ca-4091-9de2-cbe53cb9e5cf) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:53:02 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (057569d6-b4e2-468f-a12e-63b4e1de4f14) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:53:02 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (057569d6-b4e2-468f-a12e-63b4e1de4f14) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:53:03Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TestMinimumBasicScenario-volume-1479398066',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=45e5cc09-6c97-438a-a6a2-c89e7d725ce5,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='17d1dae0ad7149bbbd160525d64c0c20',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:53:03Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:53:03Z,use_quota=True,user_id='3c407d9d494d485c9c95dc3981284fba',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:53:02 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (9319b2f5-7d61-40d7-8b78-e1d9ddfb790f) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:53:02 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (9319b2f5-7d61-40d7-8b78-e1d9ddfb790f) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:53:02 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (3657c247-1ac6-428f-924c-4b1012051d37) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:53:02 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (3657c247-1ac6-428f-924c-4b1012051d37) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': '45e5cc09-6c97-438a-a6a2-c89e7d725ce5', 'volume_name': 'volume-45e5cc09-6c97-438a-a6a2-c89e7d725ce5', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:53:02 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (e4c4e732-35d7-45ec-811e-7c5c7f5fa748) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:53:02 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (e4c4e732-35d7-45ec-811e-7c5c7f5fa748) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:53:02 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (c50c4515-d71e-4b74-a113-c2bfb57d5d2c) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:53:02 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Volume 45e5cc09-6c97-438a-a6a2-c89e7d725ce5: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-45e5cc09-6c97-438a-a6a2-c89e7d725ce5', 'volume_size': 1} Apr 17 15:53:02 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] creating volume v_45e5cc09-6c97-438a-a6a2-c89e7d725ce5. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 15:53:02 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] create volume {'name': 'v_45e5cc09-6c97-438a-a6a2-c89e7d725ce5', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 15:53:02 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:53:02 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:53:05 user cinder-volume[84696]: DEBUG cinder.utils [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.845(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:53:05 user cinder-volume[84696]: DEBUG cinder.utils [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:53:05 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 73.725(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:53:05 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:53:05 user cinder-volume[84696]: DEBUG cinder.utils [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.404(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:53:05 user cinder-volume[84696]: DEBUG cinder.utils [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_93a32a7c-4c6a-472a-8f91-9d34558ba232/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:53:08 user cinder-volume[84696]: DEBUG cinder.utils [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.913(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:53:08 user cinder-volume[84696]: DEBUG cinder.utils [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:53:11 user cinder-volume[84696]: DEBUG cinder.utils [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 136.240(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:53:11 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_2c9714de-f8ea-4d1d-99f9-beb85126ad91/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:53:11 user cinder-volume[84696]: WARNING cinder.volume.manager [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (2e236988-b8a4-435d-9f21-a584f5eac774) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 15:53:11 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (78d194dd-793d-4ef0-b16a-5e0e901e6740) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:53:11 user cinder-volume[84696]: WARNING cinder.volume.manager [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (78d194dd-793d-4ef0-b16a-5e0e901e6740) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:53:11 user cinder-volume[84696]: WARNING cinder.volume.manager [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Flow 'volume_create_manager' (84ab4fd6-b295-4c89-8f6c-5cf3b6f20b59) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 15:53:11 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [req-64a49643-ef0f-426a-909b-8f85130cc57b req-636827e5-ca50-4d75-a738-9b54469bf4fe tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 15:53:11 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 15:53:11 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 15:53:11 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 15:53:11 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 15:53:11 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 15:53:11 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 15:53:11 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 15:53:11 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 15:53:11 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 15:53:11 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 15:53:11 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 15:53:11 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 15:53:11 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 15:53:11 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 15:53:11 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 15:53:11 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 15:53:11 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 15:53:11 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 15:53:11 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 15:53:11 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 15:53:11 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 15:53:11 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 15:53:11 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 15:53:11 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:53:11 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 15:53:11 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 15:53:11 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 15:53:11 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 15:53:11 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 15:53:11 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 15:53:11 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 15:53:11 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 15:53:11 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 15:53:11 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 15:53:11 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 15:53:11 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 15:53:11 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 15:53:11 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 15:53:11 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 15:53:11 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 15:53:11 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 15:53:14 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:INSERT=1,cinder:UPDATE=5,cinder:SELECT=14 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:53:15 user cinder-volume[84696]: DEBUG cinder.utils [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 40.458(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:53:15 user cinder-volume[84696]: DEBUG cinder.utils [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_cdedb820-e1c7-4b86-8286-43b3640b3a46/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:53:15 user cinder-volume[84696]: DEBUG cinder.utils [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.700(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:53:15 user cinder-volume[84696]: DEBUG cinder.utils [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:53:15 user cinder-volume[84696]: DEBUG cinder.utils [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 40.274(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:53:15 user cinder-volume[84696]: DEBUG cinder.utils [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:53:22 user cinder-volume[84696]: DEBUG cinder.utils [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 38.038(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:53:22 user cinder-volume[84696]: DEBUG cinder.utils [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_93a32a7c-4c6a-472a-8f91-9d34558ba232/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:53:22 user cinder-volume[84696]: DEBUG cinder.utils [req-778c62be-8963-4e6d-97f1-f456b759ce16 req-00547a1c-f80b-48db-8093-70f53a331fbe tempest-AttachVolumeShelveTestJSON-1745617049 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 73.161(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:53:22 user cinder-volume[84696]: DEBUG cinder.utils [req-778c62be-8963-4e6d-97f1-f456b759ce16 req-00547a1c-f80b-48db-8093-70f53a331fbe tempest-AttachVolumeShelveTestJSON-1745617049 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:13bf568b-585c-4556-ab78-e9930ad50214/luns/v_13bf568b-585c-4556-ab78-e9930ad50214 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:53:25 user cinder-volume[84696]: DEBUG cinder.utils [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 22.553(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:53:25 user cinder-volume[84696]: DEBUG cinder.utils [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:53:31 user cinder-volume[84696]: DEBUG cinder.utils [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 71.293(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:53:31 user cinder-volume[84696]: DEBUG cinder.utils [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:53:34 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=9,cinder:UPDATE=2 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:53:37 user cinder-volume[84696]: DEBUG cinder.utils [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 74.744(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:53:37 user cinder-volume[84696]: DEBUG cinder.utils [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:53:37 user cinder-volume[84696]: DEBUG cinder.utils [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 74.913(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:53:37 user cinder-volume[84696]: DEBUG cinder.utils [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_2ed5b8aa-fb48-4255-b6d5-f0f84f733834/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:53:44 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=3,cinder:UPDATE=2 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:53:44 user cinder-volume[84696]: DEBUG cinder.utils [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 41.625(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:53:44 user cinder-volume[84696]: DEBUG cinder.utils [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:53:50 user cinder-volume[84696]: DEBUG cinder.utils [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 75.530(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:53:50 user cinder-volume[84696]: DEBUG cinder.utils [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_cdedb820-e1c7-4b86-8286-43b3640b3a46/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:53:50 user cinder-volume[84696]: DEBUG cinder.utils [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 75.346(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:53:50 user cinder-volume[84696]: DEBUG cinder.utils [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:53:53 user cinder-volume[84696]: DEBUG cinder.utils [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 140.078(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:53:53 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_4a2d1c00-a134-4f73-8508-86f35b58acf7/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:53:53 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (057c958c-d875-42c6-8087-149e232298b2) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 15:53:53 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (d61e31b9-a310-449f-be85-a8c6409e5f39) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:53:53 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (d61e31b9-a310-449f-be85-a8c6409e5f39) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:53:53 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Flow 'volume_create_manager' (bd64ca5c-c8d5-4e5a-ab0a-49ca63e7ecb8) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 15:53:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 15:53:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 15:53:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 15:53:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 15:53:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 15:53:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 15:53:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 15:53:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 15:53:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 15:53:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 15:53:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 15:53:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 15:53:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 15:53:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 15:53:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 15:53:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 15:53:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 15:53:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 15:53:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 15:53:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 15:53:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 15:53:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 15:53:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 15:53:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 15:53:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:53:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 15:53:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 15:53:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 15:53:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 15:53:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 15:53:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 15:53:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 15:53:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 15:53:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 15:53:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 15:53:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 15:53:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 15:53:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 15:53:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 15:53:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 15:53:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 15:53:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 15:53:53 user cinder-volume[84696]: DEBUG cinder.utils [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 139.935(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:53:53 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:53:53 user cinder-volume[84696]: INFO cinder.message.api [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Creating message record for request_id = req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a Apr 17 15:53:53 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 15:53:53 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (501412f3-e67a-416e-8e57-176fdd23dbee) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 15:53:53 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 15:53:53 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:49:15Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-AttachVolumeNegativeTest-volume-1650632398',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=4a2d1c00-a134-4f73-8508-86f35b58acf7,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='8586cc10142d4743874f43cdb0f37aba',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:51:34Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:51:34Z,use_quota=True,user_id='9fdc7c80b4654692a52c8a4c4149324f',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 15:53:53 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:49:15Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-AttachVolumeNegativeTest-volume-1650632398',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=4a2d1c00-a134-4f73-8508-86f35b58acf7,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='8586cc10142d4743874f43cdb0f37aba',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:51:34Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:51:34Z,use_quota=True,user_id='9fdc7c80b4654692a52c8a4c4149324f',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(4a2d1c00-a134-4f73-8508-86f35b58acf7),volume_id=4a2d1c00-a134-4f73-8508-86f35b58acf7,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': '4a2d1c00-a134-4f73-8508-86f35b58acf7', 'volume_name': 'volume-4a2d1c00-a134-4f73-8508-86f35b58acf7', 'volume_size': 1}} Apr 17 15:53:53 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': '4a2d1c00-a134-4f73-8508-86f35b58acf7', 'volume': {'id': '4a2d1c00-a134-4f73-8508-86f35b58acf7', '_name_id': None, 'ec2_id': None, 'user_id': '9fdc7c80b4654692a52c8a4c4149324f', 'project_id': '8586cc10142d4743874f43cdb0f37aba', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-AttachVolumeNegativeTest-volume-1650632398', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'admin_metadata': {}, 'glance_metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T15:49:15.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-4a2d1c00-a134-4f73-8508-86f35b58acf7', 'name_id': '4a2d1c00-a134-4f73-8508-86f35b58acf7', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-AttachVolumeNegativeTest-volume-1650632398', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '8586cc10142d4743874f43cdb0f37aba', 'qos_specs': None, 'reservations': ['ff09af43-7592-43c3-b526-fbe9f6237e58', '0c1bba75-a6c1-4cba-ad55-d6fa91e1347c', 'a45604df-7ba9-46e4-9aff-b47f55715a32', 'eb18d99b-bfb4-4076-a464-297431791417'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': '9fdc7c80b4654692a52c8a4c4149324f', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-AttachVolumeNegativeTest-volume-1650632398', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '8586cc10142d4743874f43cdb0f37aba', 'qos_specs': None, 'reservations': ['ff09af43-7592-43c3-b526-fbe9f6237e58', '0c1bba75-a6c1-4cba-ad55-d6fa91e1347c', 'a45604df-7ba9-46e4-9aff-b47f55715a32', 'eb18d99b-bfb4-4076-a464-297431791417'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': '9fdc7c80b4654692a52c8a4c4149324f', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': '9fdc7c80b4654692a52c8a4c4149324f', 'metadata': {}, 'qos_specs': None, 'retry': {'num_attempts': 2, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']}}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(4a2d1c00-a134-4f73-8508-86f35b58acf7),volume_id=4a2d1c00-a134-4f73-8508-86f35b58acf7,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:49:15Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-AttachVolumeNegativeTest-volume-1650632398',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=4a2d1c00-a134-4f73-8508-86f35b58acf7,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='8586cc10142d4743874f43cdb0f37aba',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:51:34Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:51:34Z,use_quota=True,user_id='9fdc7c80b4654692a52c8a4c4149324f',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 15:53:53 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:49:15Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-AttachVolumeNegativeTest-volume-1650632398',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=4a2d1c00-a134-4f73-8508-86f35b58acf7,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='8586cc10142d4743874f43cdb0f37aba',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:51:34Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:51:34Z,use_quota=True,user_id='9fdc7c80b4654692a52c8a4c4149324f',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:49:15Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-AttachVolumeNegativeTest-volume-1650632398',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=4a2d1c00-a134-4f73-8508-86f35b58acf7,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='8586cc10142d4743874f43cdb0f37aba',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:51:34Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:51:34Z,use_quota=True,user_id='9fdc7c80b4654692a52c8a4c4149324f',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 15:53:53 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 15:53:53 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 15:53:53 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:53:53 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 15:53:53 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 15:53:53 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 15:53:53 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 15:53:53 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 15:53:53 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 15:53:53 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 15:53:53 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 15:53:53 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 15:53:53 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 15:53:53 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 15:53:53 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 15:53:53 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 15:53:53 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 15:53:53 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 15:53:53 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 15:53:53 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 15:53:53 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (501412f3-e67a-416e-8e57-176fdd23dbee) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:53:53 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (501412f3-e67a-416e-8e57-176fdd23dbee) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:53:53 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (75dce16c-0230-4a17-abfe-429811260ecb) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:53:53 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (75dce16c-0230-4a17-abfe-429811260ecb) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:53:53 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (8a486fa4-d880-4c67-82ef-b013d259fe67) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:53:53 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (8a486fa4-d880-4c67-82ef-b013d259fe67) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:53:53 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (b7923afd-b289-4b11-9a5b-ae1e0e60e247) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:53:53 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Updating volume 4a2d1c00-a134-4f73-8508-86f35b58acf7 with {'scheduled_at': datetime.datetime(2023, 4, 17, 15, 53, 53, 719579), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 15:53:53 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Volume 4a2d1c00-a134-4f73-8508-86f35b58acf7: re-scheduling SchedulerAPI.create_volume attempt 2 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 15:53:53 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Volume 4a2d1c00-a134-4f73-8508-86f35b58acf7: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 15:53:53 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] deleating volume v_4a2d1c00-a134-4f73-8508-86f35b58acf7 {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 15:53:53 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] get snapshots for volume v_4a2d1c00-a134-4f73-8508-86f35b58acf7 {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 15:53:53 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_4a2d1c00-a134-4f73-8508-86f35b58acf7/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:53:53 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_4a2d1c00-a134-4f73-8508-86f35b58acf7/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:53:53 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Volume reschedule parameters: True retry: {'num_attempts': 3, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 15:53:53 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Flow 'volume_create_manager' (23959a00-2be6-447c-aa40-478af7a7f898) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:53:53 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (48e030fb-2589-48b7-90a2-28b8d68ca273) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:53:53 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (48e030fb-2589-48b7-90a2-28b8d68ca273) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:49:15Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-AttachVolumeNegativeTest-volume-1650632398',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=4a2d1c00-a134-4f73-8508-86f35b58acf7,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='8586cc10142d4743874f43cdb0f37aba',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:53:54Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:53:54Z,use_quota=True,user_id='9fdc7c80b4654692a52c8a4c4149324f',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:53:53 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (73e259f7-6c72-415f-a171-54482295bca4) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:53:53 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (73e259f7-6c72-415f-a171-54482295bca4) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:53:53 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (0371c41b-3b63-4fbd-97e5-67c42e1fef90) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:53:53 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (0371c41b-3b63-4fbd-97e5-67c42e1fef90) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': '4a2d1c00-a134-4f73-8508-86f35b58acf7', 'volume_name': 'volume-4a2d1c00-a134-4f73-8508-86f35b58acf7', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:53:53 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (174c9ef4-246f-4aca-9744-61e50dc48214) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:53:53 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (174c9ef4-246f-4aca-9744-61e50dc48214) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:53:53 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (adad000d-4416-4985-889e-7cde1dcc9a08) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:53:53 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Volume 4a2d1c00-a134-4f73-8508-86f35b58acf7: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-4a2d1c00-a134-4f73-8508-86f35b58acf7', 'volume_size': 1} Apr 17 15:53:53 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] creating volume v_4a2d1c00-a134-4f73-8508-86f35b58acf7. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 15:53:53 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] create volume {'name': 'v_4a2d1c00-a134-4f73-8508-86f35b58acf7', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 15:53:53 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:53:53 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:53:56 user cinder-volume[84696]: DEBUG cinder.utils [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.777(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:53:56 user cinder-volume[84696]: DEBUG cinder.utils [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:53:56 user cinder-volume[84696]: DEBUG cinder.utils [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 139.904(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:53:56 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_aea45c70-7c05-4146-847e-cbb777d3dfbe/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:53:56 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (86606587-6131-4612-b937-e2c0d95449ef) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 15:53:56 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (6afe5bba-b183-49c9-80a8-fd35dab8ea03) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:53:56 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (6afe5bba-b183-49c9-80a8-fd35dab8ea03) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:53:56 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Flow 'volume_create_manager' (0671f5b5-330e-4c8a-9cbf-9a77a8054324) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 15:53:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-cc50532e-b5e8-48ec-b211-ceaeda0b2b5f tempest-CreateVolumesFromImageTest-286787744 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 15:53:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 15:53:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 15:53:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 15:53:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 15:53:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 15:53:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 15:53:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 15:53:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 15:53:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 15:53:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 15:53:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 15:53:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 15:53:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 15:53:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 15:53:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 15:53:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 15:53:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 15:53:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 15:53:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 15:53:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 15:53:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 15:53:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 15:53:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 15:53:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:53:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 15:53:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 15:53:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 15:53:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 15:53:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 15:53:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 15:53:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 15:53:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 15:53:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 15:53:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 15:53:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 15:53:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 15:53:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 15:53:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 15:53:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 15:53:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 15:53:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 15:53:56 user cinder-volume[84696]: DEBUG cinder.utils [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 72.510(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:53:56 user cinder-volume[84696]: DEBUG cinder.utils [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_93a32a7c-4c6a-472a-8f91-9d34558ba232/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:53:56 user cinder-volume[84696]: DEBUG cinder.utils [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.994(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:53:56 user cinder-volume[84696]: DEBUG cinder.utils [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_4a2d1c00-a134-4f73-8508-86f35b58acf7/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:53:56 user cinder-volume[84696]: DEBUG cinder.utils [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 139.918(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:53:56 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_fbbadf16-651a-47d5-b308-a37e56aa07f2/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:53:56 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (0a72d5cb-ba2d-4dde-a6c4-034bae01c38e) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 15:53:56 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (e1d68394-eeed-45b1-b8b8-6c999c8fede3) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:53:56 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (e1d68394-eeed-45b1-b8b8-6c999c8fede3) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:53:56 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Flow 'volume_create_manager' (5d8d0a95-6117-4590-9a91-1e9b6fe32bbf) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 15:53:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-be98cecd-d344-4ab9-b5f6-c268a7041adb tempest-CreateVolumesFromImageTest-286787744 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 15:53:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 15:53:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 15:53:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 15:53:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 15:53:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 15:53:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 15:53:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 15:53:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 15:53:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 15:53:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 15:53:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 15:53:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 15:53:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 15:53:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 15:53:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 15:53:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 15:53:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 15:53:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 15:53:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 15:53:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 15:53:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 15:53:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 15:53:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 15:53:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:53:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 15:53:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 15:53:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 15:53:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 15:53:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 15:53:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 15:53:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 15:53:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 15:53:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 15:53:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 15:53:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 15:53:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 15:53:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 15:53:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 15:53:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 15:53:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 15:53:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 15:53:56 user cinder-volume[84696]: DEBUG cinder.utils [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 139.961(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:53:56 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_dfd745ed-b0c4-4e35-b77c-a911c90ce33e/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:53:56 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (68d65eb2-d76e-4966-8ef0-9d655bd4d1a2) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 15:53:56 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (41666f3e-e2c3-4d4e-a469-9a5d210a57e9) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:53:56 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (41666f3e-e2c3-4d4e-a469-9a5d210a57e9) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:53:56 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Flow 'volume_create_manager' (9ffa0202-8736-4944-9017-f7af69c224fa) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 15:53:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-9e42bf27-3da0-4fae-8992-c1ade2f352d6 tempest-CreateVolumesFromImageTest-286787744 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 15:53:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 15:53:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 15:53:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 15:53:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 15:53:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 15:53:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 15:53:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 15:53:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 15:53:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 15:53:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 15:53:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 15:53:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 15:53:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 15:53:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 15:53:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 15:53:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 15:53:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 15:53:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 15:53:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 15:53:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 15:53:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 15:53:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 15:53:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 15:53:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:53:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 15:53:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 15:53:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 15:53:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 15:53:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 15:53:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 15:53:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 15:53:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 15:53:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 15:53:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 15:53:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 15:53:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 15:53:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 15:53:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 15:53:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 15:53:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 15:53:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 15:53:59 user cinder-volume[84696]: DEBUG cinder.utils [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 139.924(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:53:59 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_13cc73df-7780-4934-8cbd-7f372b1cf57a/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:53:59 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (f20c37de-84d6-484b-8000-206b81409724) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 15:53:59 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (712112e0-a44e-40c2-b397-553a821b91f0) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:53:59 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (712112e0-a44e-40c2-b397-553a821b91f0) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:53:59 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Flow 'volume_create_manager' (f8b288af-2482-48ea-9d14-205bbbad1b17) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 15:53:59 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-9b871d6d-58e3-4523-b328-b6ffa7aca14e tempest-CreateVolumesFromImageTest-286787744 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 15:53:59 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 15:53:59 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 15:53:59 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 15:53:59 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 15:53:59 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 15:53:59 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 15:53:59 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 15:53:59 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 15:53:59 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 15:53:59 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 15:53:59 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 15:53:59 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 15:53:59 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 15:53:59 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 15:53:59 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 15:53:59 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 15:53:59 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 15:53:59 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 15:53:59 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 15:53:59 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 15:53:59 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 15:53:59 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 15:53:59 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 15:53:59 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:53:59 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 15:53:59 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 15:53:59 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 15:53:59 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 15:53:59 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 15:53:59 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 15:53:59 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 15:53:59 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 15:53:59 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 15:53:59 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 15:53:59 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 15:53:59 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 15:53:59 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 15:53:59 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 15:53:59 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 15:53:59 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 15:53:59 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 15:53:59 user cinder-volume[84696]: DEBUG cinder.utils [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.886(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:53:59 user cinder-volume[84696]: DEBUG cinder.utils [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:53:59 user cinder-volume[84696]: DEBUG cinder.utils [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 139.978(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:53:59 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_ab5889f8-b6c7-4673-a666-f4b4e5ff5fd5/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:53:59 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (d63d7778-b87e-4d8f-b05d-c0bc4e4f2689) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 15:53:59 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (5951d843-eff2-4552-a221-3aec16358929) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:53:59 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (5951d843-eff2-4552-a221-3aec16358929) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:53:59 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Flow 'volume_create_manager' (30ac04b8-ce96-4283-ae25-58b4c523da77) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 15:53:59 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-27d7b7b3-4675-424c-af48-cc2422ffc813 tempest-CreateVolumesFromImageTest-286787744 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 15:53:59 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 15:53:59 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 15:53:59 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 15:53:59 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 15:53:59 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 15:53:59 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 15:53:59 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 15:53:59 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 15:53:59 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 15:53:59 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 15:53:59 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 15:53:59 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 15:53:59 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 15:53:59 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 15:53:59 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 15:53:59 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 15:53:59 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 15:53:59 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 15:53:59 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 15:53:59 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 15:53:59 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 15:53:59 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 15:53:59 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 15:53:59 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:53:59 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 15:53:59 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 15:53:59 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 15:53:59 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 15:53:59 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 15:53:59 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 15:53:59 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 15:53:59 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 15:53:59 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 15:53:59 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 15:53:59 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 15:53:59 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 15:53:59 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 15:53:59 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 15:53:59 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 15:53:59 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 15:53:59 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 15:53:59 user cinder-volume[84696]: DEBUG cinder.utils [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.085(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:53:59 user cinder-volume[84696]: DEBUG cinder.utils [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_4a2d1c00-a134-4f73-8508-86f35b58acf7/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:54:02 user cinder-volume[84696]: DEBUG cinder.utils [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 140.050(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:54:02 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_68f814b0-1291-4a12-ae65-dc0892203d36/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:54:02 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (d8132ee2-3794-49bc-8f00-ba6d46ed7cac) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 15:54:02 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (1b0e124e-7a5a-4e51-93bb-74f1c5af474d) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:54:02 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (1b0e124e-7a5a-4e51-93bb-74f1c5af474d) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:54:02 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Flow 'volume_create_manager' (b6e14aff-6cac-440f-bb2e-0a23781024ab) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 15:54:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-d6f50cc2-dcfd-47fa-a26e-c827106b2f51 tempest-ServerRescueNegativeTestJSON-1346397943 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 15:54:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 15:54:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 15:54:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 15:54:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 15:54:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 15:54:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 15:54:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 15:54:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 15:54:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 15:54:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 15:54:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 15:54:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 15:54:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 15:54:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 15:54:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 15:54:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 15:54:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 15:54:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 15:54:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 15:54:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 15:54:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 15:54:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 15:54:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 15:54:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:54:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 15:54:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 15:54:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 15:54:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 15:54:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 15:54:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 15:54:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 15:54:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 15:54:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 15:54:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 15:54:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 15:54:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 15:54:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 15:54:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 15:54:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 15:54:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 15:54:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 15:54:04 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=9,cinder:UPDATE=3,cinder:INSERT=1 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:54:05 user cinder-volume[84696]: DEBUG cinder.utils [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 11.993(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:54:05 user cinder-volume[84696]: DEBUG cinder.utils [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:54:05 user cinder-volume[84696]: DEBUG cinder.utils [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.169(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:54:05 user cinder-volume[84696]: DEBUG cinder.utils [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_4a2d1c00-a134-4f73-8508-86f35b58acf7/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:54:09 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-778c62be-8963-4e6d-97f1-f456b759ce16 req-4c60ec5b-6ff0-4c1b-b3ee-f1f60ca15f44 tempest-AttachVolumeShelveTestJSON-1745617049 None] terminate connection for 13bf568b-585c-4556-ab78-e9930ad50214 {{(pid=84696) terminate_connection /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:1032}} Apr 17 15:54:09 user cinder-volume[84696]: INFO cinder.volume.manager [req-778c62be-8963-4e6d-97f1-f456b759ce16 req-4c60ec5b-6ff0-4c1b-b3ee-f1f60ca15f44 tempest-AttachVolumeShelveTestJSON-1745617049 None] Terminate volume connection completed successfully. Apr 17 15:54:09 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-778c62be-8963-4e6d-97f1-f456b759ce16 req-4c60ec5b-6ff0-4c1b-b3ee-f1f60ca15f44 tempest-AttachVolumeShelveTestJSON-1745617049 None] Deleting attachment 1634a7d0-3313-448e-9888-8de015f268af. {{(pid=84696) attachment_delete /opt/stack/cinder/cinder/volume/manager.py:5006}} Apr 17 15:54:09 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-778c62be-8963-4e6d-97f1-f456b759ce16 req-4c60ec5b-6ff0-4c1b-b3ee-f1f60ca15f44 tempest-AttachVolumeShelveTestJSON-1745617049 None] remove_export for volume: 13bf568b-585c-4556-ab78-e9930ad50214. {{(pid=84696) remove_export /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:718}} Apr 17 15:54:09 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-778c62be-8963-4e6d-97f1-f456b759ce16 req-4c60ec5b-6ff0-4c1b-b3ee-f1f60ca15f44 tempest-AttachVolumeShelveTestJSON-1745617049 None] remove export {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:912}} Apr 17 15:54:09 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-778c62be-8963-4e6d-97f1-f456b759ce16 req-4c60ec5b-6ff0-4c1b-b3ee-f1f60ca15f44 tempest-AttachVolumeShelveTestJSON-1745617049 None] detach volume:Volume(_name_id=None,admin_metadata=,attach_status='attached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:25:53Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-AttachVolumeShelveTestJSON-volume-1457837636',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=13bf568b-585c-4556-ab78-e9930ad50214,launched_at=2023-04-17T15:25:55Z,metadata=,migration_status=None,multiattach=False,previous_status=None,project_id='214a0f233f114d7aa22e64048a837c0b',provider_auth='CHAP cajgFQvq cTCKdDR0kw38ze',provider_geometry=None,provider_id=None,provider_location='172.16.0.220:3260,1 iqn.2010-10.org.openstack:13bf568b-585c-4556-ab78-e9930ad50214 0',replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:25:53Z,service_uuid='e628352e-c4cf-40eb-8389-cce40663026a',shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='in-use',terminated_at=None,updated_at=2023-04-17T15:31:43Z,use_quota=True,user_id='83b9d77de46143d8b9277a55f5ad4d72',volume_attachment=,volume_type=,volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c) from target:iqn.2010-10.org.openstack:13bf568b-585c-4556-ab78-e9930ad50214. {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:913}} Apr 17 15:54:09 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [req-778c62be-8963-4e6d-97f1-f456b759ce16 req-4c60ec5b-6ff0-4c1b-b3ee-f1f60ca15f44 tempest-AttachVolumeShelveTestJSON-1745617049 None] detach volume v_13bf568b-585c-4556-ab78-e9930ad50214 from target iqn.2010-10.org.openstack:13bf568b-585c-4556-ab78-e9930ad50214 {{(pid=84696) detach_target_vol /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:606}} Apr 17 15:54:09 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-778c62be-8963-4e6d-97f1-f456b759ce16 req-4c60ec5b-6ff0-4c1b-b3ee-f1f60ca15f44 tempest-AttachVolumeShelveTestJSON-1745617049 None] Sending pool request DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:13bf568b-585c-4556-ab78-e9930ad50214/luns/v_13bf568b-585c-4556-ab78-e9930ad50214 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:54:09 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-778c62be-8963-4e6d-97f1-f456b759ce16 req-4c60ec5b-6ff0-4c1b-b3ee-f1f60ca15f44 tempest-AttachVolumeShelveTestJSON-1745617049 None] Sending DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:13bf568b-585c-4556-ab78-e9930ad50214/luns/v_13bf568b-585c-4556-ab78-e9930ad50214 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:54:12 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 140.311(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:54:12 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:54:12 user cinder-volume[84696]: ERROR oslo_service.periodic_task [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Error during VolumeManager.publish_service_capabilities: TypeError: 'NoneType' object is not subscriptable Apr 17 15:54:12 user cinder-volume[84696]: ERROR oslo_service.periodic_task Traceback (most recent call last): Apr 17 15:54:12 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/usr/local/lib/python3.10/dist-packages/oslo_service/periodic_task.py", line 216, in run_periodic_tasks Apr 17 15:54:12 user cinder-volume[84696]: ERROR oslo_service.periodic_task task(self, context) Apr 17 15:54:12 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/manager.py", line 2876, in publish_service_capabilities Apr 17 15:54:12 user cinder-volume[84696]: ERROR oslo_service.periodic_task self._report_driver_status(context) Apr 17 15:54:12 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/manager.py", line 2755, in _report_driver_status Apr 17 15:54:12 user cinder-volume[84696]: ERROR oslo_service.periodic_task volume_stats = get_stats() Apr 17 15:54:12 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/usr/local/lib/python3.10/dist-packages/oslo_utils/timeutils.py", line 319, in wrapper Apr 17 15:54:12 user cinder-volume[84696]: ERROR oslo_service.periodic_task result = func(*args, **kwargs) Apr 17 15:54:12 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/manager.py", line 2753, in get_stats Apr 17 15:54:12 user cinder-volume[84696]: ERROR oslo_service.periodic_task return self.driver.get_volume_stats(refresh=True) Apr 17 15:54:12 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/driver.py", line 666, in get_volume_stats Apr 17 15:54:12 user cinder-volume[84696]: ERROR oslo_service.periodic_task self._update_volume_stats() Apr 17 15:54:12 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 726, in _update_volume_stats Apr 17 15:54:12 user cinder-volume[84696]: ERROR oslo_service.periodic_task pool_stats = self.ra.get_pool_stats() Apr 17 15:54:12 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 897, in get_pool_stats Apr 17 15:54:12 user cinder-volume[84696]: ERROR oslo_service.periodic_task if not resp["error"] and resp["code"] == 200: Apr 17 15:54:12 user cinder-volume[84696]: ERROR oslo_service.periodic_task TypeError: 'NoneType' object is not subscriptable Apr 17 15:54:12 user cinder-volume[84696]: ERROR oslo_service.periodic_task Apr 17 15:54:12 user cinder-volume[84696]: DEBUG cinder.utils [req-778c62be-8963-4e6d-97f1-f456b759ce16 req-4c60ec5b-6ff0-4c1b-b3ee-f1f60ca15f44 tempest-AttachVolumeShelveTestJSON-1745617049 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 3.069(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:54:12 user cinder-volume[84696]: DEBUG cinder.utils [req-778c62be-8963-4e6d-97f1-f456b759ce16 req-4c60ec5b-6ff0-4c1b-b3ee-f1f60ca15f44 tempest-AttachVolumeShelveTestJSON-1745617049 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:13bf568b-585c-4556-ab78-e9930ad50214/luns/v_13bf568b-585c-4556-ab78-e9930ad50214 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:54:12 user cinder-volume[84696]: WARNING oslo.service.loopingcall [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Function 'cinder.service.Service.periodic_tasks' run outlasted interval by 80.32 sec Apr 17 15:54:12 user cinder-volume[84696]: DEBUG oslo_service.periodic_task [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Running periodic task VolumeManager.publish_service_capabilities {{(pid=84696) run_periodic_tasks /usr/local/lib/python3.10/dist-packages/oslo_service/periodic_task.py:210}} Apr 17 15:54:12 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Updating volume stats {{(pid=84696) _update_volume_stats /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:724}} Apr 17 15:54:12 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Get pool Pool-0 fsprops {{(pid=84696) get_pool_stats /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:894}} Apr 17 15:54:12 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:54:12 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:54:15 user cinder-volume[84696]: DEBUG cinder.utils [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.369(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:54:15 user cinder-volume[84696]: DEBUG cinder.utils [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:54:15 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 3.065(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:54:15 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:54:15 user cinder-volume[84696]: DEBUG cinder.utils [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.546(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:54:15 user cinder-volume[84696]: DEBUG cinder.utils [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_4a2d1c00-a134-4f73-8508-86f35b58acf7/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:54:15 user cinder-volume[84696]: DEBUG cinder.utils [req-778c62be-8963-4e6d-97f1-f456b759ce16 req-4c60ec5b-6ff0-4c1b-b3ee-f1f60ca15f44 tempest-AttachVolumeShelveTestJSON-1745617049 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.141(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:54:15 user cinder-volume[84696]: DEBUG cinder.utils [req-778c62be-8963-4e6d-97f1-f456b759ce16 req-4c60ec5b-6ff0-4c1b-b3ee-f1f60ca15f44 tempest-AttachVolumeShelveTestJSON-1745617049 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:13bf568b-585c-4556-ab78-e9930ad50214/luns/v_13bf568b-585c-4556-ab78-e9930ad50214 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:54:18 user cinder-volume[84696]: DEBUG cinder.utils [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 75.577(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:54:18 user cinder-volume[84696]: DEBUG cinder.utils [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:54:18 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.137(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:54:18 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:54:21 user cinder-volume[84696]: DEBUG cinder.utils [req-778c62be-8963-4e6d-97f1-f456b759ce16 req-4c60ec5b-6ff0-4c1b-b3ee-f1f60ca15f44 tempest-AttachVolumeShelveTestJSON-1745617049 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.282(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:54:21 user cinder-volume[84696]: DEBUG cinder.utils [req-778c62be-8963-4e6d-97f1-f456b759ce16 req-4c60ec5b-6ff0-4c1b-b3ee-f1f60ca15f44 tempest-AttachVolumeShelveTestJSON-1745617049 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:13bf568b-585c-4556-ab78-e9930ad50214/luns/v_13bf568b-585c-4556-ab78-e9930ad50214 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:54:24 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=13,cinder:INSERT=1,cinder:UPDATE=3 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:54:24 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.280(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:54:24 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:54:27 user cinder-volume[84696]: DEBUG cinder.utils [req-778c62be-8963-4e6d-97f1-f456b759ce16 req-00547a1c-f80b-48db-8093-70f53a331fbe tempest-AttachVolumeShelveTestJSON-1745617049 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 138.469(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:54:27 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-778c62be-8963-4e6d-97f1-f456b759ce16 req-00547a1c-f80b-48db-8093-70f53a331fbe tempest-AttachVolumeShelveTestJSON-1745617049 None] Geting None from DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:13bf568b-585c-4556-ab78-e9930ad50214/luns/v_13bf568b-585c-4556-ab78-e9930ad50214 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:54:27 user cinder-volume[84696]: WARNING cinder.volume.manager [req-778c62be-8963-4e6d-97f1-f456b759ce16 req-00547a1c-f80b-48db-8093-70f53a331fbe tempest-AttachVolumeShelveTestJSON-1745617049 None] Failed to detach volume on the backend, ignoring failure 'NoneType' object is not subscriptable Apr 17 15:54:30 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Volume reschedule parameters: True retry: {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 15:54:30 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Flow 'volume_create_manager' (1ff833b9-00b2-40ab-bd52-605e4971f142) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:54:30 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (3abb81ed-068f-4487-b8a2-acfdc172b9ac) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:54:30 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (3abb81ed-068f-4487-b8a2-acfdc172b9ac) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:54:30Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumeRevertTests-Volume-921998137',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=b29a7592-128a-41ee-a8e2-57780dee33e7,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='9c755d3b5d8a4d9c9918b6c4744f510e',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:54:30Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:54:30Z,use_quota=True,user_id='e362cfe4a31c444b9ab44a5d60f37ad9',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:54:30 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (d5b56f26-d4f7-4e07-a5e0-44e70a74b0f8) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:54:30 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (d5b56f26-d4f7-4e07-a5e0-44e70a74b0f8) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:54:30 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (e862499e-fd0c-46c9-820a-b9939bf93664) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:54:30 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (e862499e-fd0c-46c9-820a-b9939bf93664) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': 'b29a7592-128a-41ee-a8e2-57780dee33e7', 'volume_name': 'volume-b29a7592-128a-41ee-a8e2-57780dee33e7', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:54:30 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (a6b7fb08-fd75-40df-a8b5-2bb5fdcf5764) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:54:30 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (a6b7fb08-fd75-40df-a8b5-2bb5fdcf5764) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:54:30 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (4ce941a6-a914-46eb-b2fa-a0ba34fb5993) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:54:30 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Volume b29a7592-128a-41ee-a8e2-57780dee33e7: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-b29a7592-128a-41ee-a8e2-57780dee33e7', 'volume_size': 1} Apr 17 15:54:30 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] creating volume v_b29a7592-128a-41ee-a8e2-57780dee33e7. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 15:54:30 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] create volume {'name': 'v_b29a7592-128a-41ee-a8e2-57780dee33e7', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 15:54:30 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:54:30 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:54:30 user cinder-volume[84696]: DEBUG cinder.utils [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 0.142(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:54:30 user cinder-volume[84696]: DEBUG cinder.utils [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:54:30 user cinder-volume[84696]: DEBUG cinder.utils [req-778c62be-8963-4e6d-97f1-f456b759ce16 req-4c60ec5b-6ff0-4c1b-b3ee-f1f60ca15f44 tempest-AttachVolumeShelveTestJSON-1745617049 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.499(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:54:30 user cinder-volume[84696]: DEBUG cinder.utils [req-778c62be-8963-4e6d-97f1-f456b759ce16 req-4c60ec5b-6ff0-4c1b-b3ee-f1f60ca15f44 tempest-AttachVolumeShelveTestJSON-1745617049 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:13bf568b-585c-4556-ab78-e9930ad50214/luns/v_13bf568b-585c-4556-ab78-e9930ad50214 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:54:33 user cinder-volume[84696]: DEBUG cinder.utils [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.980(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:54:33 user cinder-volume[84696]: DEBUG cinder.utils [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_4a2d1c00-a134-4f73-8508-86f35b58acf7/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:54:33 user cinder-volume[84696]: DEBUG cinder.utils [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.806(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:54:33 user cinder-volume[84696]: DEBUG cinder.utils [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:54:33 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.502(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:54:33 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:54:33 user cinder-volume[84696]: DEBUG cinder.utils [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 3.221(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:54:33 user cinder-volume[84696]: DEBUG cinder.utils [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:54:35 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Volume reschedule parameters: True retry: {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 15:54:35 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Flow 'volume_create_manager' (64bc9ae2-f8ff-4daa-8349-c4488d54df95) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:54:35 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (5806b395-60a7-4762-a8a5-a5aa0b077236) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:54:35 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (5806b395-60a7-4762-a8a5-a5aa0b077236) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:54:35Z,deleted=False,deleted_at=None,display_description='',display_name='',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=75163b78-31b2-4676-b93b-542a8ea784c1,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='540826be4baf46dba9617b2fd47821a3',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:54:35Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:54:35Z,use_quota=True,user_id='d819e3f51368413c99691d1b45bbcfb1',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:54:35 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (6f50d91f-d980-44f9-8b14-89e1707bab15) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:54:35 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (6f50d91f-d980-44f9-8b14-89e1707bab15) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:54:35 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (35168b46-28c7-40f3-aac5-bd34807f6de9) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:54:35 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (35168b46-28c7-40f3-aac5-bd34807f6de9) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'image', 'volume_id': '75163b78-31b2-4676-b93b-542a8ea784c1', 'volume_name': 'volume-75163b78-31b2-4676-b93b-542a8ea784c1', 'volume_size': 1, 'image_id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'image_location': (None, None), 'image_meta': {'name': 'cirros-0.5.2-x86_64-disk', 'disk_format': 'qcow2', 'container_format': 'bare', 'visibility': 'public', 'size': 16300544, 'virtual_size': 117440512, 'status': 'active', 'checksum': 'b874c39491a2377b8490f5f1e89761a4', 'protected': False, 'min_ram': 0, 'min_disk': 0, 'owner': 'a43c211b4ed542538799349489b1d884', 'os_hidden': False, 'os_hash_algo': 'sha512', 'os_hash_value': '6b813aa46bb90b4da216a4d19376593fa3f4fc7e617f03a92b7fe11e9a3981cbe8f0959dbebe36225e5f53dc4492341a4863cac4ed1ee0909f3fc78ef9c3e869', 'id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'created_at': datetime.datetime(2023, 4, 17, 15, 19, 53, tzinfo=datetime.timezone.utc), 'updated_at': datetime.datetime(2023, 4, 17, 15, 19, 55, tzinfo=datetime.timezone.utc), 'tags': [], 'file': '/v2/images/b3f0dec1-7758-44a8-82e0-76ada8837428/file', 'properties': {'hw_rng_model': 'virtio', 'owner_specified.openstack.md5': '', 'owner_specified.openstack.object': 'images/cirros-0.5.2-x86_64-disk', 'owner_specified.openstack.sha256': ''}}, 'image_service': }' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:54:35 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (87712448-415e-4081-a97e-1cb26456073c) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:54:35 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (87712448-415e-4081-a97e-1cb26456073c) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:54:35 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (f1597ca6-914b-4b23-84df-15dbc8a8c06a) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:54:35 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Volume 75163b78-31b2-4676-b93b-542a8ea784c1: being created as image with specification: {'status': 'creating', 'volume_name': 'volume-75163b78-31b2-4676-b93b-542a8ea784c1', 'volume_size': 1, 'image_id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'image_location': (None, None), 'image_meta': {'name': 'cirros-0.5.2-x86_64-disk', 'disk_format': 'qcow2', 'container_format': 'bare', 'visibility': 'public', 'size': 16300544, 'virtual_size': 117440512, 'status': 'active', 'checksum': 'b874c39491a2377b8490f5f1e89761a4', 'protected': False, 'min_ram': 0, 'min_disk': 0, 'owner': 'a43c211b4ed542538799349489b1d884', 'os_hidden': False, 'os_hash_algo': 'sha512', 'os_hash_value': '6b813aa46bb90b4da216a4d19376593fa3f4fc7e617f03a92b7fe11e9a3981cbe8f0959dbebe36225e5f53dc4492341a4863cac4ed1ee0909f3fc78ef9c3e869', 'id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'created_at': datetime.datetime(2023, 4, 17, 15, 19, 53, tzinfo=datetime.timezone.utc), 'updated_at': datetime.datetime(2023, 4, 17, 15, 19, 55, tzinfo=datetime.timezone.utc), 'tags': [], 'file': '/v2/images/b3f0dec1-7758-44a8-82e0-76ada8837428/file', 'properties': {'hw_rng_model': 'virtio', 'owner_specified.openstack.md5': '', 'owner_specified.openstack.object': 'images/cirros-0.5.2-x86_64-disk', 'owner_specified.openstack.sha256': ''}}, 'image_service': } Apr 17 15:54:35 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Cloning 75163b78-31b2-4676-b93b-542a8ea784c1 from image b3f0dec1-7758-44a8-82e0-76ada8837428 at location (None, None). {{(pid=84696) _create_from_image /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1071}} Apr 17 15:54:35 user cinder-volume[84696]: DEBUG cinder.coordination [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Acquiring lock "cinder-b3f0dec1-7758-44a8-82e0-76ada8837428" by "_prepare_image_cache_entry" {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:192}} Apr 17 15:54:35 user cinder-volume[84696]: DEBUG cinder.coordination [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Lock "cinder-b3f0dec1-7758-44a8-82e0-76ada8837428" acquired by "_prepare_image_cache_entry" :: waited 0.022s {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:196}} Apr 17 15:54:35 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=11,cinder:UPDATE=4 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:54:35 user cinder-volume[84696]: DEBUG cinder.image.cache [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Found image-volume cache entry: {'id': 1, 'image_id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'volume_id': '006f20ae-5306-44fe-afdf-e433f79284e8', 'host': 'user@joviandss-2#joviandss-2', 'size': 1, 'image_updated_at': datetime.datetime(2023, 4, 17, 15, 19, 55), 'last_used': datetime.datetime(2023, 4, 17, 15, 54, 35, 499899)}. {{(pid=84696) get_entry /opt/stack/cinder/cinder/image/cache.py:76}} Apr 17 15:54:35 user cinder-volume[84696]: DEBUG cinder.image.cache [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Image-volume cache entry image_update_at = 2023-04-17 15:19:55+00:00, requested image updated_at = 2023-04-17 15:19:55+00:00. {{(pid=84696) _should_update_entry /opt/stack/cinder/cinder/image/cache.py:259}} Apr 17 15:54:35 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Found cache entry for image = b3f0dec1-7758-44a8-82e0-76ada8837428 on host user@joviandss-2#joviandss-2. {{(pid=84696) _prepare_image_cache_entry /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:886}} Apr 17 15:54:35 user cinder-volume[84696]: DEBUG cinder.coordination [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Lock "cinder-b3f0dec1-7758-44a8-82e0-76ada8837428" released by "_prepare_image_cache_entry" :: held 0.016s {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:207}} Apr 17 15:54:35 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Attempting to retrieve cache entry for image = b3f0dec1-7758-44a8-82e0-76ada8837428 on host user@joviandss-2#joviandss-2. {{(pid=84696) _create_from_image_cache /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:827}} Apr 17 15:54:35 user cinder-volume[84696]: DEBUG cinder.image.cache [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Found image-volume cache entry: {'id': 1, 'image_id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'volume_id': '006f20ae-5306-44fe-afdf-e433f79284e8', 'host': 'user@joviandss-2#joviandss-2', 'size': 1, 'image_updated_at': datetime.datetime(2023, 4, 17, 15, 19, 55), 'last_used': datetime.datetime(2023, 4, 17, 15, 54, 35, 514324)}. {{(pid=84696) get_entry /opt/stack/cinder/cinder/image/cache.py:76}} Apr 17 15:54:35 user cinder-volume[84696]: DEBUG cinder.image.cache [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Image-volume cache entry image_update_at = 2023-04-17 15:19:55+00:00, requested image updated_at = 2023-04-17 15:19:55+00:00. {{(pid=84696) _should_update_entry /opt/stack/cinder/cinder/image/cache.py:259}} Apr 17 15:54:35 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Creating from source image-volume 006f20ae-5306-44fe-afdf-e433f79284e8 {{(pid=84696) _create_from_image_cache /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:841}} Apr 17 15:54:35 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] cloned volume 006f20ae-5306-44fe-afdf-e433f79284e8 to 75163b78-31b2-4676-b93b-542a8ea784c1 {{(pid=84696) create_cloned_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:521}} Apr 17 15:54:35 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] cloning v_006f20ae-5306-44fe-afdf-e433f79284e8 to v_75163b78-31b2-4676-b93b-542a8ea784c1 {{(pid=84696) _clone_object /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:456}} Apr 17 15:54:35 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] create snapshot v_75163b78-31b2-4676-b93b-542a8ea784c1 {{(pid=84696) create_snapshot /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:635}} Apr 17 15:54:35 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:54:35 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:54:36 user cinder-volume[84696]: DEBUG cinder.utils [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 136.573(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:54:36 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:54:36 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (d2e1c41b-94dd-43bf-ad21-607a2d54aabc) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 15:54:36 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 15:54:36 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:52:20Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TestVolumeBootPattern-volume-origin-952290434',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=60979eba-96b1-4cb3-bae1-8874a86cb411,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='c8a836b47d954cb9a7c1cd7c0f790426',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:52:20Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:52:20Z,use_quota=True,user_id='a80e753322224ec1a897598b5b9ed6ea',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 15:54:36 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:52:20Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TestVolumeBootPattern-volume-origin-952290434',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=60979eba-96b1-4cb3-bae1-8874a86cb411,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='c8a836b47d954cb9a7c1cd7c0f790426',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:52:20Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:52:20Z,use_quota=True,user_id='a80e753322224ec1a897598b5b9ed6ea',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=b3f0dec1-7758-44a8-82e0-76ada8837428,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(60979eba-96b1-4cb3-bae1-8874a86cb411),volume_id=60979eba-96b1-4cb3-bae1-8874a86cb411,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'image', 'volume_id': '60979eba-96b1-4cb3-bae1-8874a86cb411', 'volume_name': 'volume-60979eba-96b1-4cb3-bae1-8874a86cb411', 'volume_size': 1, 'image_id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'image_location': (None, None), 'image_meta': {'name': 'cirros-0.5.2-x86_64-disk', 'disk_format': 'qcow2', 'container_format': 'bare', 'visibility': 'public', 'size': 16300544, 'virtual_size': 117440512, 'status': 'active', 'checksum': 'b874c39491a2377b8490f5f1e89761a4', 'protected': False, 'min_ram': 0, 'min_disk': 0, 'owner': 'a43c211b4ed542538799349489b1d884', 'os_hidden': False, 'os_hash_algo': 'sha512', 'os_hash_value': '6b813aa46bb90b4da216a4d19376593fa3f4fc7e617f03a92b7fe11e9a3981cbe8f0959dbebe36225e5f53dc4492341a4863cac4ed1ee0909f3fc78ef9c3e869', 'id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'created_at': datetime.datetime(2023, 4, 17, 15, 19, 53, tzinfo=datetime.timezone.utc), 'updated_at': datetime.datetime(2023, 4, 17, 15, 19, 55, tzinfo=datetime.timezone.utc), 'tags': [], 'file': '/v2/images/b3f0dec1-7758-44a8-82e0-76ada8837428/file', 'properties': {'hw_rng_model': 'virtio', 'owner_specified.openstack.md5': '', 'owner_specified.openstack.object': 'images/cirros-0.5.2-x86_64-disk', 'owner_specified.openstack.sha256': ''}}, 'image_service': }} Apr 17 15:54:36 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'retry': {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']}, 'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'snapshot_id': None, 'source_volid': None, 'volume_id': '60979eba-96b1-4cb3-bae1-8874a86cb411', 'volume': {'id': '60979eba-96b1-4cb3-bae1-8874a86cb411', '_name_id': None, 'ec2_id': None, 'user_id': 'a80e753322224ec1a897598b5b9ed6ea', 'project_id': 'c8a836b47d954cb9a7c1cd7c0f790426', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-TestVolumeBootPattern-volume-origin-952290434', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T15:52:19.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-60979eba-96b1-4cb3-bae1-8874a86cb411', 'name_id': '60979eba-96b1-4cb3-bae1-8874a86cb411', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-TestVolumeBootPattern-volume-origin-952290434', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': 'c8a836b47d954cb9a7c1cd7c0f790426', 'qos_specs': None, 'reservations': ['665fbb3e-6435-4ad5-b020-211f010efcfd', '6df9e628-e8f9-4f25-b1cc-f04ca7f0e972', '0f4da886-833a-4bdb-a9a5-4937f5294c9a', 'f63e6f69-e592-4e3a-8f7a-79eaf32aacf0'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'a80e753322224ec1a897598b5b9ed6ea', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-TestVolumeBootPattern-volume-origin-952290434', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': 'c8a836b47d954cb9a7c1cd7c0f790426', 'qos_specs': None, 'reservations': ['665fbb3e-6435-4ad5-b020-211f010efcfd', '6df9e628-e8f9-4f25-b1cc-f04ca7f0e972', '0f4da886-833a-4bdb-a9a5-4937f5294c9a', 'f63e6f69-e592-4e3a-8f7a-79eaf32aacf0'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'a80e753322224ec1a897598b5b9ed6ea', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': 'a80e753322224ec1a897598b5b9ed6ea', 'metadata': {}, 'qos_specs': None}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=b3f0dec1-7758-44a8-82e0-76ada8837428,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(60979eba-96b1-4cb3-bae1-8874a86cb411),volume_id=60979eba-96b1-4cb3-bae1-8874a86cb411,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:52:20Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TestVolumeBootPattern-volume-origin-952290434',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=60979eba-96b1-4cb3-bae1-8874a86cb411,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='c8a836b47d954cb9a7c1cd7c0f790426',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:52:20Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:52:20Z,use_quota=True,user_id='a80e753322224ec1a897598b5b9ed6ea',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 15:54:36 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:52:20Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TestVolumeBootPattern-volume-origin-952290434',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=60979eba-96b1-4cb3-bae1-8874a86cb411,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='c8a836b47d954cb9a7c1cd7c0f790426',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:52:20Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:52:20Z,use_quota=True,user_id='a80e753322224ec1a897598b5b9ed6ea',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:52:20Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TestVolumeBootPattern-volume-origin-952290434',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=60979eba-96b1-4cb3-bae1-8874a86cb411,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='c8a836b47d954cb9a7c1cd7c0f790426',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:52:20Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:52:20Z,use_quota=True,user_id='a80e753322224ec1a897598b5b9ed6ea',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 15:54:36 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 15:54:36 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 15:54:36 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:54:36 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 15:54:36 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1237, in execute Apr 17 15:54:36 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_from_image(context, Apr 17 15:54:36 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/utils.py", line 670, in _wrapper Apr 17 15:54:36 user cinder-volume[84696]: ERROR cinder.volume.manager return r(f, *args, **kwargs) Apr 17 15:54:36 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/tenacity/__init__.py", line 423, in __call__ Apr 17 15:54:36 user cinder-volume[84696]: ERROR cinder.volume.manager do = self.iter(retry_state=retry_state) Apr 17 15:54:36 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/tenacity/__init__.py", line 360, in iter Apr 17 15:54:36 user cinder-volume[84696]: ERROR cinder.volume.manager return fut.result() Apr 17 15:54:36 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/lib/python3.10/concurrent/futures/_base.py", line 451, in result Apr 17 15:54:36 user cinder-volume[84696]: ERROR cinder.volume.manager return self.__get_result() Apr 17 15:54:36 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result Apr 17 15:54:36 user cinder-volume[84696]: ERROR cinder.volume.manager raise self._exception Apr 17 15:54:36 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/tenacity/__init__.py", line 426, in __call__ Apr 17 15:54:36 user cinder-volume[84696]: ERROR cinder.volume.manager result = fn(*args, **kwargs) Apr 17 15:54:36 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1125, in _create_from_image Apr 17 15:54:36 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_from_image_cache_or_download( Apr 17 15:54:36 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 944, in _create_from_image_cache_or_download Apr 17 15:54:36 user cinder-volume[84696]: ERROR cinder.volume.manager model_update, cloned = self._create_from_image_cache( Apr 17 15:54:36 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 843, in _create_from_image_cache Apr 17 15:54:36 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_from_source_volume( Apr 17 15:54:36 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 659, in _create_from_source_volume Apr 17 15:54:36 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self.driver.create_cloned_volume(volume, srcvol_ref) Apr 17 15:54:36 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 525, in create_cloned_volume Apr 17 15:54:36 user cinder-volume[84696]: ERROR cinder.volume.manager self._clone_object(vname, cvname) Apr 17 15:54:36 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 461, in _clone_object Apr 17 15:54:36 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_snapshot(oname, coname) Apr 17 15:54:36 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 639, in create_snapshot Apr 17 15:54:36 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201, 204): Apr 17 15:54:36 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 15:54:36 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 15:54:36 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (d2e1c41b-94dd-43bf-ad21-607a2d54aabc) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:54:36 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (d2e1c41b-94dd-43bf-ad21-607a2d54aabc) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:54:36 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (54bd49fa-777c-458d-922b-b6f17f158e01) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:54:36 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (54bd49fa-777c-458d-922b-b6f17f158e01) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:54:36 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (a80bdf04-688e-45c1-b49a-66e13ffd5cf5) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:54:36 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (a80bdf04-688e-45c1-b49a-66e13ffd5cf5) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:54:36 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (636fc593-9b63-4d21-9a0e-3c875f1e78d7) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:54:36 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Updating volume 60979eba-96b1-4cb3-bae1-8874a86cb411 with {'scheduled_at': datetime.datetime(2023, 4, 17, 15, 54, 36, 842523), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 15:54:36 user cinder-volume[84696]: DEBUG cinder.utils [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 1.313(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:54:36 user cinder-volume[84696]: DEBUG cinder.utils [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:54:36 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Volume 60979eba-96b1-4cb3-bae1-8874a86cb411: re-scheduling SchedulerAPI.create_volume attempt 1 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 15:54:36 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Volume 60979eba-96b1-4cb3-bae1-8874a86cb411: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 15:54:36 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] deleating volume v_60979eba-96b1-4cb3-bae1-8874a86cb411 {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 15:54:36 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] get snapshots for volume v_60979eba-96b1-4cb3-bae1-8874a86cb411 {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 15:54:36 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_60979eba-96b1-4cb3-bae1-8874a86cb411/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:54:36 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_60979eba-96b1-4cb3-bae1-8874a86cb411/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:54:37 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Volume reschedule parameters: True retry: {'num_attempts': 2, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 15:54:37 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Flow 'volume_create_manager' (cab37801-3312-4efa-820f-bcf7a206953d) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:54:37 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (0e29ba6a-c3ac-42a8-bfd2-4a9b9ce89b60) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:54:37 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (0e29ba6a-c3ac-42a8-bfd2-4a9b9ce89b60) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:52:20Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TestVolumeBootPattern-volume-origin-952290434',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=60979eba-96b1-4cb3-bae1-8874a86cb411,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='c8a836b47d954cb9a7c1cd7c0f790426',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:54:37Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:54:37Z,use_quota=True,user_id='a80e753322224ec1a897598b5b9ed6ea',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:54:37 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (19719143-3e90-493d-82f6-93075aeceb7d) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:54:37 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (19719143-3e90-493d-82f6-93075aeceb7d) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:54:37 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (3e755762-1c31-4abb-ba98-11ed515f284e) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:54:37 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (3e755762-1c31-4abb-ba98-11ed515f284e) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'image', 'volume_id': '60979eba-96b1-4cb3-bae1-8874a86cb411', 'volume_name': 'volume-60979eba-96b1-4cb3-bae1-8874a86cb411', 'volume_size': 1, 'image_id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'image_location': (None, None), 'image_meta': {'name': 'cirros-0.5.2-x86_64-disk', 'disk_format': 'qcow2', 'container_format': 'bare', 'visibility': 'public', 'size': 16300544, 'virtual_size': 117440512, 'status': 'active', 'checksum': 'b874c39491a2377b8490f5f1e89761a4', 'protected': False, 'min_ram': 0, 'min_disk': 0, 'owner': 'a43c211b4ed542538799349489b1d884', 'os_hidden': False, 'os_hash_algo': 'sha512', 'os_hash_value': '6b813aa46bb90b4da216a4d19376593fa3f4fc7e617f03a92b7fe11e9a3981cbe8f0959dbebe36225e5f53dc4492341a4863cac4ed1ee0909f3fc78ef9c3e869', 'id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'created_at': datetime.datetime(2023, 4, 17, 15, 19, 53, tzinfo=datetime.timezone.utc), 'updated_at': datetime.datetime(2023, 4, 17, 15, 19, 55, tzinfo=datetime.timezone.utc), 'tags': [], 'file': '/v2/images/b3f0dec1-7758-44a8-82e0-76ada8837428/file', 'properties': {'hw_rng_model': 'virtio', 'owner_specified.openstack.md5': '', 'owner_specified.openstack.object': 'images/cirros-0.5.2-x86_64-disk', 'owner_specified.openstack.sha256': ''}}, 'image_service': }' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:54:37 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (bba824b3-b9cf-4cdf-9327-015636ef3a39) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:54:37 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (bba824b3-b9cf-4cdf-9327-015636ef3a39) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:54:37 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (6181f2ad-dc29-4e8a-8c6a-1a45cc5bf4f2) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:54:37 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Volume 60979eba-96b1-4cb3-bae1-8874a86cb411: being created as image with specification: {'status': 'creating', 'volume_name': 'volume-60979eba-96b1-4cb3-bae1-8874a86cb411', 'volume_size': 1, 'image_id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'image_location': (None, None), 'image_meta': {'name': 'cirros-0.5.2-x86_64-disk', 'disk_format': 'qcow2', 'container_format': 'bare', 'visibility': 'public', 'size': 16300544, 'virtual_size': 117440512, 'status': 'active', 'checksum': 'b874c39491a2377b8490f5f1e89761a4', 'protected': False, 'min_ram': 0, 'min_disk': 0, 'owner': 'a43c211b4ed542538799349489b1d884', 'os_hidden': False, 'os_hash_algo': 'sha512', 'os_hash_value': '6b813aa46bb90b4da216a4d19376593fa3f4fc7e617f03a92b7fe11e9a3981cbe8f0959dbebe36225e5f53dc4492341a4863cac4ed1ee0909f3fc78ef9c3e869', 'id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'created_at': datetime.datetime(2023, 4, 17, 15, 19, 53, tzinfo=datetime.timezone.utc), 'updated_at': datetime.datetime(2023, 4, 17, 15, 19, 55, tzinfo=datetime.timezone.utc), 'tags': [], 'file': '/v2/images/b3f0dec1-7758-44a8-82e0-76ada8837428/file', 'properties': {'hw_rng_model': 'virtio', 'owner_specified.openstack.md5': '', 'owner_specified.openstack.object': 'images/cirros-0.5.2-x86_64-disk', 'owner_specified.openstack.sha256': ''}}, 'image_service': } Apr 17 15:54:37 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Cloning 60979eba-96b1-4cb3-bae1-8874a86cb411 from image b3f0dec1-7758-44a8-82e0-76ada8837428 at location (None, None). {{(pid=84696) _create_from_image /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1071}} Apr 17 15:54:37 user cinder-volume[84696]: DEBUG cinder.coordination [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Acquiring lock "cinder-b3f0dec1-7758-44a8-82e0-76ada8837428" by "_prepare_image_cache_entry" {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:192}} Apr 17 15:54:37 user cinder-volume[84696]: DEBUG cinder.coordination [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Lock "cinder-b3f0dec1-7758-44a8-82e0-76ada8837428" acquired by "_prepare_image_cache_entry" :: waited 0.009s {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:196}} Apr 17 15:54:37 user cinder-volume[84696]: DEBUG cinder.image.cache [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Found image-volume cache entry: {'id': 1, 'image_id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'volume_id': '006f20ae-5306-44fe-afdf-e433f79284e8', 'host': 'user@joviandss-2#joviandss-2', 'size': 1, 'image_updated_at': datetime.datetime(2023, 4, 17, 15, 19, 55), 'last_used': datetime.datetime(2023, 4, 17, 15, 54, 37, 269943)}. {{(pid=84696) get_entry /opt/stack/cinder/cinder/image/cache.py:76}} Apr 17 15:54:37 user cinder-volume[84696]: DEBUG cinder.image.cache [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Image-volume cache entry image_update_at = 2023-04-17 15:19:55+00:00, requested image updated_at = 2023-04-17 15:19:55+00:00. {{(pid=84696) _should_update_entry /opt/stack/cinder/cinder/image/cache.py:259}} Apr 17 15:54:37 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Found cache entry for image = b3f0dec1-7758-44a8-82e0-76ada8837428 on host user@joviandss-2#joviandss-2. {{(pid=84696) _prepare_image_cache_entry /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:886}} Apr 17 15:54:37 user cinder-volume[84696]: DEBUG cinder.coordination [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Lock "cinder-b3f0dec1-7758-44a8-82e0-76ada8837428" released by "_prepare_image_cache_entry" :: held 0.012s {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:207}} Apr 17 15:54:37 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Attempting to retrieve cache entry for image = b3f0dec1-7758-44a8-82e0-76ada8837428 on host user@joviandss-2#joviandss-2. {{(pid=84696) _create_from_image_cache /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:827}} Apr 17 15:54:37 user cinder-volume[84696]: DEBUG cinder.image.cache [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Found image-volume cache entry: {'id': 1, 'image_id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'volume_id': '006f20ae-5306-44fe-afdf-e433f79284e8', 'host': 'user@joviandss-2#joviandss-2', 'size': 1, 'image_updated_at': datetime.datetime(2023, 4, 17, 15, 19, 55), 'last_used': datetime.datetime(2023, 4, 17, 15, 54, 37, 281189)}. {{(pid=84696) get_entry /opt/stack/cinder/cinder/image/cache.py:76}} Apr 17 15:54:37 user cinder-volume[84696]: DEBUG cinder.image.cache [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Image-volume cache entry image_update_at = 2023-04-17 15:19:55+00:00, requested image updated_at = 2023-04-17 15:19:55+00:00. {{(pid=84696) _should_update_entry /opt/stack/cinder/cinder/image/cache.py:259}} Apr 17 15:54:37 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Creating from source image-volume 006f20ae-5306-44fe-afdf-e433f79284e8 {{(pid=84696) _create_from_image_cache /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:841}} Apr 17 15:54:37 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] cloned volume 006f20ae-5306-44fe-afdf-e433f79284e8 to 60979eba-96b1-4cb3-bae1-8874a86cb411 {{(pid=84696) create_cloned_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:521}} Apr 17 15:54:37 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] cloning v_006f20ae-5306-44fe-afdf-e433f79284e8 to v_60979eba-96b1-4cb3-bae1-8874a86cb411 {{(pid=84696) _clone_object /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:456}} Apr 17 15:54:37 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] create snapshot v_60979eba-96b1-4cb3-bae1-8874a86cb411 {{(pid=84696) create_snapshot /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:635}} Apr 17 15:54:37 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:54:37 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:54:39 user cinder-volume[84696]: DEBUG cinder.utils [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.990(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:54:39 user cinder-volume[84696]: DEBUG cinder.utils [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_60979eba-96b1-4cb3-bae1-8874a86cb411/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:54:39 user cinder-volume[84696]: DEBUG cinder.utils [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 9.362(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:54:39 user cinder-volume[84696]: DEBUG cinder.utils [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:54:39 user cinder-volume[84696]: DEBUG cinder.utils [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 4.339(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:54:39 user cinder-volume[84696]: DEBUG cinder.utils [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:54:39 user cinder-volume[84696]: DEBUG cinder.utils [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.579(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:54:39 user cinder-volume[84696]: DEBUG cinder.utils [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:54:42 user cinder-volume[84696]: DEBUG cinder.utils [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 140.056(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:54:42 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:54:42 user cinder-volume[84696]: INFO cinder.message.api [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Creating message record for request_id = req-4381eeaf-f88c-49c3-a64f-a647fc1596cd Apr 17 15:54:42 user cinder-volume[84696]: DEBUG cinder.utils [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.649(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:54:42 user cinder-volume[84696]: DEBUG cinder.utils [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:54:42 user cinder-volume[84696]: DEBUG cinder.utils [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.067(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:54:42 user cinder-volume[84696]: DEBUG cinder.utils [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_60979eba-96b1-4cb3-bae1-8874a86cb411/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:54:42 user cinder-volume[84696]: DEBUG cinder.utils [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 140.229(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:54:42 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_2ed5b8aa-fb48-4255-b6d5-f0f84f733834/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:54:42 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (f7be3d54-6bd4-45b9-8f47-361f0f9ad395) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 15:54:42 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (4df56b16-6f65-4f9b-a7d8-91eeae6b6537) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:54:42 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (4df56b16-6f65-4f9b-a7d8-91eeae6b6537) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:54:42 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Flow 'volume_create_manager' (0d0c5f0a-7eb0-439f-a14a-41128c0344ae) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 15:54:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 15:54:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 15:54:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 15:54:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 15:54:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 15:54:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 15:54:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 15:54:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 15:54:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 15:54:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 15:54:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 15:54:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 15:54:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 15:54:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 15:54:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 15:54:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 15:54:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 15:54:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 15:54:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 15:54:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 15:54:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 15:54:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 15:54:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 15:54:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 15:54:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:54:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 15:54:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 15:54:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 15:54:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 15:54:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 15:54:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 15:54:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 15:54:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 15:54:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 15:54:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 15:54:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 15:54:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 15:54:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 15:54:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 15:54:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 15:54:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 15:54:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 15:54:42 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 15:54:42 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (5f5d5412-4da8-445a-a3ec-11f8659486fe) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 15:54:42 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 15:54:42 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:50:05Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-ServerActionsTestJSON-volume-872354177',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=2ed5b8aa-fb48-4255-b6d5-f0f84f733834,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='cb5f6b5ed2fc43b69c47e1fc18646545',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:52:23Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:52:23Z,use_quota=True,user_id='638a4f535e9740fc9d810639a5eb24d7',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 15:54:42 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:50:05Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-ServerActionsTestJSON-volume-872354177',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=2ed5b8aa-fb48-4255-b6d5-f0f84f733834,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='cb5f6b5ed2fc43b69c47e1fc18646545',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:52:23Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:52:23Z,use_quota=True,user_id='638a4f535e9740fc9d810639a5eb24d7',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(2ed5b8aa-fb48-4255-b6d5-f0f84f733834),volume_id=2ed5b8aa-fb48-4255-b6d5-f0f84f733834,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': '2ed5b8aa-fb48-4255-b6d5-f0f84f733834', 'volume_name': 'volume-2ed5b8aa-fb48-4255-b6d5-f0f84f733834', 'volume_size': 1}} Apr 17 15:54:42 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': '2ed5b8aa-fb48-4255-b6d5-f0f84f733834', 'volume': {'id': '2ed5b8aa-fb48-4255-b6d5-f0f84f733834', '_name_id': None, 'ec2_id': None, 'user_id': '638a4f535e9740fc9d810639a5eb24d7', 'project_id': 'cb5f6b5ed2fc43b69c47e1fc18646545', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-ServerActionsTestJSON-volume-872354177', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'admin_metadata': {}, 'glance_metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T15:50:04.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-2ed5b8aa-fb48-4255-b6d5-f0f84f733834', 'name_id': '2ed5b8aa-fb48-4255-b6d5-f0f84f733834', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-ServerActionsTestJSON-volume-872354177', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': 'cb5f6b5ed2fc43b69c47e1fc18646545', 'qos_specs': None, 'reservations': ['6016bd49-cebc-4e3e-a82c-96aaf7390191', '75d52529-e09a-471c-b975-4ba7744d76df', 'f897feca-3a4b-4df1-9b1f-3c8ce372c8c8', 'e5402891-ace1-4d31-9e37-0068e08e9ec2'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': '638a4f535e9740fc9d810639a5eb24d7', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-ServerActionsTestJSON-volume-872354177', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': 'cb5f6b5ed2fc43b69c47e1fc18646545', 'qos_specs': None, 'reservations': ['6016bd49-cebc-4e3e-a82c-96aaf7390191', '75d52529-e09a-471c-b975-4ba7744d76df', 'f897feca-3a4b-4df1-9b1f-3c8ce372c8c8', 'e5402891-ace1-4d31-9e37-0068e08e9ec2'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': '638a4f535e9740fc9d810639a5eb24d7', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': '638a4f535e9740fc9d810639a5eb24d7', 'metadata': {}, 'qos_specs': None, 'retry': {'num_attempts': 2, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']}}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(2ed5b8aa-fb48-4255-b6d5-f0f84f733834),volume_id=2ed5b8aa-fb48-4255-b6d5-f0f84f733834,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:50:05Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-ServerActionsTestJSON-volume-872354177',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=2ed5b8aa-fb48-4255-b6d5-f0f84f733834,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='cb5f6b5ed2fc43b69c47e1fc18646545',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:52:23Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:52:23Z,use_quota=True,user_id='638a4f535e9740fc9d810639a5eb24d7',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 15:54:42 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:50:05Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-ServerActionsTestJSON-volume-872354177',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=2ed5b8aa-fb48-4255-b6d5-f0f84f733834,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='cb5f6b5ed2fc43b69c47e1fc18646545',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:52:23Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:52:23Z,use_quota=True,user_id='638a4f535e9740fc9d810639a5eb24d7',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:50:05Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-ServerActionsTestJSON-volume-872354177',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=2ed5b8aa-fb48-4255-b6d5-f0f84f733834,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='cb5f6b5ed2fc43b69c47e1fc18646545',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:52:23Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:52:23Z,use_quota=True,user_id='638a4f535e9740fc9d810639a5eb24d7',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 15:54:42 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 15:54:42 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 15:54:42 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:54:42 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 15:54:42 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 15:54:42 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 15:54:42 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 15:54:42 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 15:54:42 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 15:54:42 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 15:54:42 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 15:54:42 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 15:54:42 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 15:54:42 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 15:54:42 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 15:54:42 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 15:54:42 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 15:54:42 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 15:54:42 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 15:54:42 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 15:54:42 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (5f5d5412-4da8-445a-a3ec-11f8659486fe) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:54:42 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (5f5d5412-4da8-445a-a3ec-11f8659486fe) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:54:43 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (7c19f377-7dfb-4c80-acfa-ec8228e097b4) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:54:43 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (7c19f377-7dfb-4c80-acfa-ec8228e097b4) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:54:43 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (55141689-4da4-41ed-8815-8d05d6cb7467) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:54:43 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (55141689-4da4-41ed-8815-8d05d6cb7467) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:54:43 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (7add7189-fa1b-4ede-9dc3-2da161948fbc) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:54:43 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Updating volume 2ed5b8aa-fb48-4255-b6d5-f0f84f733834 with {'scheduled_at': datetime.datetime(2023, 4, 17, 15, 54, 43, 19386), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 15:54:43 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Volume 2ed5b8aa-fb48-4255-b6d5-f0f84f733834: re-scheduling SchedulerAPI.create_volume attempt 2 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 15:54:43 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Volume 2ed5b8aa-fb48-4255-b6d5-f0f84f733834: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 15:54:43 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] deleating volume v_2ed5b8aa-fb48-4255-b6d5-f0f84f733834 {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 15:54:43 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] get snapshots for volume v_2ed5b8aa-fb48-4255-b6d5-f0f84f733834 {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 15:54:43 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_2ed5b8aa-fb48-4255-b6d5-f0f84f733834/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:54:43 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_2ed5b8aa-fb48-4255-b6d5-f0f84f733834/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:54:43 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Volume reschedule parameters: True retry: {'num_attempts': 3, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 15:54:43 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Flow 'volume_create_manager' (822d2eb8-53f2-47c4-9634-d531c3e46104) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:54:43 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (8967ebb0-115e-4d82-baad-be32a1566790) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:54:43 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (8967ebb0-115e-4d82-baad-be32a1566790) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:50:05Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-ServerActionsTestJSON-volume-872354177',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=2ed5b8aa-fb48-4255-b6d5-f0f84f733834,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='cb5f6b5ed2fc43b69c47e1fc18646545',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:54:43Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:54:43Z,use_quota=True,user_id='638a4f535e9740fc9d810639a5eb24d7',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:54:43 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (4dc5a06a-4105-45c6-b762-a06c80fed045) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:54:43 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (4dc5a06a-4105-45c6-b762-a06c80fed045) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:54:43 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (9cbe0274-a55b-417c-a526-722649caa74e) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:54:43 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (9cbe0274-a55b-417c-a526-722649caa74e) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': '2ed5b8aa-fb48-4255-b6d5-f0f84f733834', 'volume_name': 'volume-2ed5b8aa-fb48-4255-b6d5-f0f84f733834', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:54:43 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (b719b5c1-186a-4f76-b501-9f9936ea1070) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:54:43 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (b719b5c1-186a-4f76-b501-9f9936ea1070) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:54:43 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (7a7883be-618b-4ab2-b753-55d49f84954b) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:54:43 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Volume 2ed5b8aa-fb48-4255-b6d5-f0f84f733834: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-2ed5b8aa-fb48-4255-b6d5-f0f84f733834', 'volume_size': 1} Apr 17 15:54:43 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] creating volume v_2ed5b8aa-fb48-4255-b6d5-f0f84f733834. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 15:54:43 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] create volume {'name': 'v_2ed5b8aa-fb48-4255-b6d5-f0f84f733834', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 15:54:43 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:54:43 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:54:46 user cinder-volume[84696]: DEBUG cinder.utils [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 10.478(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:54:46 user cinder-volume[84696]: DEBUG cinder.utils [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:54:46 user cinder-volume[84696]: DEBUG cinder.utils [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.969(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:54:46 user cinder-volume[84696]: DEBUG cinder.utils [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_2ed5b8aa-fb48-4255-b6d5-f0f84f733834/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:54:46 user cinder-volume[84696]: DEBUG cinder.utils [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.799(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:54:46 user cinder-volume[84696]: DEBUG cinder.utils [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:54:49 user cinder-volume[84696]: DEBUG cinder.utils [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 11.789(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:54:49 user cinder-volume[84696]: DEBUG cinder.utils [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:54:49 user cinder-volume[84696]: DEBUG cinder.utils [req-778c62be-8963-4e6d-97f1-f456b759ce16 req-4c60ec5b-6ff0-4c1b-b3ee-f1f60ca15f44 tempest-AttachVolumeShelveTestJSON-1745617049 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.931(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:54:49 user cinder-volume[84696]: DEBUG cinder.utils [req-778c62be-8963-4e6d-97f1-f456b759ce16 req-4c60ec5b-6ff0-4c1b-b3ee-f1f60ca15f44 tempest-AttachVolumeShelveTestJSON-1745617049 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:13bf568b-585c-4556-ab78-e9930ad50214/luns/v_13bf568b-585c-4556-ab78-e9930ad50214 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:54:49 user cinder-volume[84696]: DEBUG cinder.utils [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.871(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:54:49 user cinder-volume[84696]: DEBUG cinder.utils [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:54:49 user cinder-volume[84696]: DEBUG cinder.utils [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.042(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:54:49 user cinder-volume[84696]: DEBUG cinder.utils [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_2ed5b8aa-fb48-4255-b6d5-f0f84f733834/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:54:49 user cinder-volume[84696]: DEBUG cinder.utils [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 18.579(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:54:49 user cinder-volume[84696]: DEBUG cinder.utils [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:54:49 user cinder-volume[84696]: DEBUG cinder.utils [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.211(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:54:49 user cinder-volume[84696]: DEBUG cinder.utils [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_60979eba-96b1-4cb3-bae1-8874a86cb411/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:54:52 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.928(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:54:52 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:54:55 user cinder-volume[84696]: DEBUG cinder.utils [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.012(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:54:55 user cinder-volume[84696]: DEBUG cinder.utils [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:54:55 user cinder-volume[84696]: DEBUG cinder.utils [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.184(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:54:55 user cinder-volume[84696]: DEBUG cinder.utils [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_2ed5b8aa-fb48-4255-b6d5-f0f84f733834/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:54:55 user cinder-volume[84696]: DEBUG cinder.utils [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 140.017(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:54:55 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:54:55 user cinder-volume[84696]: INFO cinder.message.api [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Creating message record for request_id = req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 Apr 17 15:54:55 user cinder-volume[84696]: DEBUG cinder.utils [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 140.208(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:54:55 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_cdedb820-e1c7-4b86-8286-43b3640b3a46/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:54:55 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (18b1c95e-08dd-44a9-9e90-087b0effba30) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 15:54:55 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (8ef09eb6-13d3-4f99-a9fe-67d1e1f58f0c) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:54:55 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (8ef09eb6-13d3-4f99-a9fe-67d1e1f58f0c) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:54:55 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Flow 'volume_create_manager' (ffc87cd6-e876-4e83-a0cb-672da84ca274) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 15:54:55 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 15:54:55 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 15:54:55 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 15:54:55 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 15:54:55 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 15:54:55 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 15:54:55 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 15:54:55 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 15:54:55 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 15:54:55 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 15:54:55 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 15:54:55 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 15:54:55 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 15:54:55 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 15:54:55 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 15:54:55 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 15:54:55 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 15:54:55 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 15:54:55 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 15:54:55 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 15:54:55 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 15:54:55 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 15:54:55 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 15:54:55 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 15:54:55 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:54:55 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 15:54:55 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 15:54:55 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 15:54:55 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 15:54:55 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 15:54:55 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 15:54:55 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 15:54:55 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 15:54:55 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 15:54:55 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 15:54:55 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 15:54:55 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 15:54:55 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 15:54:55 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 15:54:55 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 15:54:55 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 15:54:55 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 15:54:55 user cinder-volume[84696]: DEBUG cinder.utils [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 19.739(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:54:55 user cinder-volume[84696]: DEBUG cinder.utils [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:54:55 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 15:54:55 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=26,cinder:UPDATE=8,cinder:INSERT=2 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:54:55 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (bf97356d-bfe5-42d1-8d21-73dc45767690) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 15:54:55 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 15:54:55 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:47:59Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-ServerRescueNegativeTestJSON-volume-7015736',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=cdedb820-e1c7-4b86-8286-43b3640b3a46,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='9324c99bd4d040bdb5a7f1db782b2783',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:52:35Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:52:35Z,use_quota=True,user_id='8e3487d6b32d46beac556617050fd664',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 15:54:55 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:47:59Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-ServerRescueNegativeTestJSON-volume-7015736',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=cdedb820-e1c7-4b86-8286-43b3640b3a46,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='9324c99bd4d040bdb5a7f1db782b2783',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:52:35Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:52:35Z,use_quota=True,user_id='8e3487d6b32d46beac556617050fd664',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(cdedb820-e1c7-4b86-8286-43b3640b3a46),volume_id=cdedb820-e1c7-4b86-8286-43b3640b3a46,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': 'cdedb820-e1c7-4b86-8286-43b3640b3a46', 'volume_name': 'volume-cdedb820-e1c7-4b86-8286-43b3640b3a46', 'volume_size': 1}} Apr 17 15:54:55 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': 'cdedb820-e1c7-4b86-8286-43b3640b3a46', 'volume': {'id': 'cdedb820-e1c7-4b86-8286-43b3640b3a46', '_name_id': None, 'ec2_id': None, 'user_id': '8e3487d6b32d46beac556617050fd664', 'project_id': '9324c99bd4d040bdb5a7f1db782b2783', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-ServerRescueNegativeTestJSON-volume-7015736', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'admin_metadata': {}, 'glance_metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T15:47:59.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-cdedb820-e1c7-4b86-8286-43b3640b3a46', 'name_id': 'cdedb820-e1c7-4b86-8286-43b3640b3a46', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-ServerRescueNegativeTestJSON-volume-7015736', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '9324c99bd4d040bdb5a7f1db782b2783', 'qos_specs': None, 'reservations': ['f9dad618-bdce-4da3-89f6-14cc03e04ddc', 'f52bc1e3-5f13-4f5b-b4df-610d006660f1', '9b6eb134-10fa-4f48-8b07-6e9470fc5aae', 'f45d6340-19c8-4636-9836-b94ea2be5f46'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': '8e3487d6b32d46beac556617050fd664', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-ServerRescueNegativeTestJSON-volume-7015736', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '9324c99bd4d040bdb5a7f1db782b2783', 'qos_specs': None, 'reservations': ['f9dad618-bdce-4da3-89f6-14cc03e04ddc', 'f52bc1e3-5f13-4f5b-b4df-610d006660f1', '9b6eb134-10fa-4f48-8b07-6e9470fc5aae', 'f45d6340-19c8-4636-9836-b94ea2be5f46'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': '8e3487d6b32d46beac556617050fd664', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': '8e3487d6b32d46beac556617050fd664', 'metadata': {}, 'qos_specs': None, 'retry': {'num_attempts': 3, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']}}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(cdedb820-e1c7-4b86-8286-43b3640b3a46),volume_id=cdedb820-e1c7-4b86-8286-43b3640b3a46,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:47:59Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-ServerRescueNegativeTestJSON-volume-7015736',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=cdedb820-e1c7-4b86-8286-43b3640b3a46,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='9324c99bd4d040bdb5a7f1db782b2783',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:52:35Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:52:35Z,use_quota=True,user_id='8e3487d6b32d46beac556617050fd664',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 15:54:55 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:47:59Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-ServerRescueNegativeTestJSON-volume-7015736',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=cdedb820-e1c7-4b86-8286-43b3640b3a46,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='9324c99bd4d040bdb5a7f1db782b2783',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:52:35Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:52:35Z,use_quota=True,user_id='8e3487d6b32d46beac556617050fd664',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:47:59Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-ServerRescueNegativeTestJSON-volume-7015736',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=cdedb820-e1c7-4b86-8286-43b3640b3a46,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='9324c99bd4d040bdb5a7f1db782b2783',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:52:35Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:52:35Z,use_quota=True,user_id='8e3487d6b32d46beac556617050fd664',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 15:54:55 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 15:54:55 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 15:54:55 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:54:55 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 15:54:55 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 15:54:55 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 15:54:55 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 15:54:55 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 15:54:55 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 15:54:55 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 15:54:55 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 15:54:55 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 15:54:55 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 15:54:55 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 15:54:55 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 15:54:55 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 15:54:55 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 15:54:55 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 15:54:55 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 15:54:55 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 15:54:55 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (bf97356d-bfe5-42d1-8d21-73dc45767690) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:54:55 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (bf97356d-bfe5-42d1-8d21-73dc45767690) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:54:55 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (820a4a8e-fefa-4df6-a3a5-d1512e6ec15b) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:54:55 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (820a4a8e-fefa-4df6-a3a5-d1512e6ec15b) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:54:55 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (9a2a11d0-78c3-4023-880b-4bb2d8f4fd7e) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:54:55 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (9a2a11d0-78c3-4023-880b-4bb2d8f4fd7e) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:54:55 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (03f8a8f7-ceef-40f0-93f7-9b4d6ce5a8ac) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:54:55 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Updating volume cdedb820-e1c7-4b86-8286-43b3640b3a46 with {'scheduled_at': datetime.datetime(2023, 4, 17, 15, 54, 55, 350626), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 15:54:55 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Volume cdedb820-e1c7-4b86-8286-43b3640b3a46: re-scheduling SchedulerAPI.create_volume attempt 3 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 15:54:55 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Volume cdedb820-e1c7-4b86-8286-43b3640b3a46: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 15:54:55 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] deleating volume v_cdedb820-e1c7-4b86-8286-43b3640b3a46 {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 15:54:55 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] get snapshots for volume v_cdedb820-e1c7-4b86-8286-43b3640b3a46 {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 15:54:55 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_cdedb820-e1c7-4b86-8286-43b3640b3a46/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:54:55 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_cdedb820-e1c7-4b86-8286-43b3640b3a46/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:54:58 user cinder-volume[84696]: DEBUG cinder.utils [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.009(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:54:58 user cinder-volume[84696]: DEBUG cinder.utils [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:54:58 user cinder-volume[84696]: DEBUG cinder.utils [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.427(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:54:58 user cinder-volume[84696]: DEBUG cinder.utils [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_60979eba-96b1-4cb3-bae1-8874a86cb411/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:54:58 user cinder-volume[84696]: DEBUG cinder.utils [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.920(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:54:58 user cinder-volume[84696]: DEBUG cinder.utils [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_cdedb820-e1c7-4b86-8286-43b3640b3a46/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:55:01 user cinder-volume[84696]: DEBUG cinder.utils [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.986(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:55:01 user cinder-volume[84696]: DEBUG cinder.utils [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_cdedb820-e1c7-4b86-8286-43b3640b3a46/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:55:01 user cinder-volume[84696]: DEBUG cinder.utils [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 137.143(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:55:01 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_93a32a7c-4c6a-472a-8f91-9d34558ba232/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:55:01 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (6caa0046-7037-45da-bf8d-6687d5b2e814) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 15:55:01 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (055bd2a4-2fc8-496f-9b11-dea155072372) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:55:01 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (055bd2a4-2fc8-496f-9b11-dea155072372) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:55:01 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Flow 'volume_create_manager' (f2fa196b-83f5-47ca-a1b7-e5232894da6d) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 15:55:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-2e8657c3-b442-4332-be65-3527a8839b0a tempest-TestVolumeBootPattern-1545678105 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 15:55:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 15:55:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 15:55:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 15:55:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 15:55:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 15:55:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 15:55:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 15:55:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 15:55:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 15:55:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 15:55:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 15:55:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 15:55:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 15:55:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 15:55:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 15:55:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 15:55:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 15:55:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 15:55:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 15:55:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 15:55:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 15:55:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 15:55:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 15:55:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:55:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 15:55:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1237, in execute Apr 17 15:55:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_from_image(context, Apr 17 15:55:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/utils.py", line 670, in _wrapper Apr 17 15:55:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return r(f, *args, **kwargs) Apr 17 15:55:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/tenacity/__init__.py", line 423, in __call__ Apr 17 15:55:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server do = self.iter(retry_state=retry_state) Apr 17 15:55:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/tenacity/__init__.py", line 360, in iter Apr 17 15:55:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return fut.result() Apr 17 15:55:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/lib/python3.10/concurrent/futures/_base.py", line 451, in result Apr 17 15:55:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self.__get_result() Apr 17 15:55:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result Apr 17 15:55:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self._exception Apr 17 15:55:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/tenacity/__init__.py", line 426, in __call__ Apr 17 15:55:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = fn(*args, **kwargs) Apr 17 15:55:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1125, in _create_from_image Apr 17 15:55:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_from_image_cache_or_download( Apr 17 15:55:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 944, in _create_from_image_cache_or_download Apr 17 15:55:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update, cloned = self._create_from_image_cache( Apr 17 15:55:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 843, in _create_from_image_cache Apr 17 15:55:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_from_source_volume( Apr 17 15:55:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 659, in _create_from_source_volume Apr 17 15:55:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self.driver.create_cloned_volume(volume, srcvol_ref) Apr 17 15:55:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 525, in create_cloned_volume Apr 17 15:55:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self._clone_object(vname, cvname) Apr 17 15:55:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 461, in _clone_object Apr 17 15:55:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_snapshot(oname, coname) Apr 17 15:55:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 639, in create_snapshot Apr 17 15:55:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201, 204): Apr 17 15:55:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 15:55:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 15:55:04 user cinder-volume[84696]: DEBUG cinder.utils [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.399(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:55:04 user cinder-volume[84696]: DEBUG cinder.utils [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_2ed5b8aa-fb48-4255-b6d5-f0f84f733834/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:55:04 user cinder-volume[84696]: DEBUG cinder.utils [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.229(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:55:04 user cinder-volume[84696]: DEBUG cinder.utils [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:55:07 user cinder-volume[84696]: DEBUG cinder.utils [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 73.593(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:55:07 user cinder-volume[84696]: DEBUG cinder.utils [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:55:07 user cinder-volume[84696]: DEBUG cinder.utils [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 37.008(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:55:07 user cinder-volume[84696]: DEBUG cinder.utils [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:55:07 user cinder-volume[84696]: DEBUG cinder.utils [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 73.770(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:55:07 user cinder-volume[84696]: DEBUG cinder.utils [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_4a2d1c00-a134-4f73-8508-86f35b58acf7/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:55:07 user cinder-volume[84696]: DEBUG cinder.utils [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.134(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:55:07 user cinder-volume[84696]: DEBUG cinder.utils [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_cdedb820-e1c7-4b86-8286-43b3640b3a46/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:55:14 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=23,cinder:INSERT=2,cinder:UPDATE=7 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:55:14 user cinder-volume[84696]: DEBUG cinder.utils [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 37.037(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:55:14 user cinder-volume[84696]: DEBUG cinder.utils [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:55:14 user cinder-volume[84696]: DEBUG cinder.utils [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 38.799(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:55:14 user cinder-volume[84696]: DEBUG cinder.utils [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:55:14 user cinder-volume[84696]: DEBUG cinder.utils [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 37.455(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:55:14 user cinder-volume[84696]: DEBUG cinder.utils [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_60979eba-96b1-4cb3-bae1-8874a86cb411/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:55:17 user cinder-volume[84696]: DEBUG cinder.utils [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 22.019(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:55:17 user cinder-volume[84696]: DEBUG cinder.utils [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_cdedb820-e1c7-4b86-8286-43b3640b3a46/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:55:17 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Volume reschedule parameters: True retry: {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 15:55:17 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Flow 'volume_create_manager' (3abee37b-a727-4bad-9026-53191fc8d5cc) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:55:17 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (10fe12ae-19f3-41f4-a6f0-8a4a10b57b9e) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:55:17 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (10fe12ae-19f3-41f4-a6f0-8a4a10b57b9e) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:55:17Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-AttachVolumeNegativeTest-volume-1944200504',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=87514055-5a0f-459f-b10b-438d7a13d39d,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='8586cc10142d4743874f43cdb0f37aba',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:55:18Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:55:18Z,use_quota=True,user_id='9fdc7c80b4654692a52c8a4c4149324f',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:55:17 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (53008c4f-ab86-4329-87aa-c158c3e48eeb) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:55:17 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (53008c4f-ab86-4329-87aa-c158c3e48eeb) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:55:17 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (2adcf970-e94e-4020-9547-0151ea0132a4) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:55:17 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (2adcf970-e94e-4020-9547-0151ea0132a4) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': '87514055-5a0f-459f-b10b-438d7a13d39d', 'volume_name': 'volume-87514055-5a0f-459f-b10b-438d7a13d39d', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:55:17 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (2c9b614b-ac85-42b0-a207-b9c9fd58695b) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:55:17 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (2c9b614b-ac85-42b0-a207-b9c9fd58695b) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:55:17 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (5d7fa8cf-ae8d-408c-8099-546ff3a2e3a7) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:55:17 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Volume 87514055-5a0f-459f-b10b-438d7a13d39d: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-87514055-5a0f-459f-b10b-438d7a13d39d', 'volume_size': 1} Apr 17 15:55:17 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] creating volume v_87514055-5a0f-459f-b10b-438d7a13d39d. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 15:55:17 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] create volume {'name': 'v_87514055-5a0f-459f-b10b-438d7a13d39d', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 15:55:17 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:55:17 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:55:20 user cinder-volume[84696]: DEBUG cinder.utils [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 37.260(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:55:20 user cinder-volume[84696]: DEBUG cinder.utils [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:55:20 user cinder-volume[84696]: DEBUG cinder.utils [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 37.432(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:55:20 user cinder-volume[84696]: DEBUG cinder.utils [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_2ed5b8aa-fb48-4255-b6d5-f0f84f733834/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:55:20 user cinder-volume[84696]: DEBUG cinder.utils [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.864(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:55:20 user cinder-volume[84696]: DEBUG cinder.utils [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:55:22 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Volume reschedule parameters: True retry: {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 15:55:22 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Flow 'volume_create_manager' (d584aec2-1bcc-4af1-ab18-d1d5c82b4806) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:55:22 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (39d93b4d-287d-4baa-ba09-b31adfd31748) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:55:22 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (39d93b4d-287d-4baa-ba09-b31adfd31748) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:55:22Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-tempest.common.compute-volume-826817831',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=14aa38ea-7cbb-48f7-a9aa-ffcc7b66faa0,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='cb5f6b5ed2fc43b69c47e1fc18646545',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:55:22Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:55:22Z,use_quota=True,user_id='638a4f535e9740fc9d810639a5eb24d7',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:55:22 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (a72c6b3f-14ea-4ea4-bded-4d9494ce46f3) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:55:22 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (a72c6b3f-14ea-4ea4-bded-4d9494ce46f3) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:55:22 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (b905c8f2-d782-4f2c-a99b-8fc50a113d6d) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:55:22 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (b905c8f2-d782-4f2c-a99b-8fc50a113d6d) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'image', 'volume_id': '14aa38ea-7cbb-48f7-a9aa-ffcc7b66faa0', 'volume_name': 'volume-14aa38ea-7cbb-48f7-a9aa-ffcc7b66faa0', 'volume_size': 1, 'image_id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'image_location': (None, None), 'image_meta': {'name': 'cirros-0.5.2-x86_64-disk', 'disk_format': 'qcow2', 'container_format': 'bare', 'visibility': 'public', 'size': 16300544, 'virtual_size': 117440512, 'status': 'active', 'checksum': 'b874c39491a2377b8490f5f1e89761a4', 'protected': False, 'min_ram': 0, 'min_disk': 0, 'owner': 'a43c211b4ed542538799349489b1d884', 'os_hidden': False, 'os_hash_algo': 'sha512', 'os_hash_value': '6b813aa46bb90b4da216a4d19376593fa3f4fc7e617f03a92b7fe11e9a3981cbe8f0959dbebe36225e5f53dc4492341a4863cac4ed1ee0909f3fc78ef9c3e869', 'id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'created_at': datetime.datetime(2023, 4, 17, 15, 19, 53, tzinfo=datetime.timezone.utc), 'updated_at': datetime.datetime(2023, 4, 17, 15, 19, 55, tzinfo=datetime.timezone.utc), 'tags': [], 'file': '/v2/images/b3f0dec1-7758-44a8-82e0-76ada8837428/file', 'properties': {'hw_rng_model': 'virtio', 'owner_specified.openstack.md5': '', 'owner_specified.openstack.object': 'images/cirros-0.5.2-x86_64-disk', 'owner_specified.openstack.sha256': ''}}, 'image_service': }' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:55:22 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (5da83909-47f3-41ae-a77c-961c4e1acdcd) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:55:22 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (5da83909-47f3-41ae-a77c-961c4e1acdcd) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:55:22 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (4276b8a4-4d38-42bb-bbe6-9f6dc75a30f0) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:55:22 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Volume 14aa38ea-7cbb-48f7-a9aa-ffcc7b66faa0: being created as image with specification: {'status': 'creating', 'volume_name': 'volume-14aa38ea-7cbb-48f7-a9aa-ffcc7b66faa0', 'volume_size': 1, 'image_id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'image_location': (None, None), 'image_meta': {'name': 'cirros-0.5.2-x86_64-disk', 'disk_format': 'qcow2', 'container_format': 'bare', 'visibility': 'public', 'size': 16300544, 'virtual_size': 117440512, 'status': 'active', 'checksum': 'b874c39491a2377b8490f5f1e89761a4', 'protected': False, 'min_ram': 0, 'min_disk': 0, 'owner': 'a43c211b4ed542538799349489b1d884', 'os_hidden': False, 'os_hash_algo': 'sha512', 'os_hash_value': '6b813aa46bb90b4da216a4d19376593fa3f4fc7e617f03a92b7fe11e9a3981cbe8f0959dbebe36225e5f53dc4492341a4863cac4ed1ee0909f3fc78ef9c3e869', 'id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'created_at': datetime.datetime(2023, 4, 17, 15, 19, 53, tzinfo=datetime.timezone.utc), 'updated_at': datetime.datetime(2023, 4, 17, 15, 19, 55, tzinfo=datetime.timezone.utc), 'tags': [], 'file': '/v2/images/b3f0dec1-7758-44a8-82e0-76ada8837428/file', 'properties': {'hw_rng_model': 'virtio', 'owner_specified.openstack.md5': '', 'owner_specified.openstack.object': 'images/cirros-0.5.2-x86_64-disk', 'owner_specified.openstack.sha256': ''}}, 'image_service': } Apr 17 15:55:22 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Cloning 14aa38ea-7cbb-48f7-a9aa-ffcc7b66faa0 from image b3f0dec1-7758-44a8-82e0-76ada8837428 at location (None, None). {{(pid=84696) _create_from_image /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1071}} Apr 17 15:55:22 user cinder-volume[84696]: DEBUG cinder.coordination [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Acquiring lock "cinder-b3f0dec1-7758-44a8-82e0-76ada8837428" by "_prepare_image_cache_entry" {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:192}} Apr 17 15:55:22 user cinder-volume[84696]: DEBUG cinder.coordination [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Lock "cinder-b3f0dec1-7758-44a8-82e0-76ada8837428" acquired by "_prepare_image_cache_entry" :: waited 0.009s {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:196}} Apr 17 15:55:22 user cinder-volume[84696]: DEBUG cinder.image.cache [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Found image-volume cache entry: {'id': 1, 'image_id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'volume_id': '006f20ae-5306-44fe-afdf-e433f79284e8', 'host': 'user@joviandss-2#joviandss-2', 'size': 1, 'image_updated_at': datetime.datetime(2023, 4, 17, 15, 19, 55), 'last_used': datetime.datetime(2023, 4, 17, 15, 55, 22, 688555)}. {{(pid=84696) get_entry /opt/stack/cinder/cinder/image/cache.py:76}} Apr 17 15:55:22 user cinder-volume[84696]: DEBUG cinder.image.cache [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Image-volume cache entry image_update_at = 2023-04-17 15:19:55+00:00, requested image updated_at = 2023-04-17 15:19:55+00:00. {{(pid=84696) _should_update_entry /opt/stack/cinder/cinder/image/cache.py:259}} Apr 17 15:55:22 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Found cache entry for image = b3f0dec1-7758-44a8-82e0-76ada8837428 on host user@joviandss-2#joviandss-2. {{(pid=84696) _prepare_image_cache_entry /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:886}} Apr 17 15:55:22 user cinder-volume[84696]: DEBUG cinder.coordination [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Lock "cinder-b3f0dec1-7758-44a8-82e0-76ada8837428" released by "_prepare_image_cache_entry" :: held 0.012s {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:207}} Apr 17 15:55:22 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Attempting to retrieve cache entry for image = b3f0dec1-7758-44a8-82e0-76ada8837428 on host user@joviandss-2#joviandss-2. {{(pid=84696) _create_from_image_cache /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:827}} Apr 17 15:55:22 user cinder-volume[84696]: DEBUG cinder.image.cache [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Found image-volume cache entry: {'id': 1, 'image_id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'volume_id': '006f20ae-5306-44fe-afdf-e433f79284e8', 'host': 'user@joviandss-2#joviandss-2', 'size': 1, 'image_updated_at': datetime.datetime(2023, 4, 17, 15, 19, 55), 'last_used': datetime.datetime(2023, 4, 17, 15, 55, 22, 699948)}. {{(pid=84696) get_entry /opt/stack/cinder/cinder/image/cache.py:76}} Apr 17 15:55:22 user cinder-volume[84696]: DEBUG cinder.image.cache [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Image-volume cache entry image_update_at = 2023-04-17 15:19:55+00:00, requested image updated_at = 2023-04-17 15:19:55+00:00. {{(pid=84696) _should_update_entry /opt/stack/cinder/cinder/image/cache.py:259}} Apr 17 15:55:22 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Creating from source image-volume 006f20ae-5306-44fe-afdf-e433f79284e8 {{(pid=84696) _create_from_image_cache /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:841}} Apr 17 15:55:22 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] cloned volume 006f20ae-5306-44fe-afdf-e433f79284e8 to 14aa38ea-7cbb-48f7-a9aa-ffcc7b66faa0 {{(pid=84696) create_cloned_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:521}} Apr 17 15:55:22 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] cloning v_006f20ae-5306-44fe-afdf-e433f79284e8 to v_14aa38ea-7cbb-48f7-a9aa-ffcc7b66faa0 {{(pid=84696) _clone_object /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:456}} Apr 17 15:55:22 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] create snapshot v_14aa38ea-7cbb-48f7-a9aa-ffcc7b66faa0 {{(pid=84696) create_snapshot /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:635}} Apr 17 15:55:22 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:55:22 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:55:23 user cinder-volume[84696]: DEBUG cinder.utils [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 140.762(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:55:23 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:55:23 user cinder-volume[84696]: INFO cinder.message.api [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Creating message record for request_id = req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 Apr 17 15:55:23 user cinder-volume[84696]: DEBUG cinder.utils [req-778c62be-8963-4e6d-97f1-f456b759ce16 req-4c60ec5b-6ff0-4c1b-b3ee-f1f60ca15f44 tempest-AttachVolumeShelveTestJSON-1745617049 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 74.398(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:55:23 user cinder-volume[84696]: DEBUG cinder.utils [req-778c62be-8963-4e6d-97f1-f456b759ce16 req-4c60ec5b-6ff0-4c1b-b3ee-f1f60ca15f44 tempest-AttachVolumeShelveTestJSON-1745617049 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:13bf568b-585c-4556-ab78-e9930ad50214/luns/v_13bf568b-585c-4556-ab78-e9930ad50214 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:55:23 user cinder-volume[84696]: DEBUG cinder.utils [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 0.839(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:55:23 user cinder-volume[84696]: DEBUG cinder.utils [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:55:23 user cinder-volume[84696]: DEBUG cinder.utils [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.941(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:55:23 user cinder-volume[84696]: DEBUG cinder.utils [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:55:23 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 15:55:23 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (c50c4515-d71e-4b74-a113-c2bfb57d5d2c) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 15:55:23 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 15:55:23 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:53:03Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TestMinimumBasicScenario-volume-1479398066',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=45e5cc09-6c97-438a-a6a2-c89e7d725ce5,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='17d1dae0ad7149bbbd160525d64c0c20',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:53:03Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:53:03Z,use_quota=True,user_id='3c407d9d494d485c9c95dc3981284fba',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 15:55:23 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:53:03Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TestMinimumBasicScenario-volume-1479398066',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=45e5cc09-6c97-438a-a6a2-c89e7d725ce5,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='17d1dae0ad7149bbbd160525d64c0c20',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:53:03Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:53:03Z,use_quota=True,user_id='3c407d9d494d485c9c95dc3981284fba',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(45e5cc09-6c97-438a-a6a2-c89e7d725ce5),volume_id=45e5cc09-6c97-438a-a6a2-c89e7d725ce5,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': '45e5cc09-6c97-438a-a6a2-c89e7d725ce5', 'volume_name': 'volume-45e5cc09-6c97-438a-a6a2-c89e7d725ce5', 'volume_size': 1}} Apr 17 15:55:23 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'retry': {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']}, 'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': '45e5cc09-6c97-438a-a6a2-c89e7d725ce5', 'volume': {'id': '45e5cc09-6c97-438a-a6a2-c89e7d725ce5', '_name_id': None, 'ec2_id': None, 'user_id': '3c407d9d494d485c9c95dc3981284fba', 'project_id': '17d1dae0ad7149bbbd160525d64c0c20', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-TestMinimumBasicScenario-volume-1479398066', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T15:53:02.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-45e5cc09-6c97-438a-a6a2-c89e7d725ce5', 'name_id': '45e5cc09-6c97-438a-a6a2-c89e7d725ce5', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-TestMinimumBasicScenario-volume-1479398066', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '17d1dae0ad7149bbbd160525d64c0c20', 'qos_specs': None, 'reservations': ['a105b386-4e4e-4dca-ae87-4d1b3715ef82', '1ede682d-d17c-4d21-8d44-1ad1c081125a', 'df0aa6c2-bf1a-436e-9684-88c24c928b51', '6d77c826-2ba0-4ea1-b791-09b8cf39c978'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': '3c407d9d494d485c9c95dc3981284fba', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-TestMinimumBasicScenario-volume-1479398066', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '17d1dae0ad7149bbbd160525d64c0c20', 'qos_specs': None, 'reservations': ['a105b386-4e4e-4dca-ae87-4d1b3715ef82', '1ede682d-d17c-4d21-8d44-1ad1c081125a', 'df0aa6c2-bf1a-436e-9684-88c24c928b51', '6d77c826-2ba0-4ea1-b791-09b8cf39c978'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': '3c407d9d494d485c9c95dc3981284fba', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': '3c407d9d494d485c9c95dc3981284fba', 'metadata': {}, 'qos_specs': None}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(45e5cc09-6c97-438a-a6a2-c89e7d725ce5),volume_id=45e5cc09-6c97-438a-a6a2-c89e7d725ce5,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:53:03Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TestMinimumBasicScenario-volume-1479398066',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=45e5cc09-6c97-438a-a6a2-c89e7d725ce5,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='17d1dae0ad7149bbbd160525d64c0c20',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:53:03Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:53:03Z,use_quota=True,user_id='3c407d9d494d485c9c95dc3981284fba',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 15:55:23 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:53:03Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TestMinimumBasicScenario-volume-1479398066',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=45e5cc09-6c97-438a-a6a2-c89e7d725ce5,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='17d1dae0ad7149bbbd160525d64c0c20',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:53:03Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:53:03Z,use_quota=True,user_id='3c407d9d494d485c9c95dc3981284fba',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:53:03Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TestMinimumBasicScenario-volume-1479398066',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=45e5cc09-6c97-438a-a6a2-c89e7d725ce5,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='17d1dae0ad7149bbbd160525d64c0c20',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:53:03Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:53:03Z,use_quota=True,user_id='3c407d9d494d485c9c95dc3981284fba',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 15:55:23 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 15:55:23 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 15:55:23 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:55:23 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 15:55:23 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 15:55:23 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 15:55:23 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 15:55:23 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 15:55:23 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 15:55:23 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 15:55:23 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 15:55:23 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 15:55:23 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 15:55:23 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 15:55:23 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 15:55:23 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 15:55:23 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 15:55:23 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 15:55:23 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 15:55:23 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 15:55:23 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (c50c4515-d71e-4b74-a113-c2bfb57d5d2c) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:55:23 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (c50c4515-d71e-4b74-a113-c2bfb57d5d2c) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:55:23 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (e4c4e732-35d7-45ec-811e-7c5c7f5fa748) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:55:23 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (e4c4e732-35d7-45ec-811e-7c5c7f5fa748) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:55:23 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (3657c247-1ac6-428f-924c-4b1012051d37) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:55:23 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (3657c247-1ac6-428f-924c-4b1012051d37) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:55:23 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (9319b2f5-7d61-40d7-8b78-e1d9ddfb790f) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:55:23 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Updating volume 45e5cc09-6c97-438a-a6a2-c89e7d725ce5 with {'scheduled_at': datetime.datetime(2023, 4, 17, 15, 55, 23, 613938), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 15:55:23 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Volume 45e5cc09-6c97-438a-a6a2-c89e7d725ce5: re-scheduling SchedulerAPI.create_volume attempt 1 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 15:55:23 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Volume 45e5cc09-6c97-438a-a6a2-c89e7d725ce5: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 15:55:23 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] deleating volume v_45e5cc09-6c97-438a-a6a2-c89e7d725ce5 {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 15:55:23 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] get snapshots for volume v_45e5cc09-6c97-438a-a6a2-c89e7d725ce5 {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 15:55:23 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_45e5cc09-6c97-438a-a6a2-c89e7d725ce5/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:55:23 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_45e5cc09-6c97-438a-a6a2-c89e7d725ce5/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:55:23 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Volume reschedule parameters: True retry: {'num_attempts': 2, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 15:55:23 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Flow 'volume_create_manager' (cb5e75a4-dba2-45e9-85bb-081cc2ee2604) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:55:23 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (a276e14b-58c4-46c3-a804-e11638a58ecc) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:55:23 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (a276e14b-58c4-46c3-a804-e11638a58ecc) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:53:03Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TestMinimumBasicScenario-volume-1479398066',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=45e5cc09-6c97-438a-a6a2-c89e7d725ce5,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='17d1dae0ad7149bbbd160525d64c0c20',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:55:24Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:55:24Z,use_quota=True,user_id='3c407d9d494d485c9c95dc3981284fba',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:55:23 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (cfe42e67-3aa3-4197-8aa9-61ca5abb7fb3) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:55:23 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (cfe42e67-3aa3-4197-8aa9-61ca5abb7fb3) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:55:23 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (27053387-0a12-4eea-b20b-8f6a8dc30bdc) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:55:23 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (27053387-0a12-4eea-b20b-8f6a8dc30bdc) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': '45e5cc09-6c97-438a-a6a2-c89e7d725ce5', 'volume_name': 'volume-45e5cc09-6c97-438a-a6a2-c89e7d725ce5', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:55:23 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (e14e2361-1d63-4436-bc3b-dd448f9dfac5) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:55:23 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (e14e2361-1d63-4436-bc3b-dd448f9dfac5) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:55:23 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (d978d8da-9a7d-449f-b142-a305528cf7d3) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:55:23 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Volume 45e5cc09-6c97-438a-a6a2-c89e7d725ce5: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-45e5cc09-6c97-438a-a6a2-c89e7d725ce5', 'volume_size': 1} Apr 17 15:55:23 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] creating volume v_45e5cc09-6c97-438a-a6a2-c89e7d725ce5. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 15:55:23 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] create volume {'name': 'v_45e5cc09-6c97-438a-a6a2-c89e7d725ce5', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 15:55:23 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:55:23 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:55:26 user cinder-volume[84696]: DEBUG cinder.utils [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.791(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:55:26 user cinder-volume[84696]: DEBUG cinder.utils [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:55:26 user cinder-volume[84696]: DEBUG cinder.utils [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.980(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:55:26 user cinder-volume[84696]: DEBUG cinder.utils [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_45e5cc09-6c97-438a-a6a2-c89e7d725ce5/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:55:26 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 74.394(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:55:26 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:55:26 user cinder-volume[84696]: DEBUG cinder.utils [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 3.909(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:55:26 user cinder-volume[84696]: DEBUG cinder.utils [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:55:29 user cinder-volume[84696]: DEBUG cinder.utils [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.078(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:55:29 user cinder-volume[84696]: DEBUG cinder.utils [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:55:29 user cinder-volume[84696]: DEBUG cinder.utils [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.866(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:55:29 user cinder-volume[84696]: DEBUG cinder.utils [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:55:29 user cinder-volume[84696]: DEBUG cinder.utils [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.056(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:55:29 user cinder-volume[84696]: DEBUG cinder.utils [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_45e5cc09-6c97-438a-a6a2-c89e7d725ce5/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:55:32 user cinder-volume[84696]: DEBUG cinder.utils [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 10.050(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:55:32 user cinder-volume[84696]: DEBUG cinder.utils [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:55:34 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:INSERT=2,cinder:UPDATE=4,cinder:SELECT=19 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:55:35 user cinder-volume[84696]: DEBUG cinder.utils [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 40.450(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:55:35 user cinder-volume[84696]: DEBUG cinder.utils [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_cdedb820-e1c7-4b86-8286-43b3640b3a46/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:55:35 user cinder-volume[84696]: DEBUG cinder.utils [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.196(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:55:35 user cinder-volume[84696]: DEBUG cinder.utils [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_45e5cc09-6c97-438a-a6a2-c89e7d725ce5/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:55:35 user cinder-volume[84696]: DEBUG cinder.utils [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.009(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:55:35 user cinder-volume[84696]: DEBUG cinder.utils [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:55:38 user cinder-volume[84696]: DEBUG cinder.utils [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.298(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:55:38 user cinder-volume[84696]: DEBUG cinder.utils [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:55:41 user cinder-volume[84696]: DEBUG cinder.utils [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 71.470(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:55:41 user cinder-volume[84696]: DEBUG cinder.utils [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:55:41 user cinder-volume[84696]: DEBUG cinder.utils [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 19.267(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:55:41 user cinder-volume[84696]: DEBUG cinder.utils [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:55:45 user cinder-volume[84696]: DEBUG cinder.utils [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.411(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:55:45 user cinder-volume[84696]: DEBUG cinder.utils [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_45e5cc09-6c97-438a-a6a2-c89e7d725ce5/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:55:45 user cinder-volume[84696]: DEBUG cinder.utils [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.224(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:55:45 user cinder-volume[84696]: DEBUG cinder.utils [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:55:48 user cinder-volume[84696]: DEBUG cinder.utils [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 70.829(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:55:48 user cinder-volume[84696]: DEBUG cinder.utils [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:55:48 user cinder-volume[84696]: DEBUG cinder.utils [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 71.247(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:55:48 user cinder-volume[84696]: DEBUG cinder.utils [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_60979eba-96b1-4cb3-bae1-8874a86cb411/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:55:48 user cinder-volume[84696]: DEBUG cinder.utils [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 72.593(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:55:48 user cinder-volume[84696]: DEBUG cinder.utils [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:55:54 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=1 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:55:54 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=19,cinder:UPDATE=8,cinder:INSERT=1 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:55:54 user cinder-volume[84696]: DEBUG cinder.utils [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 71.372(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:55:54 user cinder-volume[84696]: DEBUG cinder.utils [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:55:54 user cinder-volume[84696]: DEBUG cinder.utils [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 71.545(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:55:54 user cinder-volume[84696]: DEBUG cinder.utils [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_2ed5b8aa-fb48-4255-b6d5-f0f84f733834/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:55:57 user cinder-volume[84696]: DEBUG cinder.utils [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 40.046(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:55:57 user cinder-volume[84696]: DEBUG cinder.utils [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:56:00 user cinder-volume[84696]: DEBUG cinder.utils [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 38.022(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:56:00 user cinder-volume[84696]: DEBUG cinder.utils [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:56:03 user cinder-volume[84696]: DEBUG cinder.utils [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.975(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:56:03 user cinder-volume[84696]: DEBUG cinder.utils [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:56:03 user cinder-volume[84696]: DEBUG cinder.utils [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 40.164(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:56:03 user cinder-volume[84696]: DEBUG cinder.utils [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_45e5cc09-6c97-438a-a6a2-c89e7d725ce5/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:56:09 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-778c62be-8963-4e6d-97f1-f456b759ce16 req-6d72b587-594c-4309-b424-7292ebb80925 tempest-AttachVolumeShelveTestJSON-1745617049 None] terminate connection for 13bf568b-585c-4556-ab78-e9930ad50214 {{(pid=84696) terminate_connection /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:1032}} Apr 17 15:56:09 user cinder-volume[84696]: INFO cinder.volume.manager [req-778c62be-8963-4e6d-97f1-f456b759ce16 req-6d72b587-594c-4309-b424-7292ebb80925 tempest-AttachVolumeShelveTestJSON-1745617049 None] Terminate volume connection completed successfully. Apr 17 15:56:09 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-778c62be-8963-4e6d-97f1-f456b759ce16 req-6d72b587-594c-4309-b424-7292ebb80925 tempest-AttachVolumeShelveTestJSON-1745617049 None] Deleting attachment 1634a7d0-3313-448e-9888-8de015f268af. {{(pid=84696) attachment_delete /opt/stack/cinder/cinder/volume/manager.py:5006}} Apr 17 15:56:09 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-778c62be-8963-4e6d-97f1-f456b759ce16 req-6d72b587-594c-4309-b424-7292ebb80925 tempest-AttachVolumeShelveTestJSON-1745617049 None] remove_export for volume: 13bf568b-585c-4556-ab78-e9930ad50214. {{(pid=84696) remove_export /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:718}} Apr 17 15:56:09 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-778c62be-8963-4e6d-97f1-f456b759ce16 req-6d72b587-594c-4309-b424-7292ebb80925 tempest-AttachVolumeShelveTestJSON-1745617049 None] remove export {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:912}} Apr 17 15:56:09 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-778c62be-8963-4e6d-97f1-f456b759ce16 req-6d72b587-594c-4309-b424-7292ebb80925 tempest-AttachVolumeShelveTestJSON-1745617049 None] detach volume:Volume(_name_id=None,admin_metadata=,attach_status='attached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:25:53Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-AttachVolumeShelveTestJSON-volume-1457837636',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=13bf568b-585c-4556-ab78-e9930ad50214,launched_at=2023-04-17T15:25:55Z,metadata=,migration_status=None,multiattach=False,previous_status=None,project_id='214a0f233f114d7aa22e64048a837c0b',provider_auth='CHAP cajgFQvq cTCKdDR0kw38ze',provider_geometry=None,provider_id=None,provider_location='172.16.0.220:3260,1 iqn.2010-10.org.openstack:13bf568b-585c-4556-ab78-e9930ad50214 0',replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:25:53Z,service_uuid='e628352e-c4cf-40eb-8389-cce40663026a',shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='in-use',terminated_at=None,updated_at=2023-04-17T15:31:43Z,use_quota=True,user_id='83b9d77de46143d8b9277a55f5ad4d72',volume_attachment=,volume_type=,volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c) from target:iqn.2010-10.org.openstack:13bf568b-585c-4556-ab78-e9930ad50214. {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:913}} Apr 17 15:56:09 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [req-778c62be-8963-4e6d-97f1-f456b759ce16 req-6d72b587-594c-4309-b424-7292ebb80925 tempest-AttachVolumeShelveTestJSON-1745617049 None] detach volume v_13bf568b-585c-4556-ab78-e9930ad50214 from target iqn.2010-10.org.openstack:13bf568b-585c-4556-ab78-e9930ad50214 {{(pid=84696) detach_target_vol /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:606}} Apr 17 15:56:09 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-778c62be-8963-4e6d-97f1-f456b759ce16 req-6d72b587-594c-4309-b424-7292ebb80925 tempest-AttachVolumeShelveTestJSON-1745617049 None] Sending pool request DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:13bf568b-585c-4556-ab78-e9930ad50214/luns/v_13bf568b-585c-4556-ab78-e9930ad50214 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:56:09 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-778c62be-8963-4e6d-97f1-f456b759ce16 req-6d72b587-594c-4309-b424-7292ebb80925 tempest-AttachVolumeShelveTestJSON-1745617049 None] Sending DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:13bf568b-585c-4556-ab78-e9930ad50214/luns/v_13bf568b-585c-4556-ab78-e9930ad50214 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:56:09 user cinder-volume[84696]: DEBUG cinder.utils [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 74.562(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:56:09 user cinder-volume[84696]: DEBUG cinder.utils [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_cdedb820-e1c7-4b86-8286-43b3640b3a46/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:56:09 user cinder-volume[84696]: DEBUG cinder.utils [req-778c62be-8963-4e6d-97f1-f456b759ce16 req-6d72b587-594c-4309-b424-7292ebb80925 tempest-AttachVolumeShelveTestJSON-1745617049 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 0.747(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:56:09 user cinder-volume[84696]: DEBUG cinder.utils [req-778c62be-8963-4e6d-97f1-f456b759ce16 req-6d72b587-594c-4309-b424-7292ebb80925 tempest-AttachVolumeShelveTestJSON-1745617049 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:13bf568b-585c-4556-ab78-e9930ad50214/luns/v_13bf568b-585c-4556-ab78-e9930ad50214 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:56:13 user cinder-volume[84696]: DEBUG cinder.utils [req-778c62be-8963-4e6d-97f1-f456b759ce16 req-6d72b587-594c-4309-b424-7292ebb80925 tempest-AttachVolumeShelveTestJSON-1745617049 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 3.816(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:56:13 user cinder-volume[84696]: DEBUG cinder.utils [req-778c62be-8963-4e6d-97f1-f456b759ce16 req-6d72b587-594c-4309-b424-7292ebb80925 tempest-AttachVolumeShelveTestJSON-1745617049 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:13bf568b-585c-4556-ab78-e9930ad50214/luns/v_13bf568b-585c-4556-ab78-e9930ad50214 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:56:13 user cinder-volume[84696]: DEBUG cinder.utils [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 139.275(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:56:13 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_4a2d1c00-a134-4f73-8508-86f35b58acf7/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:56:13 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (b7923afd-b289-4b11-9a5b-ae1e0e60e247) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 15:56:13 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (27632b54-03ef-4088-958f-dcf41f8a56ad) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:56:13 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (27632b54-03ef-4088-958f-dcf41f8a56ad) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:56:13 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Flow 'volume_create_manager' (d2e0780c-751d-4669-a6a1-af8229f83db7) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 15:56:13 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 15:56:13 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 15:56:13 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 15:56:13 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 15:56:13 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 15:56:13 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 15:56:13 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 15:56:13 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 15:56:13 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 15:56:13 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 15:56:13 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 15:56:13 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 15:56:13 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 15:56:13 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 15:56:13 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 15:56:13 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 15:56:13 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 15:56:13 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 15:56:13 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 15:56:13 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 15:56:13 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 15:56:13 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 15:56:13 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 15:56:13 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 15:56:13 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:56:13 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 15:56:13 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 15:56:13 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 15:56:13 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 15:56:13 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 15:56:13 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 15:56:13 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 15:56:13 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 15:56:13 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 15:56:13 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 15:56:13 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 15:56:13 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 15:56:13 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 15:56:13 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 15:56:13 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 15:56:13 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 15:56:13 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 15:56:13 user cinder-volume[84696]: DEBUG cinder.utils [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 139.120(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:56:13 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:56:13 user cinder-volume[84696]: INFO cinder.message.api [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Creating message record for request_id = req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a Apr 17 15:56:13 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 15:56:13 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (adad000d-4416-4985-889e-7cde1dcc9a08) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 15:56:13 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 15:56:13 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:49:15Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-AttachVolumeNegativeTest-volume-1650632398',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=4a2d1c00-a134-4f73-8508-86f35b58acf7,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='8586cc10142d4743874f43cdb0f37aba',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:53:54Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:53:54Z,use_quota=True,user_id='9fdc7c80b4654692a52c8a4c4149324f',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 15:56:13 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:49:15Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-AttachVolumeNegativeTest-volume-1650632398',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=4a2d1c00-a134-4f73-8508-86f35b58acf7,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='8586cc10142d4743874f43cdb0f37aba',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:53:54Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:53:54Z,use_quota=True,user_id='9fdc7c80b4654692a52c8a4c4149324f',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(4a2d1c00-a134-4f73-8508-86f35b58acf7),volume_id=4a2d1c00-a134-4f73-8508-86f35b58acf7,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': '4a2d1c00-a134-4f73-8508-86f35b58acf7', 'volume_name': 'volume-4a2d1c00-a134-4f73-8508-86f35b58acf7', 'volume_size': 1}} Apr 17 15:56:13 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': '4a2d1c00-a134-4f73-8508-86f35b58acf7', 'volume': {'id': '4a2d1c00-a134-4f73-8508-86f35b58acf7', '_name_id': None, 'ec2_id': None, 'user_id': '9fdc7c80b4654692a52c8a4c4149324f', 'project_id': '8586cc10142d4743874f43cdb0f37aba', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-AttachVolumeNegativeTest-volume-1650632398', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'admin_metadata': {}, 'glance_metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T15:49:15.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-4a2d1c00-a134-4f73-8508-86f35b58acf7', 'name_id': '4a2d1c00-a134-4f73-8508-86f35b58acf7', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-AttachVolumeNegativeTest-volume-1650632398', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '8586cc10142d4743874f43cdb0f37aba', 'qos_specs': None, 'reservations': ['ff09af43-7592-43c3-b526-fbe9f6237e58', '0c1bba75-a6c1-4cba-ad55-d6fa91e1347c', 'a45604df-7ba9-46e4-9aff-b47f55715a32', 'eb18d99b-bfb4-4076-a464-297431791417'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': '9fdc7c80b4654692a52c8a4c4149324f', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-AttachVolumeNegativeTest-volume-1650632398', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '8586cc10142d4743874f43cdb0f37aba', 'qos_specs': None, 'reservations': ['ff09af43-7592-43c3-b526-fbe9f6237e58', '0c1bba75-a6c1-4cba-ad55-d6fa91e1347c', 'a45604df-7ba9-46e4-9aff-b47f55715a32', 'eb18d99b-bfb4-4076-a464-297431791417'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': '9fdc7c80b4654692a52c8a4c4149324f', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': '9fdc7c80b4654692a52c8a4c4149324f', 'metadata': {}, 'qos_specs': None, 'retry': {'num_attempts': 3, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']}}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(4a2d1c00-a134-4f73-8508-86f35b58acf7),volume_id=4a2d1c00-a134-4f73-8508-86f35b58acf7,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:49:15Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-AttachVolumeNegativeTest-volume-1650632398',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=4a2d1c00-a134-4f73-8508-86f35b58acf7,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='8586cc10142d4743874f43cdb0f37aba',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:53:54Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:53:54Z,use_quota=True,user_id='9fdc7c80b4654692a52c8a4c4149324f',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 15:56:13 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:49:15Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-AttachVolumeNegativeTest-volume-1650632398',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=4a2d1c00-a134-4f73-8508-86f35b58acf7,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='8586cc10142d4743874f43cdb0f37aba',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:53:54Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:53:54Z,use_quota=True,user_id='9fdc7c80b4654692a52c8a4c4149324f',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:49:15Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-AttachVolumeNegativeTest-volume-1650632398',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=4a2d1c00-a134-4f73-8508-86f35b58acf7,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='8586cc10142d4743874f43cdb0f37aba',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:53:54Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:53:54Z,use_quota=True,user_id='9fdc7c80b4654692a52c8a4c4149324f',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 15:56:13 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 15:56:13 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 15:56:13 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:56:13 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 15:56:13 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 15:56:13 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 15:56:13 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 15:56:13 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 15:56:13 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 15:56:13 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 15:56:13 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 15:56:13 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 15:56:13 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 15:56:13 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 15:56:13 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 15:56:13 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 15:56:13 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 15:56:13 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 15:56:13 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 15:56:13 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 15:56:13 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (adad000d-4416-4985-889e-7cde1dcc9a08) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:56:13 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (adad000d-4416-4985-889e-7cde1dcc9a08) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:56:13 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (174c9ef4-246f-4aca-9744-61e50dc48214) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:56:13 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (174c9ef4-246f-4aca-9744-61e50dc48214) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:56:13 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (0371c41b-3b63-4fbd-97e5-67c42e1fef90) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:56:13 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (0371c41b-3b63-4fbd-97e5-67c42e1fef90) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:56:13 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (73e259f7-6c72-415f-a171-54482295bca4) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:56:13 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Updating volume 4a2d1c00-a134-4f73-8508-86f35b58acf7 with {'scheduled_at': datetime.datetime(2023, 4, 17, 15, 56, 13, 106944), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 15:56:13 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Volume 4a2d1c00-a134-4f73-8508-86f35b58acf7: re-scheduling SchedulerAPI.create_volume attempt 3 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 15:56:13 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Volume 4a2d1c00-a134-4f73-8508-86f35b58acf7: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 15:56:13 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] deleating volume v_4a2d1c00-a134-4f73-8508-86f35b58acf7 {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 15:56:13 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] get snapshots for volume v_4a2d1c00-a134-4f73-8508-86f35b58acf7 {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 15:56:13 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_4a2d1c00-a134-4f73-8508-86f35b58acf7/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:56:13 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_4a2d1c00-a134-4f73-8508-86f35b58acf7/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:56:16 user cinder-volume[84696]: DEBUG cinder.utils [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.959(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:56:16 user cinder-volume[84696]: DEBUG cinder.utils [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_4a2d1c00-a134-4f73-8508-86f35b58acf7/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:56:19 user cinder-volume[84696]: DEBUG cinder.utils [req-778c62be-8963-4e6d-97f1-f456b759ce16 req-6d72b587-594c-4309-b424-7292ebb80925 tempest-AttachVolumeShelveTestJSON-1745617049 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 9.961(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:56:19 user cinder-volume[84696]: DEBUG cinder.utils [req-778c62be-8963-4e6d-97f1-f456b759ce16 req-6d72b587-594c-4309-b424-7292ebb80925 tempest-AttachVolumeShelveTestJSON-1745617049 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:13bf568b-585c-4556-ab78-e9930ad50214/luns/v_13bf568b-585c-4556-ab78-e9930ad50214 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:56:19 user cinder-volume[84696]: DEBUG cinder.utils [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.034(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:56:19 user cinder-volume[84696]: DEBUG cinder.utils [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_4a2d1c00-a134-4f73-8508-86f35b58acf7/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:56:25 user cinder-volume[84696]: DEBUG cinder.utils [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.463(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:56:25 user cinder-volume[84696]: DEBUG cinder.utils [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_4a2d1c00-a134-4f73-8508-86f35b58acf7/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:56:25 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=10,cinder:INSERT=1,cinder:UPDATE=2 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:56:25 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=11,cinder:UPDATE=4,cinder:INSERT=1 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:56:25 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Volume reschedule parameters: True retry: {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 15:56:25 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Flow 'volume_create_manager' (21b8dbdb-80d0-4b1f-9c54-7ac019e3531c) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:56:25 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (d4d9be2e-5bd2-49db-9e90-cc6fbb3cef06) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:56:25 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (d4d9be2e-5bd2-49db-9e90-cc6fbb3cef06) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:56:26Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumeRetypeWithoutMigrationTest-Volume-636058295',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=bff6fdbb-a1fb-4b48-bca8-2f7efbc3f0ff,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='46c9cac22c32484181c934b27e67f512',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:56:26Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:56:26Z,use_quota=True,user_id='a0751e2174804d7ab462edae5f88d581',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(c9c52f37-7390-466f-a6ab-56ae9a61e5ed),volume_type_id=c9c52f37-7390-466f-a6ab-56ae9a61e5ed)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:56:25 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (c8a9a018-c594-4f2a-b9b3-d1b09245a3fb) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:56:25 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (c8a9a018-c594-4f2a-b9b3-d1b09245a3fb) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:56:25 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (03c9e863-3781-4ccb-877d-e1024bc6d33d) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:56:25 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (03c9e863-3781-4ccb-877d-e1024bc6d33d) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': 'bff6fdbb-a1fb-4b48-bca8-2f7efbc3f0ff', 'volume_name': 'volume-bff6fdbb-a1fb-4b48-bca8-2f7efbc3f0ff', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:56:25 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (db64d0b8-e02e-424b-bf45-8d434339190c) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:56:25 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (db64d0b8-e02e-424b-bf45-8d434339190c) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:56:25 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (e44ce900-5590-4764-9dce-e1f550b0dc9c) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:56:25 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Volume bff6fdbb-a1fb-4b48-bca8-2f7efbc3f0ff: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-bff6fdbb-a1fb-4b48-bca8-2f7efbc3f0ff', 'volume_size': 1} Apr 17 15:56:25 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] creating volume v_bff6fdbb-a1fb-4b48-bca8-2f7efbc3f0ff. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 15:56:25 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] create volume {'name': 'v_bff6fdbb-a1fb-4b48-bca8-2f7efbc3f0ff', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 15:56:25 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:56:25 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:56:28 user cinder-volume[84696]: DEBUG cinder.utils [req-778c62be-8963-4e6d-97f1-f456b759ce16 req-4c60ec5b-6ff0-4c1b-b3ee-f1f60ca15f44 tempest-AttachVolumeShelveTestJSON-1745617049 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 139.515(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:56:28 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-778c62be-8963-4e6d-97f1-f456b759ce16 req-4c60ec5b-6ff0-4c1b-b3ee-f1f60ca15f44 tempest-AttachVolumeShelveTestJSON-1745617049 None] Geting None from DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:13bf568b-585c-4556-ab78-e9930ad50214/luns/v_13bf568b-585c-4556-ab78-e9930ad50214 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:56:28 user cinder-volume[84696]: WARNING cinder.volume.manager [req-778c62be-8963-4e6d-97f1-f456b759ce16 req-4c60ec5b-6ff0-4c1b-b3ee-f1f60ca15f44 tempest-AttachVolumeShelveTestJSON-1745617049 None] Failed to detach volume on the backend, ignoring failure 'NoneType' object is not subscriptable Apr 17 15:56:28 user cinder-volume[84696]: DEBUG cinder.utils [req-778c62be-8963-4e6d-97f1-f456b759ce16 req-6d72b587-594c-4309-b424-7292ebb80925 tempest-AttachVolumeShelveTestJSON-1745617049 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 19.471(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:56:28 user cinder-volume[84696]: DEBUG cinder.utils [req-778c62be-8963-4e6d-97f1-f456b759ce16 req-6d72b587-594c-4309-b424-7292ebb80925 tempest-AttachVolumeShelveTestJSON-1745617049 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:13bf568b-585c-4556-ab78-e9930ad50214/luns/v_13bf568b-585c-4556-ab78-e9930ad50214 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:56:28 user cinder-volume[84696]: DEBUG cinder.utils [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.706(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:56:28 user cinder-volume[84696]: DEBUG cinder.utils [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:56:31 user cinder-volume[84696]: DEBUG cinder.utils [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.774(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:56:31 user cinder-volume[84696]: DEBUG cinder.utils [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:56:31 user cinder-volume[84696]: DEBUG cinder.utils [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 74.132(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:56:31 user cinder-volume[84696]: DEBUG cinder.utils [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:56:31 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 139.518(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:56:31 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:56:31 user cinder-volume[84696]: ERROR oslo_service.periodic_task [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Error during VolumeManager.publish_service_capabilities: TypeError: 'NoneType' object is not subscriptable Apr 17 15:56:31 user cinder-volume[84696]: ERROR oslo_service.periodic_task Traceback (most recent call last): Apr 17 15:56:31 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/usr/local/lib/python3.10/dist-packages/oslo_service/periodic_task.py", line 216, in run_periodic_tasks Apr 17 15:56:31 user cinder-volume[84696]: ERROR oslo_service.periodic_task task(self, context) Apr 17 15:56:31 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/manager.py", line 2876, in publish_service_capabilities Apr 17 15:56:31 user cinder-volume[84696]: ERROR oslo_service.periodic_task self._report_driver_status(context) Apr 17 15:56:31 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/manager.py", line 2755, in _report_driver_status Apr 17 15:56:31 user cinder-volume[84696]: ERROR oslo_service.periodic_task volume_stats = get_stats() Apr 17 15:56:31 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/usr/local/lib/python3.10/dist-packages/oslo_utils/timeutils.py", line 319, in wrapper Apr 17 15:56:31 user cinder-volume[84696]: ERROR oslo_service.periodic_task result = func(*args, **kwargs) Apr 17 15:56:31 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/manager.py", line 2753, in get_stats Apr 17 15:56:31 user cinder-volume[84696]: ERROR oslo_service.periodic_task return self.driver.get_volume_stats(refresh=True) Apr 17 15:56:31 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/driver.py", line 666, in get_volume_stats Apr 17 15:56:31 user cinder-volume[84696]: ERROR oslo_service.periodic_task self._update_volume_stats() Apr 17 15:56:31 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 726, in _update_volume_stats Apr 17 15:56:31 user cinder-volume[84696]: ERROR oslo_service.periodic_task pool_stats = self.ra.get_pool_stats() Apr 17 15:56:31 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 897, in get_pool_stats Apr 17 15:56:31 user cinder-volume[84696]: ERROR oslo_service.periodic_task if not resp["error"] and resp["code"] == 200: Apr 17 15:56:31 user cinder-volume[84696]: ERROR oslo_service.periodic_task TypeError: 'NoneType' object is not subscriptable Apr 17 15:56:31 user cinder-volume[84696]: ERROR oslo_service.periodic_task Apr 17 15:56:31 user cinder-volume[84696]: WARNING oslo.service.loopingcall [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Function 'cinder.service.Service.periodic_tasks' run outlasted interval by 79.52 sec Apr 17 15:56:31 user cinder-volume[84696]: DEBUG oslo_service.periodic_task [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Running periodic task VolumeManager.publish_service_capabilities {{(pid=84696) run_periodic_tasks /usr/local/lib/python3.10/dist-packages/oslo_service/periodic_task.py:210}} Apr 17 15:56:31 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Updating volume stats {{(pid=84696) _update_volume_stats /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:724}} Apr 17 15:56:31 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Get pool Pool-0 fsprops {{(pid=84696) get_pool_stats /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:894}} Apr 17 15:56:31 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:56:31 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:56:34 user cinder-volume[84696]: DEBUG cinder.utils [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.679(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:56:34 user cinder-volume[84696]: DEBUG cinder.utils [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_4a2d1c00-a134-4f73-8508-86f35b58acf7/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:56:34 user cinder-volume[84696]: DEBUG cinder.utils [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 72.099(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:56:34 user cinder-volume[84696]: DEBUG cinder.utils [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:56:34 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 3.063(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:56:34 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:56:37 user cinder-volume[84696]: DEBUG cinder.utils [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 11.918(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:56:37 user cinder-volume[84696]: DEBUG cinder.utils [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:56:37 user cinder-volume[84696]: DEBUG cinder.utils [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 74.061(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:56:37 user cinder-volume[84696]: DEBUG cinder.utils [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:56:37 user cinder-volume[84696]: DEBUG cinder.utils [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 74.251(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:56:37 user cinder-volume[84696]: DEBUG cinder.utils [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_45e5cc09-6c97-438a-a6a2-c89e7d725ce5/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:56:37 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.142(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:56:37 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:56:44 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.853(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:56:44 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:56:47 user cinder-volume[84696]: DEBUG cinder.utils [req-778c62be-8963-4e6d-97f1-f456b759ce16 req-6d72b587-594c-4309-b424-7292ebb80925 tempest-AttachVolumeShelveTestJSON-1745617049 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 38.476(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:56:47 user cinder-volume[84696]: DEBUG cinder.utils [req-778c62be-8963-4e6d-97f1-f456b759ce16 req-6d72b587-594c-4309-b424-7292ebb80925 tempest-AttachVolumeShelveTestJSON-1745617049 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:13bf568b-585c-4556-ab78-e9930ad50214/luns/v_13bf568b-585c-4556-ab78-e9930ad50214 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:56:47 user cinder-volume[84696]: DEBUG cinder.utils [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.711(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:56:47 user cinder-volume[84696]: DEBUG cinder.utils [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:56:47 user cinder-volume[84696]: DEBUG cinder.utils [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 137.172(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:56:47 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:56:47 user cinder-volume[84696]: INFO cinder.message.api [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Creating message record for request_id = req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b Apr 17 15:56:47 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 15:56:47 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (4ce941a6-a914-46eb-b2fa-a0ba34fb5993) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 15:56:47 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 15:56:47 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:54:30Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumeRevertTests-Volume-921998137',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=b29a7592-128a-41ee-a8e2-57780dee33e7,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='9c755d3b5d8a4d9c9918b6c4744f510e',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:54:30Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:54:30Z,use_quota=True,user_id='e362cfe4a31c444b9ab44a5d60f37ad9',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 15:56:47 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:54:30Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumeRevertTests-Volume-921998137',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=b29a7592-128a-41ee-a8e2-57780dee33e7,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='9c755d3b5d8a4d9c9918b6c4744f510e',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:54:30Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:54:30Z,use_quota=True,user_id='e362cfe4a31c444b9ab44a5d60f37ad9',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(b29a7592-128a-41ee-a8e2-57780dee33e7),volume_id=b29a7592-128a-41ee-a8e2-57780dee33e7,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': 'b29a7592-128a-41ee-a8e2-57780dee33e7', 'volume_name': 'volume-b29a7592-128a-41ee-a8e2-57780dee33e7', 'volume_size': 1}} Apr 17 15:56:47 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'retry': {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']}, 'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': 'b29a7592-128a-41ee-a8e2-57780dee33e7', 'volume': {'id': 'b29a7592-128a-41ee-a8e2-57780dee33e7', '_name_id': None, 'ec2_id': None, 'user_id': 'e362cfe4a31c444b9ab44a5d60f37ad9', 'project_id': '9c755d3b5d8a4d9c9918b6c4744f510e', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-VolumeRevertTests-Volume-921998137', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T15:54:30.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-b29a7592-128a-41ee-a8e2-57780dee33e7', 'name_id': 'b29a7592-128a-41ee-a8e2-57780dee33e7', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumeRevertTests-Volume-921998137', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '9c755d3b5d8a4d9c9918b6c4744f510e', 'qos_specs': None, 'reservations': ['42b0fe0b-67bc-438d-b5b5-5cd8608f187a', '35c09dbe-7331-435e-a061-4f56338d0c2b', '2a16b96e-8144-4c17-a826-7c22f889d9e1', 'a666665d-cbb8-4dfe-9bf7-027ec0df440c'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'e362cfe4a31c444b9ab44a5d60f37ad9', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumeRevertTests-Volume-921998137', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '9c755d3b5d8a4d9c9918b6c4744f510e', 'qos_specs': None, 'reservations': ['42b0fe0b-67bc-438d-b5b5-5cd8608f187a', '35c09dbe-7331-435e-a061-4f56338d0c2b', '2a16b96e-8144-4c17-a826-7c22f889d9e1', 'a666665d-cbb8-4dfe-9bf7-027ec0df440c'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'e362cfe4a31c444b9ab44a5d60f37ad9', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': 'e362cfe4a31c444b9ab44a5d60f37ad9', 'metadata': {}, 'qos_specs': None}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(b29a7592-128a-41ee-a8e2-57780dee33e7),volume_id=b29a7592-128a-41ee-a8e2-57780dee33e7,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:54:30Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumeRevertTests-Volume-921998137',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=b29a7592-128a-41ee-a8e2-57780dee33e7,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='9c755d3b5d8a4d9c9918b6c4744f510e',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:54:30Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:54:30Z,use_quota=True,user_id='e362cfe4a31c444b9ab44a5d60f37ad9',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 15:56:47 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:54:30Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumeRevertTests-Volume-921998137',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=b29a7592-128a-41ee-a8e2-57780dee33e7,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='9c755d3b5d8a4d9c9918b6c4744f510e',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:54:30Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:54:30Z,use_quota=True,user_id='e362cfe4a31c444b9ab44a5d60f37ad9',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:54:30Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumeRevertTests-Volume-921998137',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=b29a7592-128a-41ee-a8e2-57780dee33e7,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='9c755d3b5d8a4d9c9918b6c4744f510e',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:54:30Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:54:30Z,use_quota=True,user_id='e362cfe4a31c444b9ab44a5d60f37ad9',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 15:56:47 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 15:56:47 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 15:56:47 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:56:47 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 15:56:47 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 15:56:47 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 15:56:47 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 15:56:47 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 15:56:47 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 15:56:47 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 15:56:47 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 15:56:47 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 15:56:47 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 15:56:47 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 15:56:47 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 15:56:47 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 15:56:47 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 15:56:47 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 15:56:47 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 15:56:47 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 15:56:47 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (4ce941a6-a914-46eb-b2fa-a0ba34fb5993) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:56:47 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (4ce941a6-a914-46eb-b2fa-a0ba34fb5993) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:56:47 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (a6b7fb08-fd75-40df-a8b5-2bb5fdcf5764) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:56:47 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (a6b7fb08-fd75-40df-a8b5-2bb5fdcf5764) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:56:47 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (e862499e-fd0c-46c9-820a-b9939bf93664) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:56:47 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (e862499e-fd0c-46c9-820a-b9939bf93664) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:56:47 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (d5b56f26-d4f7-4e07-a5e0-44e70a74b0f8) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:56:47 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Updating volume b29a7592-128a-41ee-a8e2-57780dee33e7 with {'scheduled_at': datetime.datetime(2023, 4, 17, 15, 56, 47, 752044), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 15:56:47 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Volume b29a7592-128a-41ee-a8e2-57780dee33e7: re-scheduling SchedulerAPI.create_volume attempt 1 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 15:56:47 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Volume b29a7592-128a-41ee-a8e2-57780dee33e7: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 15:56:47 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] deleating volume v_b29a7592-128a-41ee-a8e2-57780dee33e7 {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 15:56:47 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] get snapshots for volume v_b29a7592-128a-41ee-a8e2-57780dee33e7 {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 15:56:47 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_b29a7592-128a-41ee-a8e2-57780dee33e7/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:56:47 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_b29a7592-128a-41ee-a8e2-57780dee33e7/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:56:47 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Volume reschedule parameters: True retry: {'num_attempts': 2, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 15:56:47 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Flow 'volume_create_manager' (440df6e8-ca58-404d-8b07-c6b0bc9724c5) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:56:47 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (d17cff04-6a68-4fb8-8aff-871f5be05fd2) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:56:47 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (d17cff04-6a68-4fb8-8aff-871f5be05fd2) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:54:30Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumeRevertTests-Volume-921998137',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=b29a7592-128a-41ee-a8e2-57780dee33e7,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='9c755d3b5d8a4d9c9918b6c4744f510e',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:56:48Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:56:48Z,use_quota=True,user_id='e362cfe4a31c444b9ab44a5d60f37ad9',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:56:47 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (915a393a-340d-41bd-8e81-3465e5e593ca) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:56:47 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (915a393a-340d-41bd-8e81-3465e5e593ca) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:56:47 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (13088d06-9e6f-464a-8ef0-61241efd8932) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:56:47 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (13088d06-9e6f-464a-8ef0-61241efd8932) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': 'b29a7592-128a-41ee-a8e2-57780dee33e7', 'volume_name': 'volume-b29a7592-128a-41ee-a8e2-57780dee33e7', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:56:47 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (487419f3-40d1-4f01-9a35-bbbd3ced3e9d) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:56:47 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (487419f3-40d1-4f01-9a35-bbbd3ced3e9d) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:56:47 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (4f3bedab-a546-444d-94ec-c5ed9348b765) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:56:47 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Volume b29a7592-128a-41ee-a8e2-57780dee33e7: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-b29a7592-128a-41ee-a8e2-57780dee33e7', 'volume_size': 1} Apr 17 15:56:47 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] creating volume v_b29a7592-128a-41ee-a8e2-57780dee33e7. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 15:56:47 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] create volume {'name': 'v_b29a7592-128a-41ee-a8e2-57780dee33e7', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 15:56:47 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:56:47 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:56:50 user cinder-volume[84696]: DEBUG cinder.utils [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.953(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:56:50 user cinder-volume[84696]: DEBUG cinder.utils [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_b29a7592-128a-41ee-a8e2-57780dee33e7/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:56:50 user cinder-volume[84696]: DEBUG cinder.utils [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.777(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:56:50 user cinder-volume[84696]: DEBUG cinder.utils [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:56:53 user cinder-volume[84696]: DEBUG cinder.utils [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 138.286(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:56:53 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:56:53 user cinder-volume[84696]: WARNING cinder.volume.manager [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (f1597ca6-914b-4b23-84df-15dbc8a8c06a) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 15:56:53 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 15:56:53 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:54:35Z,deleted=False,deleted_at=None,display_description='',display_name='',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=75163b78-31b2-4676-b93b-542a8ea784c1,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='540826be4baf46dba9617b2fd47821a3',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:54:35Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:54:35Z,use_quota=True,user_id='d819e3f51368413c99691d1b45bbcfb1',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 15:56:53 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:54:35Z,deleted=False,deleted_at=None,display_description='',display_name='',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=75163b78-31b2-4676-b93b-542a8ea784c1,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='540826be4baf46dba9617b2fd47821a3',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:54:35Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:54:35Z,use_quota=True,user_id='d819e3f51368413c99691d1b45bbcfb1',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=b3f0dec1-7758-44a8-82e0-76ada8837428,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(75163b78-31b2-4676-b93b-542a8ea784c1),volume_id=75163b78-31b2-4676-b93b-542a8ea784c1,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'image', 'volume_id': '75163b78-31b2-4676-b93b-542a8ea784c1', 'volume_name': 'volume-75163b78-31b2-4676-b93b-542a8ea784c1', 'volume_size': 1, 'image_id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'image_location': (None, None), 'image_meta': {'name': 'cirros-0.5.2-x86_64-disk', 'disk_format': 'qcow2', 'container_format': 'bare', 'visibility': 'public', 'size': 16300544, 'virtual_size': 117440512, 'status': 'active', 'checksum': 'b874c39491a2377b8490f5f1e89761a4', 'protected': False, 'min_ram': 0, 'min_disk': 0, 'owner': 'a43c211b4ed542538799349489b1d884', 'os_hidden': False, 'os_hash_algo': 'sha512', 'os_hash_value': '6b813aa46bb90b4da216a4d19376593fa3f4fc7e617f03a92b7fe11e9a3981cbe8f0959dbebe36225e5f53dc4492341a4863cac4ed1ee0909f3fc78ef9c3e869', 'id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'created_at': datetime.datetime(2023, 4, 17, 15, 19, 53, tzinfo=datetime.timezone.utc), 'updated_at': datetime.datetime(2023, 4, 17, 15, 19, 55, tzinfo=datetime.timezone.utc), 'tags': [], 'file': '/v2/images/b3f0dec1-7758-44a8-82e0-76ada8837428/file', 'properties': {'hw_rng_model': 'virtio', 'owner_specified.openstack.md5': '', 'owner_specified.openstack.object': 'images/cirros-0.5.2-x86_64-disk', 'owner_specified.openstack.sha256': ''}}, 'image_service': }} Apr 17 15:56:53 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'retry': {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']}, 'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'snapshot_id': None, 'source_volid': None, 'volume_id': '75163b78-31b2-4676-b93b-542a8ea784c1', 'volume': {'id': '75163b78-31b2-4676-b93b-542a8ea784c1', '_name_id': None, 'ec2_id': None, 'user_id': 'd819e3f51368413c99691d1b45bbcfb1', 'project_id': '540826be4baf46dba9617b2fd47821a3', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': '', 'display_description': '', 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T15:54:35.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-75163b78-31b2-4676-b93b-542a8ea784c1', 'name_id': '75163b78-31b2-4676-b93b-542a8ea784c1', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': '', 'display_name': '', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '540826be4baf46dba9617b2fd47821a3', 'qos_specs': None, 'reservations': ['351bb8e5-5094-4129-bd6a-8f086342ae4f', '4dc05382-d603-486b-ab14-e1caee8118dd', 'b04f0e6c-aaea-44ea-b237-e337edc22236', 'e402f3ac-509b-4dac-8852-9c619b150ea3'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'd819e3f51368413c99691d1b45bbcfb1', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': '', 'display_name': '', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '540826be4baf46dba9617b2fd47821a3', 'qos_specs': None, 'reservations': ['351bb8e5-5094-4129-bd6a-8f086342ae4f', '4dc05382-d603-486b-ab14-e1caee8118dd', 'b04f0e6c-aaea-44ea-b237-e337edc22236', 'e402f3ac-509b-4dac-8852-9c619b150ea3'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'd819e3f51368413c99691d1b45bbcfb1', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': 'd819e3f51368413c99691d1b45bbcfb1', 'metadata': {}, 'qos_specs': None}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=b3f0dec1-7758-44a8-82e0-76ada8837428,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(75163b78-31b2-4676-b93b-542a8ea784c1),volume_id=75163b78-31b2-4676-b93b-542a8ea784c1,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:54:35Z,deleted=False,deleted_at=None,display_description='',display_name='',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=75163b78-31b2-4676-b93b-542a8ea784c1,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='540826be4baf46dba9617b2fd47821a3',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:54:35Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:54:35Z,use_quota=True,user_id='d819e3f51368413c99691d1b45bbcfb1',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 15:56:53 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:54:35Z,deleted=False,deleted_at=None,display_description='',display_name='',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=75163b78-31b2-4676-b93b-542a8ea784c1,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='540826be4baf46dba9617b2fd47821a3',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:54:35Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:54:35Z,use_quota=True,user_id='d819e3f51368413c99691d1b45bbcfb1',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:54:35Z,deleted=False,deleted_at=None,display_description='',display_name='',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=75163b78-31b2-4676-b93b-542a8ea784c1,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='540826be4baf46dba9617b2fd47821a3',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:54:35Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:54:35Z,use_quota=True,user_id='d819e3f51368413c99691d1b45bbcfb1',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 15:56:53 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 15:56:53 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 15:56:53 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:56:53 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 15:56:53 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1237, in execute Apr 17 15:56:53 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_from_image(context, Apr 17 15:56:53 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/utils.py", line 670, in _wrapper Apr 17 15:56:53 user cinder-volume[84696]: ERROR cinder.volume.manager return r(f, *args, **kwargs) Apr 17 15:56:53 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/tenacity/__init__.py", line 423, in __call__ Apr 17 15:56:53 user cinder-volume[84696]: ERROR cinder.volume.manager do = self.iter(retry_state=retry_state) Apr 17 15:56:53 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/tenacity/__init__.py", line 360, in iter Apr 17 15:56:53 user cinder-volume[84696]: ERROR cinder.volume.manager return fut.result() Apr 17 15:56:53 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/lib/python3.10/concurrent/futures/_base.py", line 451, in result Apr 17 15:56:53 user cinder-volume[84696]: ERROR cinder.volume.manager return self.__get_result() Apr 17 15:56:53 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result Apr 17 15:56:53 user cinder-volume[84696]: ERROR cinder.volume.manager raise self._exception Apr 17 15:56:53 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/tenacity/__init__.py", line 426, in __call__ Apr 17 15:56:53 user cinder-volume[84696]: ERROR cinder.volume.manager result = fn(*args, **kwargs) Apr 17 15:56:53 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1125, in _create_from_image Apr 17 15:56:53 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_from_image_cache_or_download( Apr 17 15:56:53 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 944, in _create_from_image_cache_or_download Apr 17 15:56:53 user cinder-volume[84696]: ERROR cinder.volume.manager model_update, cloned = self._create_from_image_cache( Apr 17 15:56:53 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 843, in _create_from_image_cache Apr 17 15:56:53 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_from_source_volume( Apr 17 15:56:53 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 659, in _create_from_source_volume Apr 17 15:56:53 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self.driver.create_cloned_volume(volume, srcvol_ref) Apr 17 15:56:53 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 525, in create_cloned_volume Apr 17 15:56:53 user cinder-volume[84696]: ERROR cinder.volume.manager self._clone_object(vname, cvname) Apr 17 15:56:53 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 461, in _clone_object Apr 17 15:56:53 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_snapshot(oname, coname) Apr 17 15:56:53 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 639, in create_snapshot Apr 17 15:56:53 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201, 204): Apr 17 15:56:53 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 15:56:53 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 15:56:53 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (f1597ca6-914b-4b23-84df-15dbc8a8c06a) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:56:53 user cinder-volume[84696]: WARNING cinder.volume.manager [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (f1597ca6-914b-4b23-84df-15dbc8a8c06a) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:56:53 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (87712448-415e-4081-a97e-1cb26456073c) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:56:53 user cinder-volume[84696]: WARNING cinder.volume.manager [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (87712448-415e-4081-a97e-1cb26456073c) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:56:53 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (35168b46-28c7-40f3-aac5-bd34807f6de9) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:56:53 user cinder-volume[84696]: WARNING cinder.volume.manager [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (35168b46-28c7-40f3-aac5-bd34807f6de9) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:56:53 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (6f50d91f-d980-44f9-8b14-89e1707bab15) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:56:53 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Updating volume 75163b78-31b2-4676-b93b-542a8ea784c1 with {'scheduled_at': datetime.datetime(2023, 4, 17, 15, 56, 53, 864634), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 15:56:53 user cinder-volume[84696]: DEBUG cinder.utils [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 40.734(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:56:53 user cinder-volume[84696]: DEBUG cinder.utils [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_4a2d1c00-a134-4f73-8508-86f35b58acf7/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:56:53 user cinder-volume[84696]: DEBUG cinder.utils [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.069(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:56:53 user cinder-volume[84696]: DEBUG cinder.utils [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_b29a7592-128a-41ee-a8e2-57780dee33e7/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:56:53 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 22.118(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:56:53 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:56:53 user cinder-volume[84696]: DEBUG cinder.utils [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 136.991(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:56:53 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_60979eba-96b1-4cb3-bae1-8874a86cb411/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:56:53 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (636fc593-9b63-4d21-9a0e-3c875f1e78d7) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 15:56:53 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (82c0f5e5-8513-4e74-89ed-f77b1bbf1d72) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:56:53 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (82c0f5e5-8513-4e74-89ed-f77b1bbf1d72) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:56:53 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Flow 'volume_create_manager' (089d207d-9f34-4926-89a0-81b8785be524) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 15:56:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 15:56:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 15:56:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 15:56:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 15:56:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 15:56:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 15:56:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 15:56:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 15:56:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 15:56:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 15:56:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 15:56:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 15:56:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 15:56:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 15:56:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 15:56:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 15:56:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 15:56:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 15:56:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 15:56:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 15:56:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 15:56:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 15:56:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 15:56:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 15:56:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:56:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 15:56:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1237, in execute Apr 17 15:56:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_from_image(context, Apr 17 15:56:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/utils.py", line 670, in _wrapper Apr 17 15:56:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return r(f, *args, **kwargs) Apr 17 15:56:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/tenacity/__init__.py", line 423, in __call__ Apr 17 15:56:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server do = self.iter(retry_state=retry_state) Apr 17 15:56:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/tenacity/__init__.py", line 360, in iter Apr 17 15:56:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return fut.result() Apr 17 15:56:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/lib/python3.10/concurrent/futures/_base.py", line 451, in result Apr 17 15:56:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self.__get_result() Apr 17 15:56:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result Apr 17 15:56:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self._exception Apr 17 15:56:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/tenacity/__init__.py", line 426, in __call__ Apr 17 15:56:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = fn(*args, **kwargs) Apr 17 15:56:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1125, in _create_from_image Apr 17 15:56:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_from_image_cache_or_download( Apr 17 15:56:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 944, in _create_from_image_cache_or_download Apr 17 15:56:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update, cloned = self._create_from_image_cache( Apr 17 15:56:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 843, in _create_from_image_cache Apr 17 15:56:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_from_source_volume( Apr 17 15:56:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 659, in _create_from_source_volume Apr 17 15:56:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self.driver.create_cloned_volume(volume, srcvol_ref) Apr 17 15:56:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 525, in create_cloned_volume Apr 17 15:56:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self._clone_object(vname, cvname) Apr 17 15:56:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 461, in _clone_object Apr 17 15:56:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_snapshot(oname, coname) Apr 17 15:56:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 639, in create_snapshot Apr 17 15:56:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201, 204): Apr 17 15:56:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 15:56:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 15:56:53 user cinder-volume[84696]: DEBUG cinder.utils [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 136.593(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:56:53 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:56:53 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (6181f2ad-dc29-4e8a-8c6a-1a45cc5bf4f2) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 15:56:53 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 15:56:53 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:52:20Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TestVolumeBootPattern-volume-origin-952290434',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=60979eba-96b1-4cb3-bae1-8874a86cb411,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='c8a836b47d954cb9a7c1cd7c0f790426',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:54:37Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:54:37Z,use_quota=True,user_id='a80e753322224ec1a897598b5b9ed6ea',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 15:56:53 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:52:20Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TestVolumeBootPattern-volume-origin-952290434',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=60979eba-96b1-4cb3-bae1-8874a86cb411,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='c8a836b47d954cb9a7c1cd7c0f790426',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:54:37Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:54:37Z,use_quota=True,user_id='a80e753322224ec1a897598b5b9ed6ea',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=b3f0dec1-7758-44a8-82e0-76ada8837428,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(60979eba-96b1-4cb3-bae1-8874a86cb411),volume_id=60979eba-96b1-4cb3-bae1-8874a86cb411,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'image', 'volume_id': '60979eba-96b1-4cb3-bae1-8874a86cb411', 'volume_name': 'volume-60979eba-96b1-4cb3-bae1-8874a86cb411', 'volume_size': 1, 'image_id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'image_location': (None, None), 'image_meta': {'name': 'cirros-0.5.2-x86_64-disk', 'disk_format': 'qcow2', 'container_format': 'bare', 'visibility': 'public', 'size': 16300544, 'virtual_size': 117440512, 'status': 'active', 'checksum': 'b874c39491a2377b8490f5f1e89761a4', 'protected': False, 'min_ram': 0, 'min_disk': 0, 'owner': 'a43c211b4ed542538799349489b1d884', 'os_hidden': False, 'os_hash_algo': 'sha512', 'os_hash_value': '6b813aa46bb90b4da216a4d19376593fa3f4fc7e617f03a92b7fe11e9a3981cbe8f0959dbebe36225e5f53dc4492341a4863cac4ed1ee0909f3fc78ef9c3e869', 'id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'created_at': datetime.datetime(2023, 4, 17, 15, 19, 53, tzinfo=datetime.timezone.utc), 'updated_at': datetime.datetime(2023, 4, 17, 15, 19, 55, tzinfo=datetime.timezone.utc), 'tags': [], 'file': '/v2/images/b3f0dec1-7758-44a8-82e0-76ada8837428/file', 'properties': {'hw_rng_model': 'virtio', 'owner_specified.openstack.md5': '', 'owner_specified.openstack.object': 'images/cirros-0.5.2-x86_64-disk', 'owner_specified.openstack.sha256': ''}}, 'image_service': }} Apr 17 15:56:53 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'snapshot_id': None, 'source_volid': None, 'volume_id': '60979eba-96b1-4cb3-bae1-8874a86cb411', 'volume': {'id': '60979eba-96b1-4cb3-bae1-8874a86cb411', '_name_id': None, 'ec2_id': None, 'user_id': 'a80e753322224ec1a897598b5b9ed6ea', 'project_id': 'c8a836b47d954cb9a7c1cd7c0f790426', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-TestVolumeBootPattern-volume-origin-952290434', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'admin_metadata': {}, 'glance_metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T15:52:19.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-60979eba-96b1-4cb3-bae1-8874a86cb411', 'name_id': '60979eba-96b1-4cb3-bae1-8874a86cb411', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-TestVolumeBootPattern-volume-origin-952290434', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': 'c8a836b47d954cb9a7c1cd7c0f790426', 'qos_specs': None, 'reservations': ['665fbb3e-6435-4ad5-b020-211f010efcfd', '6df9e628-e8f9-4f25-b1cc-f04ca7f0e972', '0f4da886-833a-4bdb-a9a5-4937f5294c9a', 'f63e6f69-e592-4e3a-8f7a-79eaf32aacf0'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'a80e753322224ec1a897598b5b9ed6ea', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-TestVolumeBootPattern-volume-origin-952290434', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': 'c8a836b47d954cb9a7c1cd7c0f790426', 'qos_specs': None, 'reservations': ['665fbb3e-6435-4ad5-b020-211f010efcfd', '6df9e628-e8f9-4f25-b1cc-f04ca7f0e972', '0f4da886-833a-4bdb-a9a5-4937f5294c9a', 'f63e6f69-e592-4e3a-8f7a-79eaf32aacf0'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'a80e753322224ec1a897598b5b9ed6ea', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': 'a80e753322224ec1a897598b5b9ed6ea', 'metadata': {}, 'qos_specs': None, 'retry': {'num_attempts': 2, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']}}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=b3f0dec1-7758-44a8-82e0-76ada8837428,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(60979eba-96b1-4cb3-bae1-8874a86cb411),volume_id=60979eba-96b1-4cb3-bae1-8874a86cb411,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:52:20Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TestVolumeBootPattern-volume-origin-952290434',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=60979eba-96b1-4cb3-bae1-8874a86cb411,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='c8a836b47d954cb9a7c1cd7c0f790426',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:54:37Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:54:37Z,use_quota=True,user_id='a80e753322224ec1a897598b5b9ed6ea',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 15:56:53 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:52:20Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TestVolumeBootPattern-volume-origin-952290434',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=60979eba-96b1-4cb3-bae1-8874a86cb411,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='c8a836b47d954cb9a7c1cd7c0f790426',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:54:37Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:54:37Z,use_quota=True,user_id='a80e753322224ec1a897598b5b9ed6ea',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:52:20Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TestVolumeBootPattern-volume-origin-952290434',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=60979eba-96b1-4cb3-bae1-8874a86cb411,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='c8a836b47d954cb9a7c1cd7c0f790426',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:54:37Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:54:37Z,use_quota=True,user_id='a80e753322224ec1a897598b5b9ed6ea',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 15:56:53 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 15:56:53 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 15:56:53 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:56:53 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 15:56:53 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1237, in execute Apr 17 15:56:53 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_from_image(context, Apr 17 15:56:53 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/utils.py", line 670, in _wrapper Apr 17 15:56:53 user cinder-volume[84696]: ERROR cinder.volume.manager return r(f, *args, **kwargs) Apr 17 15:56:53 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/tenacity/__init__.py", line 423, in __call__ Apr 17 15:56:53 user cinder-volume[84696]: ERROR cinder.volume.manager do = self.iter(retry_state=retry_state) Apr 17 15:56:53 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/tenacity/__init__.py", line 360, in iter Apr 17 15:56:53 user cinder-volume[84696]: ERROR cinder.volume.manager return fut.result() Apr 17 15:56:53 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/lib/python3.10/concurrent/futures/_base.py", line 451, in result Apr 17 15:56:53 user cinder-volume[84696]: ERROR cinder.volume.manager return self.__get_result() Apr 17 15:56:53 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result Apr 17 15:56:53 user cinder-volume[84696]: ERROR cinder.volume.manager raise self._exception Apr 17 15:56:53 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/tenacity/__init__.py", line 426, in __call__ Apr 17 15:56:53 user cinder-volume[84696]: ERROR cinder.volume.manager result = fn(*args, **kwargs) Apr 17 15:56:53 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1125, in _create_from_image Apr 17 15:56:53 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_from_image_cache_or_download( Apr 17 15:56:53 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 944, in _create_from_image_cache_or_download Apr 17 15:56:53 user cinder-volume[84696]: ERROR cinder.volume.manager model_update, cloned = self._create_from_image_cache( Apr 17 15:56:53 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 843, in _create_from_image_cache Apr 17 15:56:53 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_from_source_volume( Apr 17 15:56:53 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 659, in _create_from_source_volume Apr 17 15:56:53 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self.driver.create_cloned_volume(volume, srcvol_ref) Apr 17 15:56:53 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 525, in create_cloned_volume Apr 17 15:56:53 user cinder-volume[84696]: ERROR cinder.volume.manager self._clone_object(vname, cvname) Apr 17 15:56:53 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 461, in _clone_object Apr 17 15:56:53 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_snapshot(oname, coname) Apr 17 15:56:53 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 639, in create_snapshot Apr 17 15:56:53 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201, 204): Apr 17 15:56:53 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 15:56:53 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 15:56:53 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (6181f2ad-dc29-4e8a-8c6a-1a45cc5bf4f2) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:56:53 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (6181f2ad-dc29-4e8a-8c6a-1a45cc5bf4f2) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:56:53 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (bba824b3-b9cf-4cdf-9327-015636ef3a39) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:56:53 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (bba824b3-b9cf-4cdf-9327-015636ef3a39) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:56:53 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (3e755762-1c31-4abb-ba98-11ed515f284e) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:56:53 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (3e755762-1c31-4abb-ba98-11ed515f284e) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:56:53 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (19719143-3e90-493d-82f6-93075aeceb7d) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:56:53 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Updating volume 60979eba-96b1-4cb3-bae1-8874a86cb411 with {'scheduled_at': datetime.datetime(2023, 4, 17, 15, 56, 53, 930936), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 15:56:53 user cinder-volume[84696]: DEBUG cinder.utils [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.958(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:56:53 user cinder-volume[84696]: DEBUG cinder.utils [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:56:53 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Volume 75163b78-31b2-4676-b93b-542a8ea784c1: re-scheduling SchedulerAPI.create_volume attempt 1 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 15:56:53 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Volume 60979eba-96b1-4cb3-bae1-8874a86cb411: re-scheduling SchedulerAPI.create_volume attempt 2 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 15:56:53 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Volume 75163b78-31b2-4676-b93b-542a8ea784c1: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 15:56:53 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] deleating volume v_75163b78-31b2-4676-b93b-542a8ea784c1 {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 15:56:53 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] get snapshots for volume v_75163b78-31b2-4676-b93b-542a8ea784c1 {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 15:56:53 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_75163b78-31b2-4676-b93b-542a8ea784c1/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:56:53 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_75163b78-31b2-4676-b93b-542a8ea784c1/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:56:53 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Volume 60979eba-96b1-4cb3-bae1-8874a86cb411: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 15:56:53 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] deleating volume v_60979eba-96b1-4cb3-bae1-8874a86cb411 {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 15:56:53 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] get snapshots for volume v_60979eba-96b1-4cb3-bae1-8874a86cb411 {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 15:56:53 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_60979eba-96b1-4cb3-bae1-8874a86cb411/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:56:53 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_60979eba-96b1-4cb3-bae1-8874a86cb411/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:56:54 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Volume reschedule parameters: True retry: {'num_attempts': 2, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 15:56:54 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Flow 'volume_create_manager' (b3bd1bf6-c485-4297-8950-6953e3362a34) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:56:54 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (7a4fa399-4332-4cef-b05b-afeb82e522d5) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:56:54 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Volume reschedule parameters: True retry: {'num_attempts': 3, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 15:56:54 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Flow 'volume_create_manager' (865561c5-292c-4abc-9f91-06e1f05fe0c2) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:56:54 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (217e28fa-02c6-477f-a0e6-6097aad3a3b8) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:56:54 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (7a4fa399-4332-4cef-b05b-afeb82e522d5) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:54:35Z,deleted=False,deleted_at=None,display_description='',display_name='',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=75163b78-31b2-4676-b93b-542a8ea784c1,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='540826be4baf46dba9617b2fd47821a3',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:56:54Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:56:54Z,use_quota=True,user_id='d819e3f51368413c99691d1b45bbcfb1',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:56:54 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (9ebb7d34-65cd-4f48-a5e2-e36a75702ea7) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:56:54 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (9ebb7d34-65cd-4f48-a5e2-e36a75702ea7) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:56:54 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (d4b29370-88cf-4816-b75b-7ce60e9dd025) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:56:54 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (217e28fa-02c6-477f-a0e6-6097aad3a3b8) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:52:20Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TestVolumeBootPattern-volume-origin-952290434',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=60979eba-96b1-4cb3-bae1-8874a86cb411,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='c8a836b47d954cb9a7c1cd7c0f790426',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:56:54Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:56:54Z,use_quota=True,user_id='a80e753322224ec1a897598b5b9ed6ea',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:56:54 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (4ff6d4e8-85fa-467e-acdc-870b80c1f4fc) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:56:54 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (4ff6d4e8-85fa-467e-acdc-870b80c1f4fc) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:56:54 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (6646d7ca-1d1c-4a96-8aa3-70fff100ad4f) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:56:54 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (d4b29370-88cf-4816-b75b-7ce60e9dd025) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'image', 'volume_id': '75163b78-31b2-4676-b93b-542a8ea784c1', 'volume_name': 'volume-75163b78-31b2-4676-b93b-542a8ea784c1', 'volume_size': 1, 'image_id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'image_location': (None, None), 'image_meta': {'name': 'cirros-0.5.2-x86_64-disk', 'disk_format': 'qcow2', 'container_format': 'bare', 'visibility': 'public', 'size': 16300544, 'virtual_size': 117440512, 'status': 'active', 'checksum': 'b874c39491a2377b8490f5f1e89761a4', 'protected': False, 'min_ram': 0, 'min_disk': 0, 'owner': 'a43c211b4ed542538799349489b1d884', 'os_hidden': False, 'os_hash_algo': 'sha512', 'os_hash_value': '6b813aa46bb90b4da216a4d19376593fa3f4fc7e617f03a92b7fe11e9a3981cbe8f0959dbebe36225e5f53dc4492341a4863cac4ed1ee0909f3fc78ef9c3e869', 'id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'created_at': datetime.datetime(2023, 4, 17, 15, 19, 53, tzinfo=datetime.timezone.utc), 'updated_at': datetime.datetime(2023, 4, 17, 15, 19, 55, tzinfo=datetime.timezone.utc), 'tags': [], 'file': '/v2/images/b3f0dec1-7758-44a8-82e0-76ada8837428/file', 'properties': {'hw_rng_model': 'virtio', 'owner_specified.openstack.md5': '', 'owner_specified.openstack.object': 'images/cirros-0.5.2-x86_64-disk', 'owner_specified.openstack.sha256': ''}}, 'image_service': }' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:56:54 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (c93c4cc0-9f18-4132-9b7a-2f91ee796536) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:56:54 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (c93c4cc0-9f18-4132-9b7a-2f91ee796536) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:56:54 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (994f4aa9-d679-40ca-ab50-69c4b240fe33) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:56:54 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Volume 75163b78-31b2-4676-b93b-542a8ea784c1: being created as image with specification: {'status': 'creating', 'volume_name': 'volume-75163b78-31b2-4676-b93b-542a8ea784c1', 'volume_size': 1, 'image_id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'image_location': (None, None), 'image_meta': {'name': 'cirros-0.5.2-x86_64-disk', 'disk_format': 'qcow2', 'container_format': 'bare', 'visibility': 'public', 'size': 16300544, 'virtual_size': 117440512, 'status': 'active', 'checksum': 'b874c39491a2377b8490f5f1e89761a4', 'protected': False, 'min_ram': 0, 'min_disk': 0, 'owner': 'a43c211b4ed542538799349489b1d884', 'os_hidden': False, 'os_hash_algo': 'sha512', 'os_hash_value': '6b813aa46bb90b4da216a4d19376593fa3f4fc7e617f03a92b7fe11e9a3981cbe8f0959dbebe36225e5f53dc4492341a4863cac4ed1ee0909f3fc78ef9c3e869', 'id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'created_at': datetime.datetime(2023, 4, 17, 15, 19, 53, tzinfo=datetime.timezone.utc), 'updated_at': datetime.datetime(2023, 4, 17, 15, 19, 55, tzinfo=datetime.timezone.utc), 'tags': [], 'file': '/v2/images/b3f0dec1-7758-44a8-82e0-76ada8837428/file', 'properties': {'hw_rng_model': 'virtio', 'owner_specified.openstack.md5': '', 'owner_specified.openstack.object': 'images/cirros-0.5.2-x86_64-disk', 'owner_specified.openstack.sha256': ''}}, 'image_service': } Apr 17 15:56:54 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Cloning 75163b78-31b2-4676-b93b-542a8ea784c1 from image b3f0dec1-7758-44a8-82e0-76ada8837428 at location (None, None). {{(pid=84696) _create_from_image /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1071}} Apr 17 15:56:54 user cinder-volume[84696]: DEBUG cinder.coordination [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Acquiring lock "cinder-b3f0dec1-7758-44a8-82e0-76ada8837428" by "_prepare_image_cache_entry" {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:192}} Apr 17 15:56:54 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (6646d7ca-1d1c-4a96-8aa3-70fff100ad4f) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'image', 'volume_id': '60979eba-96b1-4cb3-bae1-8874a86cb411', 'volume_name': 'volume-60979eba-96b1-4cb3-bae1-8874a86cb411', 'volume_size': 1, 'image_id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'image_location': (None, None), 'image_meta': {'name': 'cirros-0.5.2-x86_64-disk', 'disk_format': 'qcow2', 'container_format': 'bare', 'visibility': 'public', 'size': 16300544, 'virtual_size': 117440512, 'status': 'active', 'checksum': 'b874c39491a2377b8490f5f1e89761a4', 'protected': False, 'min_ram': 0, 'min_disk': 0, 'owner': 'a43c211b4ed542538799349489b1d884', 'os_hidden': False, 'os_hash_algo': 'sha512', 'os_hash_value': '6b813aa46bb90b4da216a4d19376593fa3f4fc7e617f03a92b7fe11e9a3981cbe8f0959dbebe36225e5f53dc4492341a4863cac4ed1ee0909f3fc78ef9c3e869', 'id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'created_at': datetime.datetime(2023, 4, 17, 15, 19, 53, tzinfo=datetime.timezone.utc), 'updated_at': datetime.datetime(2023, 4, 17, 15, 19, 55, tzinfo=datetime.timezone.utc), 'tags': [], 'file': '/v2/images/b3f0dec1-7758-44a8-82e0-76ada8837428/file', 'properties': {'hw_rng_model': 'virtio', 'owner_specified.openstack.md5': '', 'owner_specified.openstack.object': 'images/cirros-0.5.2-x86_64-disk', 'owner_specified.openstack.sha256': ''}}, 'image_service': }' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:56:54 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (2a119d5d-67ee-4e23-aab1-ba8763a728e7) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:56:54 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (2a119d5d-67ee-4e23-aab1-ba8763a728e7) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:56:54 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (e424f74d-587b-48a5-95ce-cb5f915cf222) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:56:54 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Volume 60979eba-96b1-4cb3-bae1-8874a86cb411: being created as image with specification: {'status': 'creating', 'volume_name': 'volume-60979eba-96b1-4cb3-bae1-8874a86cb411', 'volume_size': 1, 'image_id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'image_location': (None, None), 'image_meta': {'name': 'cirros-0.5.2-x86_64-disk', 'disk_format': 'qcow2', 'container_format': 'bare', 'visibility': 'public', 'size': 16300544, 'virtual_size': 117440512, 'status': 'active', 'checksum': 'b874c39491a2377b8490f5f1e89761a4', 'protected': False, 'min_ram': 0, 'min_disk': 0, 'owner': 'a43c211b4ed542538799349489b1d884', 'os_hidden': False, 'os_hash_algo': 'sha512', 'os_hash_value': '6b813aa46bb90b4da216a4d19376593fa3f4fc7e617f03a92b7fe11e9a3981cbe8f0959dbebe36225e5f53dc4492341a4863cac4ed1ee0909f3fc78ef9c3e869', 'id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'created_at': datetime.datetime(2023, 4, 17, 15, 19, 53, tzinfo=datetime.timezone.utc), 'updated_at': datetime.datetime(2023, 4, 17, 15, 19, 55, tzinfo=datetime.timezone.utc), 'tags': [], 'file': '/v2/images/b3f0dec1-7758-44a8-82e0-76ada8837428/file', 'properties': {'hw_rng_model': 'virtio', 'owner_specified.openstack.md5': '', 'owner_specified.openstack.object': 'images/cirros-0.5.2-x86_64-disk', 'owner_specified.openstack.sha256': ''}}, 'image_service': } Apr 17 15:56:54 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Cloning 60979eba-96b1-4cb3-bae1-8874a86cb411 from image b3f0dec1-7758-44a8-82e0-76ada8837428 at location (None, None). {{(pid=84696) _create_from_image /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1071}} Apr 17 15:56:54 user cinder-volume[84696]: DEBUG cinder.coordination [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Acquiring lock "cinder-b3f0dec1-7758-44a8-82e0-76ada8837428" by "_prepare_image_cache_entry" {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:192}} Apr 17 15:56:54 user cinder-volume[84696]: DEBUG cinder.coordination [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Lock "cinder-b3f0dec1-7758-44a8-82e0-76ada8837428" acquired by "_prepare_image_cache_entry" :: waited 0.039s {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:196}} Apr 17 15:56:54 user cinder-volume[84696]: DEBUG cinder.image.cache [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Found image-volume cache entry: {'id': 1, 'image_id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'volume_id': '006f20ae-5306-44fe-afdf-e433f79284e8', 'host': 'user@joviandss-2#joviandss-2', 'size': 1, 'image_updated_at': datetime.datetime(2023, 4, 17, 15, 19, 55), 'last_used': datetime.datetime(2023, 4, 17, 15, 56, 54, 360652)}. {{(pid=84696) get_entry /opt/stack/cinder/cinder/image/cache.py:76}} Apr 17 15:56:54 user cinder-volume[84696]: DEBUG cinder.image.cache [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Image-volume cache entry image_update_at = 2023-04-17 15:19:55+00:00, requested image updated_at = 2023-04-17 15:19:55+00:00. {{(pid=84696) _should_update_entry /opt/stack/cinder/cinder/image/cache.py:259}} Apr 17 15:56:54 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Found cache entry for image = b3f0dec1-7758-44a8-82e0-76ada8837428 on host user@joviandss-2#joviandss-2. {{(pid=84696) _prepare_image_cache_entry /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:886}} Apr 17 15:56:54 user cinder-volume[84696]: DEBUG cinder.coordination [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Lock "cinder-b3f0dec1-7758-44a8-82e0-76ada8837428" released by "_prepare_image_cache_entry" :: held 0.015s {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:207}} Apr 17 15:56:54 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Attempting to retrieve cache entry for image = b3f0dec1-7758-44a8-82e0-76ada8837428 on host user@joviandss-2#joviandss-2. {{(pid=84696) _create_from_image_cache /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:827}} Apr 17 15:56:54 user cinder-volume[84696]: DEBUG cinder.image.cache [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Found image-volume cache entry: {'id': 1, 'image_id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'volume_id': '006f20ae-5306-44fe-afdf-e433f79284e8', 'host': 'user@joviandss-2#joviandss-2', 'size': 1, 'image_updated_at': datetime.datetime(2023, 4, 17, 15, 19, 55), 'last_used': datetime.datetime(2023, 4, 17, 15, 56, 54, 375032)}. {{(pid=84696) get_entry /opt/stack/cinder/cinder/image/cache.py:76}} Apr 17 15:56:54 user cinder-volume[84696]: DEBUG cinder.image.cache [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Image-volume cache entry image_update_at = 2023-04-17 15:19:55+00:00, requested image updated_at = 2023-04-17 15:19:55+00:00. {{(pid=84696) _should_update_entry /opt/stack/cinder/cinder/image/cache.py:259}} Apr 17 15:56:54 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Creating from source image-volume 006f20ae-5306-44fe-afdf-e433f79284e8 {{(pid=84696) _create_from_image_cache /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:841}} Apr 17 15:56:54 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] cloned volume 006f20ae-5306-44fe-afdf-e433f79284e8 to 75163b78-31b2-4676-b93b-542a8ea784c1 {{(pid=84696) create_cloned_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:521}} Apr 17 15:56:54 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] cloning v_006f20ae-5306-44fe-afdf-e433f79284e8 to v_75163b78-31b2-4676-b93b-542a8ea784c1 {{(pid=84696) _clone_object /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:456}} Apr 17 15:56:54 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] create snapshot v_75163b78-31b2-4676-b93b-542a8ea784c1 {{(pid=84696) create_snapshot /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:635}} Apr 17 15:56:54 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:56:54 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:56:55 user cinder-volume[84696]: DEBUG cinder.coordination [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Lock "cinder-b3f0dec1-7758-44a8-82e0-76ada8837428" acquired by "_prepare_image_cache_entry" :: waited 1.024s {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:196}} Apr 17 15:56:55 user cinder-volume[84696]: DEBUG cinder.image.cache [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Found image-volume cache entry: {'id': 1, 'image_id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'volume_id': '006f20ae-5306-44fe-afdf-e433f79284e8', 'host': 'user@joviandss-2#joviandss-2', 'size': 1, 'image_updated_at': datetime.datetime(2023, 4, 17, 15, 19, 55), 'last_used': datetime.datetime(2023, 4, 17, 15, 56, 55, 380849)}. {{(pid=84696) get_entry /opt/stack/cinder/cinder/image/cache.py:76}} Apr 17 15:56:55 user cinder-volume[84696]: DEBUG cinder.image.cache [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Image-volume cache entry image_update_at = 2023-04-17 15:19:55+00:00, requested image updated_at = 2023-04-17 15:19:55+00:00. {{(pid=84696) _should_update_entry /opt/stack/cinder/cinder/image/cache.py:259}} Apr 17 15:56:55 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Found cache entry for image = b3f0dec1-7758-44a8-82e0-76ada8837428 on host user@joviandss-2#joviandss-2. {{(pid=84696) _prepare_image_cache_entry /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:886}} Apr 17 15:56:55 user cinder-volume[84696]: DEBUG cinder.coordination [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Lock "cinder-b3f0dec1-7758-44a8-82e0-76ada8837428" released by "_prepare_image_cache_entry" :: held 0.013s {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:207}} Apr 17 15:56:55 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Attempting to retrieve cache entry for image = b3f0dec1-7758-44a8-82e0-76ada8837428 on host user@joviandss-2#joviandss-2. {{(pid=84696) _create_from_image_cache /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:827}} Apr 17 15:56:55 user cinder-volume[84696]: DEBUG cinder.image.cache [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Found image-volume cache entry: {'id': 1, 'image_id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'volume_id': '006f20ae-5306-44fe-afdf-e433f79284e8', 'host': 'user@joviandss-2#joviandss-2', 'size': 1, 'image_updated_at': datetime.datetime(2023, 4, 17, 15, 19, 55), 'last_used': datetime.datetime(2023, 4, 17, 15, 56, 55, 393719)}. {{(pid=84696) get_entry /opt/stack/cinder/cinder/image/cache.py:76}} Apr 17 15:56:55 user cinder-volume[84696]: DEBUG cinder.image.cache [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Image-volume cache entry image_update_at = 2023-04-17 15:19:55+00:00, requested image updated_at = 2023-04-17 15:19:55+00:00. {{(pid=84696) _should_update_entry /opt/stack/cinder/cinder/image/cache.py:259}} Apr 17 15:56:55 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Creating from source image-volume 006f20ae-5306-44fe-afdf-e433f79284e8 {{(pid=84696) _create_from_image_cache /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:841}} Apr 17 15:56:55 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] cloned volume 006f20ae-5306-44fe-afdf-e433f79284e8 to 60979eba-96b1-4cb3-bae1-8874a86cb411 {{(pid=84696) create_cloned_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:521}} Apr 17 15:56:55 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] cloning v_006f20ae-5306-44fe-afdf-e433f79284e8 to v_60979eba-96b1-4cb3-bae1-8874a86cb411 {{(pid=84696) _clone_object /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:456}} Apr 17 15:56:55 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] create snapshot v_60979eba-96b1-4cb3-bae1-8874a86cb411 {{(pid=84696) create_snapshot /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:635}} Apr 17 15:56:55 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:56:55 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:56:56 user cinder-volume[84696]: DEBUG cinder.utils [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.502(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:56:56 user cinder-volume[84696]: DEBUG cinder.utils [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:56:56 user cinder-volume[84696]: DEBUG cinder.utils [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.923(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:56:56 user cinder-volume[84696]: DEBUG cinder.utils [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_60979eba-96b1-4cb3-bae1-8874a86cb411/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:56:56 user cinder-volume[84696]: DEBUG cinder.utils [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.933(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:56:56 user cinder-volume[84696]: DEBUG cinder.utils [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_75163b78-31b2-4676-b93b-542a8ea784c1/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:56:56 user cinder-volume[84696]: DEBUG cinder.utils [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 1.490(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:56:56 user cinder-volume[84696]: DEBUG cinder.utils [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:56:59 user cinder-volume[84696]: DEBUG cinder.utils [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.165(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:56:59 user cinder-volume[84696]: DEBUG cinder.utils [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_b29a7592-128a-41ee-a8e2-57780dee33e7/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:56:59 user cinder-volume[84696]: DEBUG cinder.utils [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 11.989(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:56:59 user cinder-volume[84696]: DEBUG cinder.utils [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:56:59 user cinder-volume[84696]: DEBUG cinder.utils [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.997(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:56:59 user cinder-volume[84696]: DEBUG cinder.utils [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_60979eba-96b1-4cb3-bae1-8874a86cb411/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:56:59 user cinder-volume[84696]: DEBUG cinder.utils [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.579(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:56:59 user cinder-volume[84696]: DEBUG cinder.utils [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:56:59 user cinder-volume[84696]: DEBUG cinder.utils [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 136.755(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:56:59 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:56:59 user cinder-volume[84696]: INFO cinder.message.api [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Creating message record for request_id = req-4381eeaf-f88c-49c3-a64f-a647fc1596cd Apr 17 15:56:59 user cinder-volume[84696]: DEBUG cinder.utils [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 4.569(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:56:59 user cinder-volume[84696]: DEBUG cinder.utils [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:56:59 user cinder-volume[84696]: DEBUG cinder.utils [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.015(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:56:59 user cinder-volume[84696]: DEBUG cinder.utils [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_75163b78-31b2-4676-b93b-542a8ea784c1/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:56:59 user cinder-volume[84696]: DEBUG cinder.utils [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 136.934(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:56:59 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_2ed5b8aa-fb48-4255-b6d5-f0f84f733834/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:56:59 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (7add7189-fa1b-4ede-9dc3-2da161948fbc) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 15:56:59 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (660f6bd8-d8e2-4441-9b4c-f30e552b77bc) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:56:59 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (660f6bd8-d8e2-4441-9b4c-f30e552b77bc) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:56:59 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Flow 'volume_create_manager' (2ff98574-8f57-4cce-a4ee-2226cedf8965) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 15:57:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 15:57:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 15:57:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 15:57:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 15:57:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 15:57:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 15:57:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 15:57:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 15:57:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 15:57:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 15:57:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 15:57:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 15:57:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 15:57:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 15:57:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 15:57:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 15:57:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 15:57:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 15:57:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 15:57:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 15:57:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 15:57:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 15:57:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 15:57:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 15:57:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:57:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 15:57:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 15:57:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 15:57:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 15:57:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 15:57:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 15:57:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 15:57:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 15:57:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 15:57:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 15:57:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 15:57:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 15:57:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 15:57:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 15:57:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 15:57:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 15:57:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 15:57:00 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 15:57:00 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=26,cinder:UPDATE=4,cinder:INSERT=3 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:57:00 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=24,cinder:UPDATE=13,cinder:INSERT=2 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:57:00 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (7a7883be-618b-4ab2-b753-55d49f84954b) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 15:57:00 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 15:57:00 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:50:05Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-ServerActionsTestJSON-volume-872354177',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=2ed5b8aa-fb48-4255-b6d5-f0f84f733834,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='cb5f6b5ed2fc43b69c47e1fc18646545',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:54:43Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:54:43Z,use_quota=True,user_id='638a4f535e9740fc9d810639a5eb24d7',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 15:57:00 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:50:05Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-ServerActionsTestJSON-volume-872354177',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=2ed5b8aa-fb48-4255-b6d5-f0f84f733834,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='cb5f6b5ed2fc43b69c47e1fc18646545',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:54:43Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:54:43Z,use_quota=True,user_id='638a4f535e9740fc9d810639a5eb24d7',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(2ed5b8aa-fb48-4255-b6d5-f0f84f733834),volume_id=2ed5b8aa-fb48-4255-b6d5-f0f84f733834,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': '2ed5b8aa-fb48-4255-b6d5-f0f84f733834', 'volume_name': 'volume-2ed5b8aa-fb48-4255-b6d5-f0f84f733834', 'volume_size': 1}} Apr 17 15:57:00 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': '2ed5b8aa-fb48-4255-b6d5-f0f84f733834', 'volume': {'id': '2ed5b8aa-fb48-4255-b6d5-f0f84f733834', '_name_id': None, 'ec2_id': None, 'user_id': '638a4f535e9740fc9d810639a5eb24d7', 'project_id': 'cb5f6b5ed2fc43b69c47e1fc18646545', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-ServerActionsTestJSON-volume-872354177', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'admin_metadata': {}, 'glance_metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T15:50:04.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-2ed5b8aa-fb48-4255-b6d5-f0f84f733834', 'name_id': '2ed5b8aa-fb48-4255-b6d5-f0f84f733834', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-ServerActionsTestJSON-volume-872354177', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': 'cb5f6b5ed2fc43b69c47e1fc18646545', 'qos_specs': None, 'reservations': ['6016bd49-cebc-4e3e-a82c-96aaf7390191', '75d52529-e09a-471c-b975-4ba7744d76df', 'f897feca-3a4b-4df1-9b1f-3c8ce372c8c8', 'e5402891-ace1-4d31-9e37-0068e08e9ec2'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': '638a4f535e9740fc9d810639a5eb24d7', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-ServerActionsTestJSON-volume-872354177', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': 'cb5f6b5ed2fc43b69c47e1fc18646545', 'qos_specs': None, 'reservations': ['6016bd49-cebc-4e3e-a82c-96aaf7390191', '75d52529-e09a-471c-b975-4ba7744d76df', 'f897feca-3a4b-4df1-9b1f-3c8ce372c8c8', 'e5402891-ace1-4d31-9e37-0068e08e9ec2'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': '638a4f535e9740fc9d810639a5eb24d7', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': '638a4f535e9740fc9d810639a5eb24d7', 'metadata': {}, 'qos_specs': None, 'retry': {'num_attempts': 3, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']}}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(2ed5b8aa-fb48-4255-b6d5-f0f84f733834),volume_id=2ed5b8aa-fb48-4255-b6d5-f0f84f733834,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:50:05Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-ServerActionsTestJSON-volume-872354177',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=2ed5b8aa-fb48-4255-b6d5-f0f84f733834,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='cb5f6b5ed2fc43b69c47e1fc18646545',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:54:43Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:54:43Z,use_quota=True,user_id='638a4f535e9740fc9d810639a5eb24d7',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 15:57:00 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:50:05Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-ServerActionsTestJSON-volume-872354177',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=2ed5b8aa-fb48-4255-b6d5-f0f84f733834,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='cb5f6b5ed2fc43b69c47e1fc18646545',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:54:43Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:54:43Z,use_quota=True,user_id='638a4f535e9740fc9d810639a5eb24d7',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:50:05Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-ServerActionsTestJSON-volume-872354177',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=2ed5b8aa-fb48-4255-b6d5-f0f84f733834,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='cb5f6b5ed2fc43b69c47e1fc18646545',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:54:43Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:54:43Z,use_quota=True,user_id='638a4f535e9740fc9d810639a5eb24d7',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 15:57:00 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 15:57:00 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 15:57:00 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:57:00 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 15:57:00 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 15:57:00 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 15:57:00 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 15:57:00 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 15:57:00 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 15:57:00 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 15:57:00 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 15:57:00 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 15:57:00 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 15:57:00 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 15:57:00 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 15:57:00 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 15:57:00 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 15:57:00 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 15:57:00 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 15:57:00 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 15:57:00 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (7a7883be-618b-4ab2-b753-55d49f84954b) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:57:00 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (7a7883be-618b-4ab2-b753-55d49f84954b) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:57:00 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (b719b5c1-186a-4f76-b501-9f9936ea1070) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:57:00 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (b719b5c1-186a-4f76-b501-9f9936ea1070) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:57:00 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (9cbe0274-a55b-417c-a526-722649caa74e) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:57:00 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (9cbe0274-a55b-417c-a526-722649caa74e) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:57:00 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (4dc5a06a-4105-45c6-b762-a06c80fed045) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:57:00 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Updating volume 2ed5b8aa-fb48-4255-b6d5-f0f84f733834 with {'scheduled_at': datetime.datetime(2023, 4, 17, 15, 57, 0, 66734), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 15:57:00 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Volume 2ed5b8aa-fb48-4255-b6d5-f0f84f733834: re-scheduling SchedulerAPI.create_volume attempt 3 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 15:57:00 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Volume 2ed5b8aa-fb48-4255-b6d5-f0f84f733834: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 15:57:00 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] deleating volume v_2ed5b8aa-fb48-4255-b6d5-f0f84f733834 {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 15:57:00 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] get snapshots for volume v_2ed5b8aa-fb48-4255-b6d5-f0f84f733834 {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 15:57:00 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_2ed5b8aa-fb48-4255-b6d5-f0f84f733834/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:57:00 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_2ed5b8aa-fb48-4255-b6d5-f0f84f733834/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:57:03 user cinder-volume[84696]: DEBUG cinder.utils [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.943(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:57:03 user cinder-volume[84696]: DEBUG cinder.utils [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_2ed5b8aa-fb48-4255-b6d5-f0f84f733834/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:57:06 user cinder-volume[84696]: DEBUG cinder.utils [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 10.787(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:57:06 user cinder-volume[84696]: DEBUG cinder.utils [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:57:06 user cinder-volume[84696]: DEBUG cinder.utils [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 40.227(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:57:06 user cinder-volume[84696]: DEBUG cinder.utils [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:57:06 user cinder-volume[84696]: DEBUG cinder.utils [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.105(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:57:06 user cinder-volume[84696]: DEBUG cinder.utils [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_2ed5b8aa-fb48-4255-b6d5-f0f84f733834/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:57:06 user cinder-volume[84696]: DEBUG cinder.utils [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.236(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:57:06 user cinder-volume[84696]: DEBUG cinder.utils [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_75163b78-31b2-4676-b93b-542a8ea784c1/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:57:06 user cinder-volume[84696]: DEBUG cinder.utils [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.229(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:57:06 user cinder-volume[84696]: DEBUG cinder.utils [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_60979eba-96b1-4cb3-bae1-8874a86cb411/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:57:06 user cinder-volume[84696]: DEBUG cinder.utils [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 11.810(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:57:06 user cinder-volume[84696]: DEBUG cinder.utils [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:57:09 user cinder-volume[84696]: DEBUG cinder.utils [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.267(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:57:09 user cinder-volume[84696]: DEBUG cinder.utils [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:57:09 user cinder-volume[84696]: DEBUG cinder.utils [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.446(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:57:09 user cinder-volume[84696]: DEBUG cinder.utils [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_b29a7592-128a-41ee-a8e2-57780dee33e7/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:57:12 user cinder-volume[84696]: DEBUG cinder.utils [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.223(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:57:12 user cinder-volume[84696]: DEBUG cinder.utils [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_2ed5b8aa-fb48-4255-b6d5-f0f84f733834/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:57:12 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 40.567(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:57:12 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:57:14 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=3,cinder:UPDATE=2 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:57:15 user cinder-volume[84696]: DEBUG cinder.utils [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 140.003(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:57:15 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_cdedb820-e1c7-4b86-8286-43b3640b3a46/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:57:15 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (03f8a8f7-ceef-40f0-93f7-9b4d6ce5a8ac) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 15:57:15 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (268945a7-b980-4e7b-a0ed-3456111a0c53) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:57:15 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (268945a7-b980-4e7b-a0ed-3456111a0c53) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:57:15 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Flow 'volume_create_manager' (f71e7e7d-84a9-4c36-a4dc-68f30540149b) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 15:57:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-9f132425-c80e-4ea2-bea2-42fd2887f8a2 tempest-ServerRescueNegativeTestJSON-1346397943 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 15:57:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 15:57:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 15:57:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 15:57:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 15:57:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 15:57:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 15:57:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 15:57:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 15:57:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 15:57:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 15:57:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 15:57:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 15:57:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 15:57:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 15:57:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 15:57:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 15:57:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 15:57:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 15:57:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 15:57:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 15:57:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 15:57:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 15:57:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 15:57:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:57:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 15:57:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 15:57:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 15:57:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 15:57:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 15:57:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 15:57:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 15:57:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 15:57:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 15:57:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 15:57:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 15:57:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 15:57:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 15:57:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 15:57:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 15:57:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 15:57:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 15:57:15 user cinder-volume[84696]: DEBUG cinder.utils [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.447(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:57:15 user cinder-volume[84696]: DEBUG cinder.utils [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_60979eba-96b1-4cb3-bae1-8874a86cb411/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:57:15 user cinder-volume[84696]: DEBUG cinder.utils [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.456(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:57:15 user cinder-volume[84696]: DEBUG cinder.utils [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_75163b78-31b2-4676-b93b-542a8ea784c1/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:57:15 user cinder-volume[84696]: DEBUG cinder.utils [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 20.012(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:57:15 user cinder-volume[84696]: DEBUG cinder.utils [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:57:15 user cinder-volume[84696]: DEBUG cinder.utils [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.031(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:57:15 user cinder-volume[84696]: DEBUG cinder.utils [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:57:21 user cinder-volume[84696]: DEBUG cinder.utils [req-778c62be-8963-4e6d-97f1-f456b759ce16 req-6d72b587-594c-4309-b424-7292ebb80925 tempest-AttachVolumeShelveTestJSON-1745617049 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 72.457(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:57:21 user cinder-volume[84696]: DEBUG cinder.utils [req-778c62be-8963-4e6d-97f1-f456b759ce16 req-6d72b587-594c-4309-b424-7292ebb80925 tempest-AttachVolumeShelveTestJSON-1745617049 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:13bf568b-585c-4556-ab78-e9930ad50214/luns/v_13bf568b-585c-4556-ab78-e9930ad50214 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:57:21 user cinder-volume[84696]: DEBUG cinder.utils [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.570(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:57:21 user cinder-volume[84696]: DEBUG cinder.utils [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_2ed5b8aa-fb48-4255-b6d5-f0f84f733834/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:57:28 user cinder-volume[84696]: DEBUG cinder.utils [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 40.339(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:57:28 user cinder-volume[84696]: DEBUG cinder.utils [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:57:28 user cinder-volume[84696]: DEBUG cinder.utils [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 40.519(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:57:28 user cinder-volume[84696]: DEBUG cinder.utils [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_b29a7592-128a-41ee-a8e2-57780dee33e7/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:57:28 user cinder-volume[84696]: DEBUG cinder.utils [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 75.188(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:57:28 user cinder-volume[84696]: DEBUG cinder.utils [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_4a2d1c00-a134-4f73-8508-86f35b58acf7/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:57:34 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=8,cinder:INSERT=1,cinder:UPDATE=3 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:57:34 user cinder-volume[84696]: DEBUG cinder.utils [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.084(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:57:34 user cinder-volume[84696]: DEBUG cinder.utils [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:57:34 user cinder-volume[84696]: DEBUG cinder.utils [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 40.103(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:57:34 user cinder-volume[84696]: DEBUG cinder.utils [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:57:34 user cinder-volume[84696]: DEBUG cinder.utils [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 40.523(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:57:34 user cinder-volume[84696]: DEBUG cinder.utils [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_60979eba-96b1-4cb3-bae1-8874a86cb411/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:57:34 user cinder-volume[84696]: DEBUG cinder.utils [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 40.532(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:57:34 user cinder-volume[84696]: DEBUG cinder.utils [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_75163b78-31b2-4676-b93b-542a8ea784c1/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:57:37 user cinder-volume[84696]: DEBUG cinder.utils [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 139.950(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:57:37 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:57:37 user cinder-volume[84696]: INFO cinder.message.api [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Creating message record for request_id = req-5a40883b-32f9-400a-a3f7-129f4d4677bb Apr 17 15:57:37 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 15:57:37 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (5d7fa8cf-ae8d-408c-8099-546ff3a2e3a7) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 15:57:37 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 15:57:37 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:55:17Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-AttachVolumeNegativeTest-volume-1944200504',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=87514055-5a0f-459f-b10b-438d7a13d39d,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='8586cc10142d4743874f43cdb0f37aba',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:55:18Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:55:18Z,use_quota=True,user_id='9fdc7c80b4654692a52c8a4c4149324f',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 15:57:37 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:55:17Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-AttachVolumeNegativeTest-volume-1944200504',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=87514055-5a0f-459f-b10b-438d7a13d39d,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='8586cc10142d4743874f43cdb0f37aba',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:55:18Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:55:18Z,use_quota=True,user_id='9fdc7c80b4654692a52c8a4c4149324f',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(87514055-5a0f-459f-b10b-438d7a13d39d),volume_id=87514055-5a0f-459f-b10b-438d7a13d39d,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': '87514055-5a0f-459f-b10b-438d7a13d39d', 'volume_name': 'volume-87514055-5a0f-459f-b10b-438d7a13d39d', 'volume_size': 1}} Apr 17 15:57:37 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'retry': {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']}, 'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': '87514055-5a0f-459f-b10b-438d7a13d39d', 'volume': {'id': '87514055-5a0f-459f-b10b-438d7a13d39d', '_name_id': None, 'ec2_id': None, 'user_id': '9fdc7c80b4654692a52c8a4c4149324f', 'project_id': '8586cc10142d4743874f43cdb0f37aba', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-AttachVolumeNegativeTest-volume-1944200504', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T15:55:17.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-87514055-5a0f-459f-b10b-438d7a13d39d', 'name_id': '87514055-5a0f-459f-b10b-438d7a13d39d', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-AttachVolumeNegativeTest-volume-1944200504', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '8586cc10142d4743874f43cdb0f37aba', 'qos_specs': None, 'reservations': ['1c53c030-381a-4511-90a6-073111ef116f', '7bbd5e8b-d708-4476-a281-0ce37fa87819', '438a4611-10fd-4bd4-a60e-52564e4a0b82', 'a28101d3-8126-4720-b09f-75c7afd4c0dd'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': '9fdc7c80b4654692a52c8a4c4149324f', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-AttachVolumeNegativeTest-volume-1944200504', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '8586cc10142d4743874f43cdb0f37aba', 'qos_specs': None, 'reservations': ['1c53c030-381a-4511-90a6-073111ef116f', '7bbd5e8b-d708-4476-a281-0ce37fa87819', '438a4611-10fd-4bd4-a60e-52564e4a0b82', 'a28101d3-8126-4720-b09f-75c7afd4c0dd'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': '9fdc7c80b4654692a52c8a4c4149324f', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': '9fdc7c80b4654692a52c8a4c4149324f', 'metadata': {}, 'qos_specs': None}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(87514055-5a0f-459f-b10b-438d7a13d39d),volume_id=87514055-5a0f-459f-b10b-438d7a13d39d,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:55:17Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-AttachVolumeNegativeTest-volume-1944200504',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=87514055-5a0f-459f-b10b-438d7a13d39d,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='8586cc10142d4743874f43cdb0f37aba',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:55:18Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:55:18Z,use_quota=True,user_id='9fdc7c80b4654692a52c8a4c4149324f',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 15:57:37 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:55:17Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-AttachVolumeNegativeTest-volume-1944200504',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=87514055-5a0f-459f-b10b-438d7a13d39d,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='8586cc10142d4743874f43cdb0f37aba',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:55:18Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:55:18Z,use_quota=True,user_id='9fdc7c80b4654692a52c8a4c4149324f',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:55:17Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-AttachVolumeNegativeTest-volume-1944200504',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=87514055-5a0f-459f-b10b-438d7a13d39d,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='8586cc10142d4743874f43cdb0f37aba',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:55:18Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:55:18Z,use_quota=True,user_id='9fdc7c80b4654692a52c8a4c4149324f',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 15:57:37 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 15:57:37 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 15:57:37 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:57:37 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 15:57:37 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 15:57:37 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 15:57:37 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 15:57:37 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 15:57:37 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 15:57:37 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 15:57:37 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 15:57:37 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 15:57:37 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 15:57:37 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 15:57:37 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 15:57:37 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 15:57:37 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 15:57:37 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 15:57:37 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 15:57:37 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 15:57:37 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (5d7fa8cf-ae8d-408c-8099-546ff3a2e3a7) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:57:37 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (5d7fa8cf-ae8d-408c-8099-546ff3a2e3a7) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:57:37 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (2c9b614b-ac85-42b0-a207-b9c9fd58695b) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:57:37 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (2c9b614b-ac85-42b0-a207-b9c9fd58695b) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:57:37 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (2adcf970-e94e-4020-9547-0151ea0132a4) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:57:37 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (2adcf970-e94e-4020-9547-0151ea0132a4) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:57:37 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (53008c4f-ab86-4329-87aa-c158c3e48eeb) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:57:37 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Updating volume 87514055-5a0f-459f-b10b-438d7a13d39d with {'scheduled_at': datetime.datetime(2023, 4, 17, 15, 57, 37, 629903), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 15:57:37 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Volume 87514055-5a0f-459f-b10b-438d7a13d39d: re-scheduling SchedulerAPI.create_volume attempt 1 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 15:57:37 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Volume 87514055-5a0f-459f-b10b-438d7a13d39d: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 15:57:37 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] deleating volume v_87514055-5a0f-459f-b10b-438d7a13d39d {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 15:57:37 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] get snapshots for volume v_87514055-5a0f-459f-b10b-438d7a13d39d {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 15:57:37 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_87514055-5a0f-459f-b10b-438d7a13d39d/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:57:37 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_87514055-5a0f-459f-b10b-438d7a13d39d/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:57:37 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Volume reschedule parameters: True retry: {'num_attempts': 2, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 15:57:37 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Flow 'volume_create_manager' (7524b499-f0fc-474f-ab3d-77e348250070) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:57:37 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (6c017e80-6a0a-40ce-9c74-78f17c2dc622) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:57:37 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (6c017e80-6a0a-40ce-9c74-78f17c2dc622) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:55:17Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-AttachVolumeNegativeTest-volume-1944200504',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=87514055-5a0f-459f-b10b-438d7a13d39d,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='8586cc10142d4743874f43cdb0f37aba',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:57:38Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:57:38Z,use_quota=True,user_id='9fdc7c80b4654692a52c8a4c4149324f',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:57:37 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (9db71fee-70ac-4ddc-bef2-8b370efd6663) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:57:37 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (9db71fee-70ac-4ddc-bef2-8b370efd6663) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:57:37 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (1fb0c416-a2fa-4f26-aa3f-5597a03192af) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:57:37 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (1fb0c416-a2fa-4f26-aa3f-5597a03192af) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': '87514055-5a0f-459f-b10b-438d7a13d39d', 'volume_name': 'volume-87514055-5a0f-459f-b10b-438d7a13d39d', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:57:37 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (47f88e3e-d4e7-46dc-8d02-0198752e12f5) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:57:37 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (47f88e3e-d4e7-46dc-8d02-0198752e12f5) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:57:37 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (1079e44a-eaf6-401d-adfc-6a2732d9bf6c) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:57:37 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Volume 87514055-5a0f-459f-b10b-438d7a13d39d: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-87514055-5a0f-459f-b10b-438d7a13d39d', 'volume_size': 1} Apr 17 15:57:37 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] creating volume v_87514055-5a0f-459f-b10b-438d7a13d39d. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 15:57:37 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] create volume {'name': 'v_87514055-5a0f-459f-b10b-438d7a13d39d', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 15:57:37 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:57:37 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:57:40 user cinder-volume[84696]: DEBUG cinder.utils [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.798(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:57:40 user cinder-volume[84696]: DEBUG cinder.utils [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:57:40 user cinder-volume[84696]: DEBUG cinder.utils [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.979(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:57:40 user cinder-volume[84696]: DEBUG cinder.utils [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_87514055-5a0f-459f-b10b-438d7a13d39d/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:57:40 user cinder-volume[84696]: DEBUG cinder.utils [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 40.546(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:57:40 user cinder-volume[84696]: DEBUG cinder.utils [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_2ed5b8aa-fb48-4255-b6d5-f0f84f733834/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:57:40 user cinder-volume[84696]: DEBUG cinder.utils [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 74.671(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:57:40 user cinder-volume[84696]: DEBUG cinder.utils [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:57:40 user cinder-volume[84696]: DEBUG cinder.utils [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 137.927(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:57:40 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:57:40 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (4276b8a4-4d38-42bb-bbe6-9f6dc75a30f0) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 15:57:40 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 15:57:40 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:55:22Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-tempest.common.compute-volume-826817831',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=14aa38ea-7cbb-48f7-a9aa-ffcc7b66faa0,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='cb5f6b5ed2fc43b69c47e1fc18646545',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:55:22Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:55:22Z,use_quota=True,user_id='638a4f535e9740fc9d810639a5eb24d7',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 15:57:40 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:55:22Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-tempest.common.compute-volume-826817831',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=14aa38ea-7cbb-48f7-a9aa-ffcc7b66faa0,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='cb5f6b5ed2fc43b69c47e1fc18646545',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:55:22Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:55:22Z,use_quota=True,user_id='638a4f535e9740fc9d810639a5eb24d7',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=b3f0dec1-7758-44a8-82e0-76ada8837428,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(14aa38ea-7cbb-48f7-a9aa-ffcc7b66faa0),volume_id=14aa38ea-7cbb-48f7-a9aa-ffcc7b66faa0,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'image', 'volume_id': '14aa38ea-7cbb-48f7-a9aa-ffcc7b66faa0', 'volume_name': 'volume-14aa38ea-7cbb-48f7-a9aa-ffcc7b66faa0', 'volume_size': 1, 'image_id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'image_location': (None, None), 'image_meta': {'name': 'cirros-0.5.2-x86_64-disk', 'disk_format': 'qcow2', 'container_format': 'bare', 'visibility': 'public', 'size': 16300544, 'virtual_size': 117440512, 'status': 'active', 'checksum': 'b874c39491a2377b8490f5f1e89761a4', 'protected': False, 'min_ram': 0, 'min_disk': 0, 'owner': 'a43c211b4ed542538799349489b1d884', 'os_hidden': False, 'os_hash_algo': 'sha512', 'os_hash_value': '6b813aa46bb90b4da216a4d19376593fa3f4fc7e617f03a92b7fe11e9a3981cbe8f0959dbebe36225e5f53dc4492341a4863cac4ed1ee0909f3fc78ef9c3e869', 'id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'created_at': datetime.datetime(2023, 4, 17, 15, 19, 53, tzinfo=datetime.timezone.utc), 'updated_at': datetime.datetime(2023, 4, 17, 15, 19, 55, tzinfo=datetime.timezone.utc), 'tags': [], 'file': '/v2/images/b3f0dec1-7758-44a8-82e0-76ada8837428/file', 'properties': {'hw_rng_model': 'virtio', 'owner_specified.openstack.md5': '', 'owner_specified.openstack.object': 'images/cirros-0.5.2-x86_64-disk', 'owner_specified.openstack.sha256': ''}}, 'image_service': }} Apr 17 15:57:40 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'retry': {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']}, 'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'snapshot_id': None, 'source_volid': None, 'volume_id': '14aa38ea-7cbb-48f7-a9aa-ffcc7b66faa0', 'volume': {'id': '14aa38ea-7cbb-48f7-a9aa-ffcc7b66faa0', '_name_id': None, 'ec2_id': None, 'user_id': '638a4f535e9740fc9d810639a5eb24d7', 'project_id': 'cb5f6b5ed2fc43b69c47e1fc18646545', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-tempest.common.compute-volume-826817831', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T15:55:22.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-14aa38ea-7cbb-48f7-a9aa-ffcc7b66faa0', 'name_id': '14aa38ea-7cbb-48f7-a9aa-ffcc7b66faa0', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-tempest.common.compute-volume-826817831', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': 'cb5f6b5ed2fc43b69c47e1fc18646545', 'qos_specs': None, 'reservations': ['c3b21a3e-ae73-41f9-8d32-500bd32a5696', '477b1752-2fec-4e18-835a-e7ca57617158', '2e94e8e6-37cc-46c3-b87c-ae7717df65c2', 'a5c6c20e-c7ee-491f-b333-35f64711994a'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': '638a4f535e9740fc9d810639a5eb24d7', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-tempest.common.compute-volume-826817831', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': 'cb5f6b5ed2fc43b69c47e1fc18646545', 'qos_specs': None, 'reservations': ['c3b21a3e-ae73-41f9-8d32-500bd32a5696', '477b1752-2fec-4e18-835a-e7ca57617158', '2e94e8e6-37cc-46c3-b87c-ae7717df65c2', 'a5c6c20e-c7ee-491f-b333-35f64711994a'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': '638a4f535e9740fc9d810639a5eb24d7', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': '638a4f535e9740fc9d810639a5eb24d7', 'metadata': {}, 'qos_specs': None}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=b3f0dec1-7758-44a8-82e0-76ada8837428,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(14aa38ea-7cbb-48f7-a9aa-ffcc7b66faa0),volume_id=14aa38ea-7cbb-48f7-a9aa-ffcc7b66faa0,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:55:22Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-tempest.common.compute-volume-826817831',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=14aa38ea-7cbb-48f7-a9aa-ffcc7b66faa0,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='cb5f6b5ed2fc43b69c47e1fc18646545',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:55:22Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:55:22Z,use_quota=True,user_id='638a4f535e9740fc9d810639a5eb24d7',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 15:57:40 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:55:22Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-tempest.common.compute-volume-826817831',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=14aa38ea-7cbb-48f7-a9aa-ffcc7b66faa0,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='cb5f6b5ed2fc43b69c47e1fc18646545',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:55:22Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:55:22Z,use_quota=True,user_id='638a4f535e9740fc9d810639a5eb24d7',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:55:22Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-tempest.common.compute-volume-826817831',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=14aa38ea-7cbb-48f7-a9aa-ffcc7b66faa0,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='cb5f6b5ed2fc43b69c47e1fc18646545',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:55:22Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:55:22Z,use_quota=True,user_id='638a4f535e9740fc9d810639a5eb24d7',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 15:57:40 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 15:57:40 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 15:57:40 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:57:40 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 15:57:40 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1237, in execute Apr 17 15:57:40 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_from_image(context, Apr 17 15:57:40 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/utils.py", line 670, in _wrapper Apr 17 15:57:40 user cinder-volume[84696]: ERROR cinder.volume.manager return r(f, *args, **kwargs) Apr 17 15:57:40 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/tenacity/__init__.py", line 423, in __call__ Apr 17 15:57:40 user cinder-volume[84696]: ERROR cinder.volume.manager do = self.iter(retry_state=retry_state) Apr 17 15:57:40 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/tenacity/__init__.py", line 360, in iter Apr 17 15:57:40 user cinder-volume[84696]: ERROR cinder.volume.manager return fut.result() Apr 17 15:57:40 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/lib/python3.10/concurrent/futures/_base.py", line 451, in result Apr 17 15:57:40 user cinder-volume[84696]: ERROR cinder.volume.manager return self.__get_result() Apr 17 15:57:40 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result Apr 17 15:57:40 user cinder-volume[84696]: ERROR cinder.volume.manager raise self._exception Apr 17 15:57:40 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/tenacity/__init__.py", line 426, in __call__ Apr 17 15:57:40 user cinder-volume[84696]: ERROR cinder.volume.manager result = fn(*args, **kwargs) Apr 17 15:57:40 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1125, in _create_from_image Apr 17 15:57:40 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_from_image_cache_or_download( Apr 17 15:57:40 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 944, in _create_from_image_cache_or_download Apr 17 15:57:40 user cinder-volume[84696]: ERROR cinder.volume.manager model_update, cloned = self._create_from_image_cache( Apr 17 15:57:40 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 843, in _create_from_image_cache Apr 17 15:57:40 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_from_source_volume( Apr 17 15:57:40 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 659, in _create_from_source_volume Apr 17 15:57:40 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self.driver.create_cloned_volume(volume, srcvol_ref) Apr 17 15:57:40 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 525, in create_cloned_volume Apr 17 15:57:40 user cinder-volume[84696]: ERROR cinder.volume.manager self._clone_object(vname, cvname) Apr 17 15:57:40 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 461, in _clone_object Apr 17 15:57:40 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_snapshot(oname, coname) Apr 17 15:57:40 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 639, in create_snapshot Apr 17 15:57:40 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201, 204): Apr 17 15:57:40 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 15:57:40 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 15:57:40 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (4276b8a4-4d38-42bb-bbe6-9f6dc75a30f0) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:57:40 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (4276b8a4-4d38-42bb-bbe6-9f6dc75a30f0) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:57:40 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (5da83909-47f3-41ae-a77c-961c4e1acdcd) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:57:40 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (5da83909-47f3-41ae-a77c-961c4e1acdcd) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:57:40 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (b905c8f2-d782-4f2c-a99b-8fc50a113d6d) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:57:40 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (b905c8f2-d782-4f2c-a99b-8fc50a113d6d) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:57:40 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (a72c6b3f-14ea-4ea4-bded-4d9494ce46f3) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:57:40 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Updating volume 14aa38ea-7cbb-48f7-a9aa-ffcc7b66faa0 with {'scheduled_at': datetime.datetime(2023, 4, 17, 15, 57, 40, 687413), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 15:57:40 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Volume 14aa38ea-7cbb-48f7-a9aa-ffcc7b66faa0: re-scheduling SchedulerAPI.create_volume attempt 1 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 15:57:40 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Volume 14aa38ea-7cbb-48f7-a9aa-ffcc7b66faa0: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 15:57:40 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] deleating volume v_14aa38ea-7cbb-48f7-a9aa-ffcc7b66faa0 {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 15:57:40 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] get snapshots for volume v_14aa38ea-7cbb-48f7-a9aa-ffcc7b66faa0 {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 15:57:40 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_14aa38ea-7cbb-48f7-a9aa-ffcc7b66faa0/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:57:40 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_14aa38ea-7cbb-48f7-a9aa-ffcc7b66faa0/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:57:40 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Volume reschedule parameters: True retry: {'num_attempts': 2, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 15:57:40 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Flow 'volume_create_manager' (ee047975-88bc-49e6-ad52-dc7b144acb87) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:57:40 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (f17af5f5-9351-435c-993f-6146b54ea7ec) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:57:40 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (f17af5f5-9351-435c-993f-6146b54ea7ec) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:55:22Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-tempest.common.compute-volume-826817831',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=14aa38ea-7cbb-48f7-a9aa-ffcc7b66faa0,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='cb5f6b5ed2fc43b69c47e1fc18646545',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:57:41Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:57:41Z,use_quota=True,user_id='638a4f535e9740fc9d810639a5eb24d7',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:57:40 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (7ad071b2-5c5a-4e67-a1e3-1c6ca780c749) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:57:40 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (7ad071b2-5c5a-4e67-a1e3-1c6ca780c749) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:57:40 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (7f53bd87-cac2-4a04-b3b9-6350e800247b) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:57:40 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (7f53bd87-cac2-4a04-b3b9-6350e800247b) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'image', 'volume_id': '14aa38ea-7cbb-48f7-a9aa-ffcc7b66faa0', 'volume_name': 'volume-14aa38ea-7cbb-48f7-a9aa-ffcc7b66faa0', 'volume_size': 1, 'image_id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'image_location': (None, None), 'image_meta': {'name': 'cirros-0.5.2-x86_64-disk', 'disk_format': 'qcow2', 'container_format': 'bare', 'visibility': 'public', 'size': 16300544, 'virtual_size': 117440512, 'status': 'active', 'checksum': 'b874c39491a2377b8490f5f1e89761a4', 'protected': False, 'min_ram': 0, 'min_disk': 0, 'owner': 'a43c211b4ed542538799349489b1d884', 'os_hidden': False, 'os_hash_algo': 'sha512', 'os_hash_value': '6b813aa46bb90b4da216a4d19376593fa3f4fc7e617f03a92b7fe11e9a3981cbe8f0959dbebe36225e5f53dc4492341a4863cac4ed1ee0909f3fc78ef9c3e869', 'id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'created_at': datetime.datetime(2023, 4, 17, 15, 19, 53, tzinfo=datetime.timezone.utc), 'updated_at': datetime.datetime(2023, 4, 17, 15, 19, 55, tzinfo=datetime.timezone.utc), 'tags': [], 'file': '/v2/images/b3f0dec1-7758-44a8-82e0-76ada8837428/file', 'properties': {'hw_rng_model': 'virtio', 'owner_specified.openstack.md5': '', 'owner_specified.openstack.object': 'images/cirros-0.5.2-x86_64-disk', 'owner_specified.openstack.sha256': ''}}, 'image_service': }' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:57:40 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (7169e5d6-8d5d-4c85-b533-c2de1fe81112) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:57:40 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (7169e5d6-8d5d-4c85-b533-c2de1fe81112) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:57:40 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (6293c97f-7299-43fd-99bf-ee8e029e0520) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:57:40 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Volume 14aa38ea-7cbb-48f7-a9aa-ffcc7b66faa0: being created as image with specification: {'status': 'creating', 'volume_name': 'volume-14aa38ea-7cbb-48f7-a9aa-ffcc7b66faa0', 'volume_size': 1, 'image_id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'image_location': (None, None), 'image_meta': {'name': 'cirros-0.5.2-x86_64-disk', 'disk_format': 'qcow2', 'container_format': 'bare', 'visibility': 'public', 'size': 16300544, 'virtual_size': 117440512, 'status': 'active', 'checksum': 'b874c39491a2377b8490f5f1e89761a4', 'protected': False, 'min_ram': 0, 'min_disk': 0, 'owner': 'a43c211b4ed542538799349489b1d884', 'os_hidden': False, 'os_hash_algo': 'sha512', 'os_hash_value': '6b813aa46bb90b4da216a4d19376593fa3f4fc7e617f03a92b7fe11e9a3981cbe8f0959dbebe36225e5f53dc4492341a4863cac4ed1ee0909f3fc78ef9c3e869', 'id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'created_at': datetime.datetime(2023, 4, 17, 15, 19, 53, tzinfo=datetime.timezone.utc), 'updated_at': datetime.datetime(2023, 4, 17, 15, 19, 55, tzinfo=datetime.timezone.utc), 'tags': [], 'file': '/v2/images/b3f0dec1-7758-44a8-82e0-76ada8837428/file', 'properties': {'hw_rng_model': 'virtio', 'owner_specified.openstack.md5': '', 'owner_specified.openstack.object': 'images/cirros-0.5.2-x86_64-disk', 'owner_specified.openstack.sha256': ''}}, 'image_service': } Apr 17 15:57:40 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Cloning 14aa38ea-7cbb-48f7-a9aa-ffcc7b66faa0 from image b3f0dec1-7758-44a8-82e0-76ada8837428 at location (None, None). {{(pid=84696) _create_from_image /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1071}} Apr 17 15:57:40 user cinder-volume[84696]: DEBUG cinder.coordination [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Acquiring lock "cinder-b3f0dec1-7758-44a8-82e0-76ada8837428" by "_prepare_image_cache_entry" {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:192}} Apr 17 15:57:40 user cinder-volume[84696]: DEBUG cinder.coordination [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Lock "cinder-b3f0dec1-7758-44a8-82e0-76ada8837428" acquired by "_prepare_image_cache_entry" :: waited 0.010s {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:196}} Apr 17 15:57:41 user cinder-volume[84696]: DEBUG cinder.image.cache [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Found image-volume cache entry: {'id': 1, 'image_id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'volume_id': '006f20ae-5306-44fe-afdf-e433f79284e8', 'host': 'user@joviandss-2#joviandss-2', 'size': 1, 'image_updated_at': datetime.datetime(2023, 4, 17, 15, 19, 55), 'last_used': datetime.datetime(2023, 4, 17, 15, 57, 41, 971)}. {{(pid=84696) get_entry /opt/stack/cinder/cinder/image/cache.py:76}} Apr 17 15:57:41 user cinder-volume[84696]: DEBUG cinder.image.cache [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Image-volume cache entry image_update_at = 2023-04-17 15:19:55+00:00, requested image updated_at = 2023-04-17 15:19:55+00:00. {{(pid=84696) _should_update_entry /opt/stack/cinder/cinder/image/cache.py:259}} Apr 17 15:57:41 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Found cache entry for image = b3f0dec1-7758-44a8-82e0-76ada8837428 on host user@joviandss-2#joviandss-2. {{(pid=84696) _prepare_image_cache_entry /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:886}} Apr 17 15:57:41 user cinder-volume[84696]: DEBUG cinder.coordination [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Lock "cinder-b3f0dec1-7758-44a8-82e0-76ada8837428" released by "_prepare_image_cache_entry" :: held 0.018s {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:207}} Apr 17 15:57:41 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Attempting to retrieve cache entry for image = b3f0dec1-7758-44a8-82e0-76ada8837428 on host user@joviandss-2#joviandss-2. {{(pid=84696) _create_from_image_cache /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:827}} Apr 17 15:57:41 user cinder-volume[84696]: DEBUG cinder.image.cache [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Found image-volume cache entry: {'id': 1, 'image_id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'volume_id': '006f20ae-5306-44fe-afdf-e433f79284e8', 'host': 'user@joviandss-2#joviandss-2', 'size': 1, 'image_updated_at': datetime.datetime(2023, 4, 17, 15, 19, 55), 'last_used': datetime.datetime(2023, 4, 17, 15, 57, 41, 17896)}. {{(pid=84696) get_entry /opt/stack/cinder/cinder/image/cache.py:76}} Apr 17 15:57:41 user cinder-volume[84696]: DEBUG cinder.image.cache [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Image-volume cache entry image_update_at = 2023-04-17 15:19:55+00:00, requested image updated_at = 2023-04-17 15:19:55+00:00. {{(pid=84696) _should_update_entry /opt/stack/cinder/cinder/image/cache.py:259}} Apr 17 15:57:41 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Creating from source image-volume 006f20ae-5306-44fe-afdf-e433f79284e8 {{(pid=84696) _create_from_image_cache /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:841}} Apr 17 15:57:41 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] cloned volume 006f20ae-5306-44fe-afdf-e433f79284e8 to 14aa38ea-7cbb-48f7-a9aa-ffcc7b66faa0 {{(pid=84696) create_cloned_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:521}} Apr 17 15:57:41 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] cloning v_006f20ae-5306-44fe-afdf-e433f79284e8 to v_14aa38ea-7cbb-48f7-a9aa-ffcc7b66faa0 {{(pid=84696) _clone_object /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:456}} Apr 17 15:57:41 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] create snapshot v_14aa38ea-7cbb-48f7-a9aa-ffcc7b66faa0 {{(pid=84696) create_snapshot /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:635}} Apr 17 15:57:41 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:57:41 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:57:43 user cinder-volume[84696]: DEBUG cinder.utils [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 139.883(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:57:43 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:57:43 user cinder-volume[84696]: INFO cinder.message.api [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Creating message record for request_id = req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 Apr 17 15:57:43 user cinder-volume[84696]: DEBUG cinder.utils [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.681(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:57:43 user cinder-volume[84696]: DEBUG cinder.utils [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:57:43 user cinder-volume[84696]: DEBUG cinder.utils [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 3.004(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:57:43 user cinder-volume[84696]: DEBUG cinder.utils [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_14aa38ea-7cbb-48f7-a9aa-ffcc7b66faa0/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:57:43 user cinder-volume[84696]: DEBUG cinder.utils [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.881(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:57:43 user cinder-volume[84696]: DEBUG cinder.utils [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:57:43 user cinder-volume[84696]: DEBUG cinder.utils [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.063(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:57:43 user cinder-volume[84696]: DEBUG cinder.utils [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_87514055-5a0f-459f-b10b-438d7a13d39d/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:57:43 user cinder-volume[84696]: DEBUG cinder.utils [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 140.081(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:57:43 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_45e5cc09-6c97-438a-a6a2-c89e7d725ce5/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:57:43 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (9319b2f5-7d61-40d7-8b78-e1d9ddfb790f) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 15:57:43 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (057569d6-b4e2-468f-a12e-63b4e1de4f14) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:57:43 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (057569d6-b4e2-468f-a12e-63b4e1de4f14) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:57:43 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Flow 'volume_create_manager' (a7bdd606-b7ca-4091-9de2-cbe53cb9e5cf) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 15:57:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 15:57:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 15:57:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 15:57:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 15:57:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 15:57:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 15:57:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 15:57:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 15:57:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 15:57:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 15:57:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 15:57:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 15:57:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 15:57:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 15:57:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 15:57:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 15:57:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 15:57:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 15:57:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 15:57:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 15:57:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 15:57:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 15:57:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 15:57:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 15:57:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:57:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 15:57:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 15:57:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 15:57:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 15:57:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 15:57:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 15:57:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 15:57:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 15:57:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 15:57:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 15:57:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 15:57:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 15:57:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 15:57:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 15:57:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 15:57:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 15:57:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 15:57:43 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 15:57:43 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (d978d8da-9a7d-449f-b142-a305528cf7d3) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 15:57:43 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 15:57:43 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:53:03Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TestMinimumBasicScenario-volume-1479398066',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=45e5cc09-6c97-438a-a6a2-c89e7d725ce5,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='17d1dae0ad7149bbbd160525d64c0c20',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:55:24Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:55:24Z,use_quota=True,user_id='3c407d9d494d485c9c95dc3981284fba',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 15:57:43 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:53:03Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TestMinimumBasicScenario-volume-1479398066',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=45e5cc09-6c97-438a-a6a2-c89e7d725ce5,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='17d1dae0ad7149bbbd160525d64c0c20',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:55:24Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:55:24Z,use_quota=True,user_id='3c407d9d494d485c9c95dc3981284fba',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(45e5cc09-6c97-438a-a6a2-c89e7d725ce5),volume_id=45e5cc09-6c97-438a-a6a2-c89e7d725ce5,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': '45e5cc09-6c97-438a-a6a2-c89e7d725ce5', 'volume_name': 'volume-45e5cc09-6c97-438a-a6a2-c89e7d725ce5', 'volume_size': 1}} Apr 17 15:57:43 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': '45e5cc09-6c97-438a-a6a2-c89e7d725ce5', 'volume': {'id': '45e5cc09-6c97-438a-a6a2-c89e7d725ce5', '_name_id': None, 'ec2_id': None, 'user_id': '3c407d9d494d485c9c95dc3981284fba', 'project_id': '17d1dae0ad7149bbbd160525d64c0c20', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-TestMinimumBasicScenario-volume-1479398066', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'admin_metadata': {}, 'glance_metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T15:53:02.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-45e5cc09-6c97-438a-a6a2-c89e7d725ce5', 'name_id': '45e5cc09-6c97-438a-a6a2-c89e7d725ce5', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-TestMinimumBasicScenario-volume-1479398066', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '17d1dae0ad7149bbbd160525d64c0c20', 'qos_specs': None, 'reservations': ['a105b386-4e4e-4dca-ae87-4d1b3715ef82', '1ede682d-d17c-4d21-8d44-1ad1c081125a', 'df0aa6c2-bf1a-436e-9684-88c24c928b51', '6d77c826-2ba0-4ea1-b791-09b8cf39c978'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': '3c407d9d494d485c9c95dc3981284fba', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-TestMinimumBasicScenario-volume-1479398066', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '17d1dae0ad7149bbbd160525d64c0c20', 'qos_specs': None, 'reservations': ['a105b386-4e4e-4dca-ae87-4d1b3715ef82', '1ede682d-d17c-4d21-8d44-1ad1c081125a', 'df0aa6c2-bf1a-436e-9684-88c24c928b51', '6d77c826-2ba0-4ea1-b791-09b8cf39c978'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': '3c407d9d494d485c9c95dc3981284fba', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': '3c407d9d494d485c9c95dc3981284fba', 'metadata': {}, 'qos_specs': None, 'retry': {'num_attempts': 2, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']}}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(45e5cc09-6c97-438a-a6a2-c89e7d725ce5),volume_id=45e5cc09-6c97-438a-a6a2-c89e7d725ce5,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:53:03Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TestMinimumBasicScenario-volume-1479398066',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=45e5cc09-6c97-438a-a6a2-c89e7d725ce5,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='17d1dae0ad7149bbbd160525d64c0c20',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:55:24Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:55:24Z,use_quota=True,user_id='3c407d9d494d485c9c95dc3981284fba',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 15:57:43 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:53:03Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TestMinimumBasicScenario-volume-1479398066',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=45e5cc09-6c97-438a-a6a2-c89e7d725ce5,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='17d1dae0ad7149bbbd160525d64c0c20',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:55:24Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:55:24Z,use_quota=True,user_id='3c407d9d494d485c9c95dc3981284fba',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:53:03Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TestMinimumBasicScenario-volume-1479398066',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=45e5cc09-6c97-438a-a6a2-c89e7d725ce5,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='17d1dae0ad7149bbbd160525d64c0c20',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:55:24Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:55:24Z,use_quota=True,user_id='3c407d9d494d485c9c95dc3981284fba',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 15:57:43 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 15:57:43 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 15:57:43 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:57:43 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 15:57:43 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 15:57:43 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 15:57:43 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 15:57:43 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 15:57:43 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 15:57:43 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 15:57:43 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 15:57:43 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 15:57:43 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 15:57:43 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 15:57:43 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 15:57:43 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 15:57:43 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 15:57:43 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 15:57:43 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 15:57:43 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 15:57:43 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (d978d8da-9a7d-449f-b142-a305528cf7d3) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:57:43 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (d978d8da-9a7d-449f-b142-a305528cf7d3) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:57:43 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (e14e2361-1d63-4436-bc3b-dd448f9dfac5) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:57:43 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (e14e2361-1d63-4436-bc3b-dd448f9dfac5) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:57:43 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (27053387-0a12-4eea-b20b-8f6a8dc30bdc) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:57:43 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (27053387-0a12-4eea-b20b-8f6a8dc30bdc) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:57:43 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (cfe42e67-3aa3-4197-8aa9-61ca5abb7fb3) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:57:43 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Updating volume 45e5cc09-6c97-438a-a6a2-c89e7d725ce5 with {'scheduled_at': datetime.datetime(2023, 4, 17, 15, 57, 43, 799692), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 15:57:43 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Volume 45e5cc09-6c97-438a-a6a2-c89e7d725ce5: re-scheduling SchedulerAPI.create_volume attempt 2 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 15:57:43 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Volume 45e5cc09-6c97-438a-a6a2-c89e7d725ce5: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 15:57:43 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] deleating volume v_45e5cc09-6c97-438a-a6a2-c89e7d725ce5 {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 15:57:43 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] get snapshots for volume v_45e5cc09-6c97-438a-a6a2-c89e7d725ce5 {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 15:57:43 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_45e5cc09-6c97-438a-a6a2-c89e7d725ce5/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:57:43 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_45e5cc09-6c97-438a-a6a2-c89e7d725ce5/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:57:43 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Volume reschedule parameters: True retry: {'num_attempts': 3, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 15:57:43 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Flow 'volume_create_manager' (748603ab-e9fd-413b-9ecc-babf53ee3f3b) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:57:43 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (531f262f-7a09-49a6-b5f9-b8c622dd9382) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:57:43 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (531f262f-7a09-49a6-b5f9-b8c622dd9382) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:53:03Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TestMinimumBasicScenario-volume-1479398066',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=45e5cc09-6c97-438a-a6a2-c89e7d725ce5,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='17d1dae0ad7149bbbd160525d64c0c20',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:57:44Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:57:44Z,use_quota=True,user_id='3c407d9d494d485c9c95dc3981284fba',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:57:43 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (458dc8a3-8d60-43b2-a4a5-5d529293df10) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:57:43 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (458dc8a3-8d60-43b2-a4a5-5d529293df10) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:57:43 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (442b4e40-5a82-49b5-84d7-3ca093921162) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:57:43 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (442b4e40-5a82-49b5-84d7-3ca093921162) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': '45e5cc09-6c97-438a-a6a2-c89e7d725ce5', 'volume_name': 'volume-45e5cc09-6c97-438a-a6a2-c89e7d725ce5', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:57:43 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (0a8f26a6-0581-42bd-8d58-5e224ca90391) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:57:43 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (0a8f26a6-0581-42bd-8d58-5e224ca90391) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:57:44 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (691f22bc-e2c1-43bc-b87e-457f041d6b10) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:57:44 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Volume 45e5cc09-6c97-438a-a6a2-c89e7d725ce5: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-45e5cc09-6c97-438a-a6a2-c89e7d725ce5', 'volume_size': 1} Apr 17 15:57:44 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] creating volume v_45e5cc09-6c97-438a-a6a2-c89e7d725ce5. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 15:57:44 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] create volume {'name': 'v_45e5cc09-6c97-438a-a6a2-c89e7d725ce5', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 15:57:44 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:57:44 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:57:46 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 75.033(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:57:46 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:57:46 user cinder-volume[84696]: DEBUG cinder.utils [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.955(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:57:46 user cinder-volume[84696]: DEBUG cinder.utils [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_45e5cc09-6c97-438a-a6a2-c89e7d725ce5/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:57:46 user cinder-volume[84696]: DEBUG cinder.utils [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.782(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:57:46 user cinder-volume[84696]: DEBUG cinder.utils [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:57:46 user cinder-volume[84696]: DEBUG cinder.utils [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.072(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:57:46 user cinder-volume[84696]: DEBUG cinder.utils [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_14aa38ea-7cbb-48f7-a9aa-ffcc7b66faa0/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:57:46 user cinder-volume[84696]: DEBUG cinder.utils [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.753(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:57:46 user cinder-volume[84696]: DEBUG cinder.utils [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:57:47 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=23,cinder:UPDATE=10 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:57:47 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Volume reschedule parameters: True retry: {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 15:57:47 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Flow 'volume_create_manager' (b5b426a6-1f65-4716-a4a7-8298feedc9da) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:57:47 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (3c3f1053-b1fc-43ad-88d3-ca0a3dff442a) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:57:47 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (3c3f1053-b1fc-43ad-88d3-ca0a3dff442a) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:57:47Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumeRevertTests-Volume-1536424582',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=9a399a62-6a8f-4b6d-8ecf-8dcdf57e5fbb,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='9c755d3b5d8a4d9c9918b6c4744f510e',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:57:47Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:57:47Z,use_quota=True,user_id='e362cfe4a31c444b9ab44a5d60f37ad9',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:57:47 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (ddaa5868-0c7e-4fad-ac69-846477b8c783) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:57:47 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (ddaa5868-0c7e-4fad-ac69-846477b8c783) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:57:47 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (d5e95f2f-b2ca-4d16-89a5-b33816fc5ac3) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:57:47 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (d5e95f2f-b2ca-4d16-89a5-b33816fc5ac3) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': '9a399a62-6a8f-4b6d-8ecf-8dcdf57e5fbb', 'volume_name': 'volume-9a399a62-6a8f-4b6d-8ecf-8dcdf57e5fbb', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:57:47 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (4459bc91-9990-4b6d-bd17-93ea53768161) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:57:47 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (4459bc91-9990-4b6d-bd17-93ea53768161) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:57:47 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (fa6c819a-c3c4-4f92-97d3-d0278e26aca5) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:57:47 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Volume 9a399a62-6a8f-4b6d-8ecf-8dcdf57e5fbb: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-9a399a62-6a8f-4b6d-8ecf-8dcdf57e5fbb', 'volume_size': 1} Apr 17 15:57:47 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] creating volume v_9a399a62-6a8f-4b6d-8ecf-8dcdf57e5fbb. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 15:57:47 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] create volume {'name': 'v_9a399a62-6a8f-4b6d-8ecf-8dcdf57e5fbb', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 15:57:47 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:57:47 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:57:49 user cinder-volume[84696]: DEBUG cinder.utils [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.568(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:57:49 user cinder-volume[84696]: DEBUG cinder.utils [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:57:49 user cinder-volume[84696]: DEBUG cinder.utils [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.195(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:57:49 user cinder-volume[84696]: DEBUG cinder.utils [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_87514055-5a0f-459f-b10b-438d7a13d39d/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:57:49 user cinder-volume[84696]: DEBUG cinder.utils [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.025(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:57:49 user cinder-volume[84696]: DEBUG cinder.utils [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_45e5cc09-6c97-438a-a6a2-c89e7d725ce5/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:57:49 user cinder-volume[84696]: DEBUG cinder.utils [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.017(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:57:49 user cinder-volume[84696]: DEBUG cinder.utils [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:57:49 user cinder-volume[84696]: DEBUG cinder.utils [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.851(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:57:49 user cinder-volume[84696]: DEBUG cinder.utils [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:57:52 user cinder-volume[84696]: DEBUG cinder.utils [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.209(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:57:52 user cinder-volume[84696]: DEBUG cinder.utils [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_14aa38ea-7cbb-48f7-a9aa-ffcc7b66faa0/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:57:52 user cinder-volume[84696]: DEBUG cinder.utils [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 11.891(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:57:52 user cinder-volume[84696]: DEBUG cinder.utils [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:57:52 user cinder-volume[84696]: DEBUG cinder.utils [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.644(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:57:52 user cinder-volume[84696]: DEBUG cinder.utils [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:57:55 user cinder-volume[84696]: DEBUG cinder.utils [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 11.991(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:57:55 user cinder-volume[84696]: DEBUG cinder.utils [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:57:55 user cinder-volume[84696]: DEBUG cinder.utils [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.168(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:57:55 user cinder-volume[84696]: DEBUG cinder.utils [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_45e5cc09-6c97-438a-a6a2-c89e7d725ce5/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:57:59 user cinder-volume[84696]: DEBUG cinder.utils [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.230(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:57:59 user cinder-volume[84696]: DEBUG cinder.utils [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:57:59 user cinder-volume[84696]: DEBUG cinder.utils [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 11.786(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:57:59 user cinder-volume[84696]: DEBUG cinder.utils [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:57:59 user cinder-volume[84696]: DEBUG cinder.utils [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.412(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:57:59 user cinder-volume[84696]: DEBUG cinder.utils [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_87514055-5a0f-459f-b10b-438d7a13d39d/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:58:02 user cinder-volume[84696]: DEBUG cinder.utils [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 74.373(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:58:02 user cinder-volume[84696]: DEBUG cinder.utils [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_b29a7592-128a-41ee-a8e2-57780dee33e7/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:58:02 user cinder-volume[84696]: DEBUG cinder.utils [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 74.197(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:58:02 user cinder-volume[84696]: DEBUG cinder.utils [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:58:02 user cinder-volume[84696]: DEBUG cinder.utils [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.461(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:58:02 user cinder-volume[84696]: DEBUG cinder.utils [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_14aa38ea-7cbb-48f7-a9aa-ffcc7b66faa0/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:58:02 user cinder-volume[84696]: DEBUG cinder.utils [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.142(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:58:02 user cinder-volume[84696]: DEBUG cinder.utils [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:58:05 user cinder-volume[84696]: DEBUG cinder.utils [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.239(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:58:05 user cinder-volume[84696]: DEBUG cinder.utils [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:58:05 user cinder-volume[84696]: DEBUG cinder.utils [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.417(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:58:05 user cinder-volume[84696]: DEBUG cinder.utils [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_45e5cc09-6c97-438a-a6a2-c89e7d725ce5/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:58:08 user cinder-volume[84696]: DEBUG cinder.utils [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 73.930(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:58:08 user cinder-volume[84696]: DEBUG cinder.utils [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:58:08 user cinder-volume[84696]: DEBUG cinder.utils [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 72.913(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:58:08 user cinder-volume[84696]: DEBUG cinder.utils [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:58:08 user cinder-volume[84696]: DEBUG cinder.utils [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 74.359(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:58:08 user cinder-volume[84696]: DEBUG cinder.utils [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_75163b78-31b2-4676-b93b-542a8ea784c1/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:58:08 user cinder-volume[84696]: DEBUG cinder.utils [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.039(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:58:08 user cinder-volume[84696]: DEBUG cinder.utils [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:58:08 user cinder-volume[84696]: DEBUG cinder.utils [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 74.353(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:58:08 user cinder-volume[84696]: DEBUG cinder.utils [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_60979eba-96b1-4cb3-bae1-8874a86cb411/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:58:14 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=24,cinder:INSERT=5,cinder:UPDATE=4 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:58:14 user cinder-volume[84696]: DEBUG cinder.utils [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 74.367(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:58:14 user cinder-volume[84696]: DEBUG cinder.utils [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_2ed5b8aa-fb48-4255-b6d5-f0f84f733834/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:58:17 user cinder-volume[84696]: DEBUG cinder.utils [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.873(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:58:17 user cinder-volume[84696]: DEBUG cinder.utils [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_87514055-5a0f-459f-b10b-438d7a13d39d/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:58:17 user cinder-volume[84696]: DEBUG cinder.utils [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.695(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:58:17 user cinder-volume[84696]: DEBUG cinder.utils [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:58:20 user cinder-volume[84696]: DEBUG cinder.utils [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.568(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:58:20 user cinder-volume[84696]: DEBUG cinder.utils [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:58:20 user cinder-volume[84696]: DEBUG cinder.utils [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.890(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:58:20 user cinder-volume[84696]: DEBUG cinder.utils [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_14aa38ea-7cbb-48f7-a9aa-ffcc7b66faa0/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:58:23 user cinder-volume[84696]: DEBUG cinder.utils [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.671(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:58:23 user cinder-volume[84696]: DEBUG cinder.utils [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:58:23 user cinder-volume[84696]: DEBUG cinder.utils [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.848(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:58:23 user cinder-volume[84696]: DEBUG cinder.utils [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_45e5cc09-6c97-438a-a6a2-c89e7d725ce5/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:58:24 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=7,cinder:UPDATE=3 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:58:26 user cinder-volume[84696]: DEBUG cinder.utils [req-778c62be-8963-4e6d-97f1-f456b759ce16 req-6d72b587-594c-4309-b424-7292ebb80925 tempest-AttachVolumeShelveTestJSON-1745617049 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 137.545(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:58:26 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-778c62be-8963-4e6d-97f1-f456b759ce16 req-6d72b587-594c-4309-b424-7292ebb80925 tempest-AttachVolumeShelveTestJSON-1745617049 None] Geting None from DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:13bf568b-585c-4556-ab78-e9930ad50214/luns/v_13bf568b-585c-4556-ab78-e9930ad50214 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:58:26 user cinder-volume[84696]: WARNING cinder.volume.manager [req-778c62be-8963-4e6d-97f1-f456b759ce16 req-6d72b587-594c-4309-b424-7292ebb80925 tempest-AttachVolumeShelveTestJSON-1745617049 None] Failed to detach volume on the backend, ignoring failure 'NoneType' object is not subscriptable Apr 17 15:58:26 user cinder-volume[84696]: DEBUG cinder.utils [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.468(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:58:26 user cinder-volume[84696]: DEBUG cinder.utils [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:58:32 user cinder-volume[84696]: DEBUG cinder.utils [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 139.763(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:58:32 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_4a2d1c00-a134-4f73-8508-86f35b58acf7/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:58:32 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (73e259f7-6c72-415f-a171-54482295bca4) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 15:58:32 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (48e030fb-2589-48b7-90a2-28b8d68ca273) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:58:32 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (48e030fb-2589-48b7-90a2-28b8d68ca273) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:58:32 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Flow 'volume_create_manager' (23959a00-2be6-447c-aa40-478af7a7f898) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 15:58:32 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-c68061bd-1bdf-4eaf-8a2b-14e439bed46a tempest-AttachVolumeNegativeTest-331389537 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 15:58:32 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 15:58:32 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 15:58:32 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 15:58:32 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 15:58:32 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 15:58:32 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 15:58:32 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 15:58:32 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 15:58:32 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 15:58:32 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 15:58:32 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 15:58:32 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 15:58:32 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 15:58:32 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 15:58:32 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 15:58:32 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 15:58:32 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 15:58:32 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 15:58:32 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 15:58:32 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 15:58:32 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 15:58:32 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 15:58:32 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 15:58:32 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:58:32 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 15:58:32 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 15:58:32 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 15:58:32 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 15:58:32 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 15:58:32 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 15:58:32 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 15:58:32 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 15:58:32 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 15:58:32 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 15:58:32 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 15:58:32 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 15:58:32 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 15:58:32 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 15:58:32 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 15:58:32 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 15:58:32 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 15:58:44 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=3 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:58:44 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=2,cinder:UPDATE=2 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:58:46 user cinder-volume[84696]: DEBUG cinder.utils [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 140.810(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:58:46 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:58:46 user cinder-volume[84696]: INFO cinder.message.api [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Creating message record for request_id = req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 Apr 17 15:58:46 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 15:58:46 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (e44ce900-5590-4764-9dce-e1f550b0dc9c) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 15:58:46 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 15:58:46 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:56:26Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumeRetypeWithoutMigrationTest-Volume-636058295',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=bff6fdbb-a1fb-4b48-bca8-2f7efbc3f0ff,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='46c9cac22c32484181c934b27e67f512',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:56:26Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:56:26Z,use_quota=True,user_id='a0751e2174804d7ab462edae5f88d581',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(c9c52f37-7390-466f-a6ab-56ae9a61e5ed),volume_type_id=c9c52f37-7390-466f-a6ab-56ae9a61e5ed)}, 'provides': None} Apr 17 15:58:46 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:56:26Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumeRetypeWithoutMigrationTest-Volume-636058295',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=bff6fdbb-a1fb-4b48-bca8-2f7efbc3f0ff,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='46c9cac22c32484181c934b27e67f512',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:56:26Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:56:26Z,use_quota=True,user_id='a0751e2174804d7ab462edae5f88d581',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(c9c52f37-7390-466f-a6ab-56ae9a61e5ed),volume_type_id=c9c52f37-7390-466f-a6ab-56ae9a61e5ed), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(bff6fdbb-a1fb-4b48-bca8-2f7efbc3f0ff),volume_id=bff6fdbb-a1fb-4b48-bca8-2f7efbc3f0ff,volume_properties=VolumeProperties,volume_type=VolumeType(c9c52f37-7390-466f-a6ab-56ae9a61e5ed))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': 'bff6fdbb-a1fb-4b48-bca8-2f7efbc3f0ff', 'volume_name': 'volume-bff6fdbb-a1fb-4b48-bca8-2f7efbc3f0ff', 'volume_size': 1}} Apr 17 15:58:46 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'retry': {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']}, 'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': 'bff6fdbb-a1fb-4b48-bca8-2f7efbc3f0ff', 'volume': {'id': 'bff6fdbb-a1fb-4b48-bca8-2f7efbc3f0ff', '_name_id': None, 'ec2_id': None, 'user_id': 'a0751e2174804d7ab462edae5f88d581', 'project_id': '46c9cac22c32484181c934b27e67f512', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-VolumeRetypeWithoutMigrationTest-Volume-636058295', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': 'c9c52f37-7390-466f-a6ab-56ae9a61e5ed', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'volume_type': {'id': 'c9c52f37-7390-466f-a6ab-56ae9a61e5ed', 'name': 'volume-type-1', 'description': None, 'is_public': True, 'extra_specs': {}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:56:25.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T15:56:25.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-bff6fdbb-a1fb-4b48-bca8-2f7efbc3f0ff', 'name_id': 'bff6fdbb-a1fb-4b48-bca8-2f7efbc3f0ff', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': 'c9c52f37-7390-466f-a6ab-56ae9a61e5ed', 'name': 'volume-type-1', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:56:25.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumeRetypeWithoutMigrationTest-Volume-636058295', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '46c9cac22c32484181c934b27e67f512', 'qos_specs': None, 'reservations': ['622ac28a-652f-4d3c-b14a-4e0cf9309855', '3f27124e-4ba9-448f-a619-0c453485c180', '93656d14-fa21-40b5-acf9-87e241ec0ef5', 'a859cf24-fdaa-4c02-a26d-31ecb6289b76'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'a0751e2174804d7ab462edae5f88d581', 'volume_type_id': 'c9c52f37-7390-466f-a6ab-56ae9a61e5ed'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumeRetypeWithoutMigrationTest-Volume-636058295', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '46c9cac22c32484181c934b27e67f512', 'qos_specs': None, 'reservations': ['622ac28a-652f-4d3c-b14a-4e0cf9309855', '3f27124e-4ba9-448f-a619-0c453485c180', '93656d14-fa21-40b5-acf9-87e241ec0ef5', 'a859cf24-fdaa-4c02-a26d-31ecb6289b76'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'a0751e2174804d7ab462edae5f88d581', 'volume_type_id': 'c9c52f37-7390-466f-a6ab-56ae9a61e5ed'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:56:25Z,deleted=False,deleted_at=None,description=None,extra_specs={},id=c9c52f37-7390-466f-a6ab-56ae9a61e5ed,is_public=True,name='volume-type-1',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:56:25Z,deleted=False,deleted_at=None,description=None,extra_specs={},id=c9c52f37-7390-466f-a6ab-56ae9a61e5ed,is_public=True,name='volume-type-1',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': 'a0751e2174804d7ab462edae5f88d581', 'metadata': {}, 'qos_specs': None}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(bff6fdbb-a1fb-4b48-bca8-2f7efbc3f0ff),volume_id=bff6fdbb-a1fb-4b48-bca8-2f7efbc3f0ff,volume_properties=VolumeProperties,volume_type=VolumeType(c9c52f37-7390-466f-a6ab-56ae9a61e5ed)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:56:26Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumeRetypeWithoutMigrationTest-Volume-636058295',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=bff6fdbb-a1fb-4b48-bca8-2f7efbc3f0ff,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='46c9cac22c32484181c934b27e67f512',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:56:26Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:56:26Z,use_quota=True,user_id='a0751e2174804d7ab462edae5f88d581',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(c9c52f37-7390-466f-a6ab-56ae9a61e5ed),volume_type_id=c9c52f37-7390-466f-a6ab-56ae9a61e5ed), 'context': }, 'provides': None} Apr 17 15:58:46 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:56:26Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumeRetypeWithoutMigrationTest-Volume-636058295',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=bff6fdbb-a1fb-4b48-bca8-2f7efbc3f0ff,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='46c9cac22c32484181c934b27e67f512',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:56:26Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:56:26Z,use_quota=True,user_id='a0751e2174804d7ab462edae5f88d581',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(c9c52f37-7390-466f-a6ab-56ae9a61e5ed),volume_type_id=c9c52f37-7390-466f-a6ab-56ae9a61e5ed)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:56:26Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumeRetypeWithoutMigrationTest-Volume-636058295',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=bff6fdbb-a1fb-4b48-bca8-2f7efbc3f0ff,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='46c9cac22c32484181c934b27e67f512',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:56:26Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:56:26Z,use_quota=True,user_id='a0751e2174804d7ab462edae5f88d581',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(c9c52f37-7390-466f-a6ab-56ae9a61e5ed),volume_type_id=c9c52f37-7390-466f-a6ab-56ae9a61e5ed)} Apr 17 15:58:46 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 15:58:46 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 15:58:46 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:58:46 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 15:58:46 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 15:58:46 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 15:58:46 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 15:58:46 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 15:58:46 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 15:58:46 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 15:58:46 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 15:58:46 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 15:58:46 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 15:58:46 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 15:58:46 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 15:58:46 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 15:58:46 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 15:58:46 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 15:58:46 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 15:58:46 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 15:58:46 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (e44ce900-5590-4764-9dce-e1f550b0dc9c) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:58:46 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (e44ce900-5590-4764-9dce-e1f550b0dc9c) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:58:46 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (db64d0b8-e02e-424b-bf45-8d434339190c) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:58:46 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (db64d0b8-e02e-424b-bf45-8d434339190c) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:58:46 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (03c9e863-3781-4ccb-877d-e1024bc6d33d) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:58:46 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (03c9e863-3781-4ccb-877d-e1024bc6d33d) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:58:46 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (c8a9a018-c594-4f2a-b9b3-d1b09245a3fb) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:58:46 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Updating volume bff6fdbb-a1fb-4b48-bca8-2f7efbc3f0ff with {'scheduled_at': datetime.datetime(2023, 4, 17, 15, 58, 46, 853307), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 15:58:46 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Volume bff6fdbb-a1fb-4b48-bca8-2f7efbc3f0ff: re-scheduling SchedulerAPI.create_volume attempt 1 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 15:58:46 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Volume bff6fdbb-a1fb-4b48-bca8-2f7efbc3f0ff: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 15:58:46 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] deleating volume v_bff6fdbb-a1fb-4b48-bca8-2f7efbc3f0ff {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 15:58:46 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] get snapshots for volume v_bff6fdbb-a1fb-4b48-bca8-2f7efbc3f0ff {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 15:58:46 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_bff6fdbb-a1fb-4b48-bca8-2f7efbc3f0ff/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:58:46 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_bff6fdbb-a1fb-4b48-bca8-2f7efbc3f0ff/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:58:46 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Volume reschedule parameters: True retry: {'num_attempts': 2, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 15:58:47 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Flow 'volume_create_manager' (9b1aa7ce-a821-42f3-a57a-8536cbc161b7) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:58:47 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (b2c35c35-5246-49c2-98d9-d0327dbbde2b) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:58:47 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (b2c35c35-5246-49c2-98d9-d0327dbbde2b) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:56:26Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumeRetypeWithoutMigrationTest-Volume-636058295',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=bff6fdbb-a1fb-4b48-bca8-2f7efbc3f0ff,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='46c9cac22c32484181c934b27e67f512',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:58:47Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:58:47Z,use_quota=True,user_id='a0751e2174804d7ab462edae5f88d581',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(c9c52f37-7390-466f-a6ab-56ae9a61e5ed),volume_type_id=c9c52f37-7390-466f-a6ab-56ae9a61e5ed)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:58:47 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (51865b1d-35c0-44f9-8c4d-2f75409a79d7) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:58:47 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (51865b1d-35c0-44f9-8c4d-2f75409a79d7) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:58:47 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (80f20411-2ada-47f3-af2d-dbec4f4428fa) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:58:47 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (80f20411-2ada-47f3-af2d-dbec4f4428fa) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': 'bff6fdbb-a1fb-4b48-bca8-2f7efbc3f0ff', 'volume_name': 'volume-bff6fdbb-a1fb-4b48-bca8-2f7efbc3f0ff', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:58:47 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (7bd1a7db-0187-4994-8ace-37be25ed3b59) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:58:47 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (7bd1a7db-0187-4994-8ace-37be25ed3b59) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:58:47 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (beed5bb1-af79-4e0a-88f6-18309da5d010) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:58:47 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Volume bff6fdbb-a1fb-4b48-bca8-2f7efbc3f0ff: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-bff6fdbb-a1fb-4b48-bca8-2f7efbc3f0ff', 'volume_size': 1} Apr 17 15:58:47 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] creating volume v_bff6fdbb-a1fb-4b48-bca8-2f7efbc3f0ff. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 15:58:47 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] create volume {'name': 'v_bff6fdbb-a1fb-4b48-bca8-2f7efbc3f0ff', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 15:58:47 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:58:47 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:58:49 user cinder-volume[84696]: DEBUG cinder.utils [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.795(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:58:49 user cinder-volume[84696]: DEBUG cinder.utils [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:58:49 user cinder-volume[84696]: DEBUG cinder.utils [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.969(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:58:49 user cinder-volume[84696]: DEBUG cinder.utils [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_bff6fdbb-a1fb-4b48-bca8-2f7efbc3f0ff/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:58:49 user cinder-volume[84696]: DEBUG cinder.utils [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 72.196(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:58:49 user cinder-volume[84696]: DEBUG cinder.utils [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_87514055-5a0f-459f-b10b-438d7a13d39d/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:58:49 user cinder-volume[84696]: DEBUG cinder.utils [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 72.017(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:58:49 user cinder-volume[84696]: DEBUG cinder.utils [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:58:52 user cinder-volume[84696]: DEBUG cinder.utils [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.867(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:58:52 user cinder-volume[84696]: DEBUG cinder.utils [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:58:52 user cinder-volume[84696]: DEBUG cinder.utils [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 72.210(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:58:52 user cinder-volume[84696]: DEBUG cinder.utils [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_14aa38ea-7cbb-48f7-a9aa-ffcc7b66faa0/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:58:52 user cinder-volume[84696]: DEBUG cinder.utils [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.042(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:58:52 user cinder-volume[84696]: DEBUG cinder.utils [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_bff6fdbb-a1fb-4b48-bca8-2f7efbc3f0ff/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:58:52 user cinder-volume[84696]: DEBUG cinder.utils [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 71.891(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:58:52 user cinder-volume[84696]: DEBUG cinder.utils [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:58:52 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 141.177(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:58:52 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:58:52 user cinder-volume[84696]: ERROR oslo_service.periodic_task [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Error during VolumeManager.publish_service_capabilities: TypeError: 'NoneType' object is not subscriptable Apr 17 15:58:52 user cinder-volume[84696]: ERROR oslo_service.periodic_task Traceback (most recent call last): Apr 17 15:58:52 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/usr/local/lib/python3.10/dist-packages/oslo_service/periodic_task.py", line 216, in run_periodic_tasks Apr 17 15:58:52 user cinder-volume[84696]: ERROR oslo_service.periodic_task task(self, context) Apr 17 15:58:52 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/manager.py", line 2876, in publish_service_capabilities Apr 17 15:58:52 user cinder-volume[84696]: ERROR oslo_service.periodic_task self._report_driver_status(context) Apr 17 15:58:52 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/manager.py", line 2755, in _report_driver_status Apr 17 15:58:52 user cinder-volume[84696]: ERROR oslo_service.periodic_task volume_stats = get_stats() Apr 17 15:58:52 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/usr/local/lib/python3.10/dist-packages/oslo_utils/timeutils.py", line 319, in wrapper Apr 17 15:58:52 user cinder-volume[84696]: ERROR oslo_service.periodic_task result = func(*args, **kwargs) Apr 17 15:58:52 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/manager.py", line 2753, in get_stats Apr 17 15:58:52 user cinder-volume[84696]: ERROR oslo_service.periodic_task return self.driver.get_volume_stats(refresh=True) Apr 17 15:58:52 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/driver.py", line 666, in get_volume_stats Apr 17 15:58:52 user cinder-volume[84696]: ERROR oslo_service.periodic_task self._update_volume_stats() Apr 17 15:58:52 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 726, in _update_volume_stats Apr 17 15:58:52 user cinder-volume[84696]: ERROR oslo_service.periodic_task pool_stats = self.ra.get_pool_stats() Apr 17 15:58:52 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 897, in get_pool_stats Apr 17 15:58:52 user cinder-volume[84696]: ERROR oslo_service.periodic_task if not resp["error"] and resp["code"] == 200: Apr 17 15:58:52 user cinder-volume[84696]: ERROR oslo_service.periodic_task TypeError: 'NoneType' object is not subscriptable Apr 17 15:58:52 user cinder-volume[84696]: ERROR oslo_service.periodic_task Apr 17 15:58:52 user cinder-volume[84696]: WARNING oslo.service.loopingcall [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Function 'cinder.service.Service.periodic_tasks' run outlasted interval by 81.18 sec Apr 17 15:58:52 user cinder-volume[84696]: DEBUG oslo_service.periodic_task [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Running periodic task VolumeManager.publish_service_capabilities {{(pid=84696) run_periodic_tasks /usr/local/lib/python3.10/dist-packages/oslo_service/periodic_task.py:210}} Apr 17 15:58:52 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Updating volume stats {{(pid=84696) _update_volume_stats /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:724}} Apr 17 15:58:52 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Get pool Pool-0 fsprops {{(pid=84696) get_pool_stats /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:894}} Apr 17 15:58:52 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:58:52 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:58:53 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Volume reschedule parameters: True retry: {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 15:58:53 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Flow 'volume_create_manager' (0c6cf15d-a93f-4047-8d8d-426ad50dd3b5) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:58:53 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (f420ec26-2fc9-4a88-a8f8-96c2cebdacbb) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:58:53 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (f420ec26-2fc9-4a88-a8f8-96c2cebdacbb) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:58:54Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TestVolumeBootPattern-volume-origin-619636601',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=fd313648-5fc1-4e6a-97a0-92ac803b36fd,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='c8a836b47d954cb9a7c1cd7c0f790426',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:58:54Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:58:54Z,use_quota=True,user_id='a80e753322224ec1a897598b5b9ed6ea',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:58:53 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (435cbcff-be2f-4bb7-bcc2-abf1daaaf114) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:58:53 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (435cbcff-be2f-4bb7-bcc2-abf1daaaf114) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:58:53 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (b2b7a1f1-1964-4350-bbc5-ad6aef647d46) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:58:53 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (b2b7a1f1-1964-4350-bbc5-ad6aef647d46) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'image', 'volume_id': 'fd313648-5fc1-4e6a-97a0-92ac803b36fd', 'volume_name': 'volume-fd313648-5fc1-4e6a-97a0-92ac803b36fd', 'volume_size': 1, 'image_id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'image_location': (None, None), 'image_meta': {'name': 'cirros-0.5.2-x86_64-disk', 'disk_format': 'qcow2', 'container_format': 'bare', 'visibility': 'public', 'size': 16300544, 'virtual_size': 117440512, 'status': 'active', 'checksum': 'b874c39491a2377b8490f5f1e89761a4', 'protected': False, 'min_ram': 0, 'min_disk': 0, 'owner': 'a43c211b4ed542538799349489b1d884', 'os_hidden': False, 'os_hash_algo': 'sha512', 'os_hash_value': '6b813aa46bb90b4da216a4d19376593fa3f4fc7e617f03a92b7fe11e9a3981cbe8f0959dbebe36225e5f53dc4492341a4863cac4ed1ee0909f3fc78ef9c3e869', 'id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'created_at': datetime.datetime(2023, 4, 17, 15, 19, 53, tzinfo=datetime.timezone.utc), 'updated_at': datetime.datetime(2023, 4, 17, 15, 19, 55, tzinfo=datetime.timezone.utc), 'tags': [], 'file': '/v2/images/b3f0dec1-7758-44a8-82e0-76ada8837428/file', 'properties': {'hw_rng_model': 'virtio', 'owner_specified.openstack.md5': '', 'owner_specified.openstack.object': 'images/cirros-0.5.2-x86_64-disk', 'owner_specified.openstack.sha256': ''}}, 'image_service': }' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:58:53 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (5fe711b6-06d8-48e1-8ec3-a2ad9ff5330f) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:58:53 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (5fe711b6-06d8-48e1-8ec3-a2ad9ff5330f) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:58:53 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (d2b724be-e363-4692-aa27-06c48d6720f2) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:58:53 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Volume fd313648-5fc1-4e6a-97a0-92ac803b36fd: being created as image with specification: {'status': 'creating', 'volume_name': 'volume-fd313648-5fc1-4e6a-97a0-92ac803b36fd', 'volume_size': 1, 'image_id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'image_location': (None, None), 'image_meta': {'name': 'cirros-0.5.2-x86_64-disk', 'disk_format': 'qcow2', 'container_format': 'bare', 'visibility': 'public', 'size': 16300544, 'virtual_size': 117440512, 'status': 'active', 'checksum': 'b874c39491a2377b8490f5f1e89761a4', 'protected': False, 'min_ram': 0, 'min_disk': 0, 'owner': 'a43c211b4ed542538799349489b1d884', 'os_hidden': False, 'os_hash_algo': 'sha512', 'os_hash_value': '6b813aa46bb90b4da216a4d19376593fa3f4fc7e617f03a92b7fe11e9a3981cbe8f0959dbebe36225e5f53dc4492341a4863cac4ed1ee0909f3fc78ef9c3e869', 'id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'created_at': datetime.datetime(2023, 4, 17, 15, 19, 53, tzinfo=datetime.timezone.utc), 'updated_at': datetime.datetime(2023, 4, 17, 15, 19, 55, tzinfo=datetime.timezone.utc), 'tags': [], 'file': '/v2/images/b3f0dec1-7758-44a8-82e0-76ada8837428/file', 'properties': {'hw_rng_model': 'virtio', 'owner_specified.openstack.md5': '', 'owner_specified.openstack.object': 'images/cirros-0.5.2-x86_64-disk', 'owner_specified.openstack.sha256': ''}}, 'image_service': } Apr 17 15:58:53 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Cloning fd313648-5fc1-4e6a-97a0-92ac803b36fd from image b3f0dec1-7758-44a8-82e0-76ada8837428 at location (None, None). {{(pid=84696) _create_from_image /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1071}} Apr 17 15:58:53 user cinder-volume[84696]: DEBUG cinder.coordination [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Acquiring lock "cinder-b3f0dec1-7758-44a8-82e0-76ada8837428" by "_prepare_image_cache_entry" {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:192}} Apr 17 15:58:53 user cinder-volume[84696]: DEBUG cinder.coordination [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Lock "cinder-b3f0dec1-7758-44a8-82e0-76ada8837428" acquired by "_prepare_image_cache_entry" :: waited 0.008s {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:196}} Apr 17 15:58:53 user cinder-volume[84696]: DEBUG cinder.image.cache [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Found image-volume cache entry: {'id': 1, 'image_id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'volume_id': '006f20ae-5306-44fe-afdf-e433f79284e8', 'host': 'user@joviandss-2#joviandss-2', 'size': 1, 'image_updated_at': datetime.datetime(2023, 4, 17, 15, 19, 55), 'last_used': datetime.datetime(2023, 4, 17, 15, 58, 53, 987404)}. {{(pid=84696) get_entry /opt/stack/cinder/cinder/image/cache.py:76}} Apr 17 15:58:53 user cinder-volume[84696]: DEBUG cinder.image.cache [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Image-volume cache entry image_update_at = 2023-04-17 15:19:55+00:00, requested image updated_at = 2023-04-17 15:19:55+00:00. {{(pid=84696) _should_update_entry /opt/stack/cinder/cinder/image/cache.py:259}} Apr 17 15:58:53 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Found cache entry for image = b3f0dec1-7758-44a8-82e0-76ada8837428 on host user@joviandss-2#joviandss-2. {{(pid=84696) _prepare_image_cache_entry /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:886}} Apr 17 15:58:53 user cinder-volume[84696]: DEBUG cinder.coordination [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Lock "cinder-b3f0dec1-7758-44a8-82e0-76ada8837428" released by "_prepare_image_cache_entry" :: held 0.014s {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:207}} Apr 17 15:58:53 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Attempting to retrieve cache entry for image = b3f0dec1-7758-44a8-82e0-76ada8837428 on host user@joviandss-2#joviandss-2. {{(pid=84696) _create_from_image_cache /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:827}} Apr 17 15:58:54 user cinder-volume[84696]: DEBUG cinder.image.cache [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Found image-volume cache entry: {'id': 1, 'image_id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'volume_id': '006f20ae-5306-44fe-afdf-e433f79284e8', 'host': 'user@joviandss-2#joviandss-2', 'size': 1, 'image_updated_at': datetime.datetime(2023, 4, 17, 15, 19, 55), 'last_used': datetime.datetime(2023, 4, 17, 15, 58, 54, 912)}. {{(pid=84696) get_entry /opt/stack/cinder/cinder/image/cache.py:76}} Apr 17 15:58:54 user cinder-volume[84696]: DEBUG cinder.image.cache [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Image-volume cache entry image_update_at = 2023-04-17 15:19:55+00:00, requested image updated_at = 2023-04-17 15:19:55+00:00. {{(pid=84696) _should_update_entry /opt/stack/cinder/cinder/image/cache.py:259}} Apr 17 15:58:54 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Creating from source image-volume 006f20ae-5306-44fe-afdf-e433f79284e8 {{(pid=84696) _create_from_image_cache /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:841}} Apr 17 15:58:54 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] cloned volume 006f20ae-5306-44fe-afdf-e433f79284e8 to fd313648-5fc1-4e6a-97a0-92ac803b36fd {{(pid=84696) create_cloned_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:521}} Apr 17 15:58:54 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] cloning v_006f20ae-5306-44fe-afdf-e433f79284e8 to v_fd313648-5fc1-4e6a-97a0-92ac803b36fd {{(pid=84696) _clone_object /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:456}} Apr 17 15:58:54 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] create snapshot v_fd313648-5fc1-4e6a-97a0-92ac803b36fd {{(pid=84696) create_snapshot /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:635}} Apr 17 15:58:54 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:58:54 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:58:55 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 3.064(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:58:55 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:58:56 user cinder-volume[84696]: DEBUG cinder.utils [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 1.983(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:58:56 user cinder-volume[84696]: DEBUG cinder.utils [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:58:56 user cinder-volume[84696]: DEBUG cinder.utils [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 72.172(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:58:56 user cinder-volume[84696]: DEBUG cinder.utils [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_45e5cc09-6c97-438a-a6a2-c89e7d725ce5/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:58:56 user cinder-volume[84696]: DEBUG cinder.utils [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 71.998(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:58:56 user cinder-volume[84696]: DEBUG cinder.utils [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:58:59 user cinder-volume[84696]: DEBUG cinder.utils [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.011(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:58:59 user cinder-volume[84696]: DEBUG cinder.utils [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:58:59 user cinder-volume[84696]: DEBUG cinder.utils [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 71.785(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:58:59 user cinder-volume[84696]: DEBUG cinder.utils [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:58:59 user cinder-volume[84696]: DEBUG cinder.utils [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.053(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:58:59 user cinder-volume[84696]: DEBUG cinder.utils [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:58:59 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.139(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:58:59 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:58:59 user cinder-volume[84696]: DEBUG cinder.utils [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.188(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:58:59 user cinder-volume[84696]: DEBUG cinder.utils [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_bff6fdbb-a1fb-4b48-bca8-2f7efbc3f0ff/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:59:05 user cinder-volume[84696]: DEBUG cinder.utils [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 11.195(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:59:05 user cinder-volume[84696]: DEBUG cinder.utils [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:59:05 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.282(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:59:05 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:59:08 user cinder-volume[84696]: DEBUG cinder.utils [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 140.485(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:59:08 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_b29a7592-128a-41ee-a8e2-57780dee33e7/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:59:08 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (d5b56f26-d4f7-4e07-a5e0-44e70a74b0f8) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 15:59:08 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (3abb81ed-068f-4487-b8a2-acfdc172b9ac) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:59:08 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (3abb81ed-068f-4487-b8a2-acfdc172b9ac) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:59:08 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Flow 'volume_create_manager' (1ff833b9-00b2-40ab-bd52-605e4971f142) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 15:59:08 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 15:59:08 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 15:59:08 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 15:59:08 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 15:59:08 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 15:59:08 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 15:59:08 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 15:59:08 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 15:59:08 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 15:59:08 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 15:59:08 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 15:59:08 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 15:59:08 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 15:59:08 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 15:59:08 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 15:59:08 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 15:59:08 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 15:59:08 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 15:59:08 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 15:59:08 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 15:59:08 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 15:59:08 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 15:59:08 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 15:59:08 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 15:59:08 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:59:08 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 15:59:08 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 15:59:08 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 15:59:08 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 15:59:08 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 15:59:08 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 15:59:08 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 15:59:08 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 15:59:08 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 15:59:08 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 15:59:08 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 15:59:08 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 15:59:08 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 15:59:08 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 15:59:08 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 15:59:08 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 15:59:08 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 15:59:08 user cinder-volume[84696]: DEBUG cinder.utils [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 140.325(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:59:08 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:59:08 user cinder-volume[84696]: INFO cinder.message.api [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Creating message record for request_id = req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b Apr 17 15:59:08 user cinder-volume[84696]: DEBUG cinder.utils [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.422(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:59:08 user cinder-volume[84696]: DEBUG cinder.utils [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_bff6fdbb-a1fb-4b48-bca8-2f7efbc3f0ff/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:59:08 user cinder-volume[84696]: DEBUG cinder.utils [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.251(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:59:08 user cinder-volume[84696]: DEBUG cinder.utils [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:59:08 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 15:59:08 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (4f3bedab-a546-444d-94ec-c5ed9348b765) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 15:59:08 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 15:59:08 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:54:30Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumeRevertTests-Volume-921998137',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=b29a7592-128a-41ee-a8e2-57780dee33e7,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='9c755d3b5d8a4d9c9918b6c4744f510e',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:56:48Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:56:48Z,use_quota=True,user_id='e362cfe4a31c444b9ab44a5d60f37ad9',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 15:59:08 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:54:30Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumeRevertTests-Volume-921998137',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=b29a7592-128a-41ee-a8e2-57780dee33e7,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='9c755d3b5d8a4d9c9918b6c4744f510e',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:56:48Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:56:48Z,use_quota=True,user_id='e362cfe4a31c444b9ab44a5d60f37ad9',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(b29a7592-128a-41ee-a8e2-57780dee33e7),volume_id=b29a7592-128a-41ee-a8e2-57780dee33e7,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': 'b29a7592-128a-41ee-a8e2-57780dee33e7', 'volume_name': 'volume-b29a7592-128a-41ee-a8e2-57780dee33e7', 'volume_size': 1}} Apr 17 15:59:08 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': 'b29a7592-128a-41ee-a8e2-57780dee33e7', 'volume': {'id': 'b29a7592-128a-41ee-a8e2-57780dee33e7', '_name_id': None, 'ec2_id': None, 'user_id': 'e362cfe4a31c444b9ab44a5d60f37ad9', 'project_id': '9c755d3b5d8a4d9c9918b6c4744f510e', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-VolumeRevertTests-Volume-921998137', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'admin_metadata': {}, 'glance_metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T15:54:30.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-b29a7592-128a-41ee-a8e2-57780dee33e7', 'name_id': 'b29a7592-128a-41ee-a8e2-57780dee33e7', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumeRevertTests-Volume-921998137', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '9c755d3b5d8a4d9c9918b6c4744f510e', 'qos_specs': None, 'reservations': ['42b0fe0b-67bc-438d-b5b5-5cd8608f187a', '35c09dbe-7331-435e-a061-4f56338d0c2b', '2a16b96e-8144-4c17-a826-7c22f889d9e1', 'a666665d-cbb8-4dfe-9bf7-027ec0df440c'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'e362cfe4a31c444b9ab44a5d60f37ad9', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumeRevertTests-Volume-921998137', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '9c755d3b5d8a4d9c9918b6c4744f510e', 'qos_specs': None, 'reservations': ['42b0fe0b-67bc-438d-b5b5-5cd8608f187a', '35c09dbe-7331-435e-a061-4f56338d0c2b', '2a16b96e-8144-4c17-a826-7c22f889d9e1', 'a666665d-cbb8-4dfe-9bf7-027ec0df440c'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'e362cfe4a31c444b9ab44a5d60f37ad9', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': 'e362cfe4a31c444b9ab44a5d60f37ad9', 'metadata': {}, 'qos_specs': None, 'retry': {'num_attempts': 2, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']}}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(b29a7592-128a-41ee-a8e2-57780dee33e7),volume_id=b29a7592-128a-41ee-a8e2-57780dee33e7,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:54:30Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumeRevertTests-Volume-921998137',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=b29a7592-128a-41ee-a8e2-57780dee33e7,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='9c755d3b5d8a4d9c9918b6c4744f510e',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:56:48Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:56:48Z,use_quota=True,user_id='e362cfe4a31c444b9ab44a5d60f37ad9',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 15:59:08 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:54:30Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumeRevertTests-Volume-921998137',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=b29a7592-128a-41ee-a8e2-57780dee33e7,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='9c755d3b5d8a4d9c9918b6c4744f510e',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:56:48Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:56:48Z,use_quota=True,user_id='e362cfe4a31c444b9ab44a5d60f37ad9',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:54:30Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumeRevertTests-Volume-921998137',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=b29a7592-128a-41ee-a8e2-57780dee33e7,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='9c755d3b5d8a4d9c9918b6c4744f510e',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:56:48Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:56:48Z,use_quota=True,user_id='e362cfe4a31c444b9ab44a5d60f37ad9',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 15:59:08 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 15:59:08 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 15:59:08 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:59:08 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 15:59:08 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 15:59:08 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 15:59:08 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 15:59:08 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 15:59:08 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 15:59:08 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 15:59:08 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 15:59:08 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 15:59:08 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 15:59:08 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 15:59:08 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 15:59:08 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 15:59:08 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 15:59:08 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 15:59:08 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 15:59:08 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 15:59:08 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (4f3bedab-a546-444d-94ec-c5ed9348b765) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:59:08 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (4f3bedab-a546-444d-94ec-c5ed9348b765) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:59:08 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (487419f3-40d1-4f01-9a35-bbbd3ced3e9d) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:59:08 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (487419f3-40d1-4f01-9a35-bbbd3ced3e9d) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:59:08 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (13088d06-9e6f-464a-8ef0-61241efd8932) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:59:08 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (13088d06-9e6f-464a-8ef0-61241efd8932) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:59:08 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (915a393a-340d-41bd-8e81-3465e5e593ca) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:59:08 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Updating volume b29a7592-128a-41ee-a8e2-57780dee33e7 with {'scheduled_at': datetime.datetime(2023, 4, 17, 15, 59, 8, 372187), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 15:59:08 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Volume b29a7592-128a-41ee-a8e2-57780dee33e7: re-scheduling SchedulerAPI.create_volume attempt 2 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 15:59:08 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Volume b29a7592-128a-41ee-a8e2-57780dee33e7: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 15:59:08 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] deleating volume v_b29a7592-128a-41ee-a8e2-57780dee33e7 {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 15:59:08 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] get snapshots for volume v_b29a7592-128a-41ee-a8e2-57780dee33e7 {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 15:59:08 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_b29a7592-128a-41ee-a8e2-57780dee33e7/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:59:08 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_b29a7592-128a-41ee-a8e2-57780dee33e7/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:59:08 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Volume reschedule parameters: True retry: {'num_attempts': 3, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 15:59:08 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Flow 'volume_create_manager' (78b522d0-6048-4086-8445-73b5cdb6254e) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:59:08 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (f38b98c4-bfbe-41d0-bfcc-8ab694d2bcea) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:59:08 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (f38b98c4-bfbe-41d0-bfcc-8ab694d2bcea) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:54:30Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumeRevertTests-Volume-921998137',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=b29a7592-128a-41ee-a8e2-57780dee33e7,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='9c755d3b5d8a4d9c9918b6c4744f510e',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:59:09Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:59:09Z,use_quota=True,user_id='e362cfe4a31c444b9ab44a5d60f37ad9',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:59:08 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (9fda5870-733f-436b-90f4-6a13f218e372) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:59:08 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (9fda5870-733f-436b-90f4-6a13f218e372) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:59:08 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (05bb19e4-e525-42d4-ad48-90f4fc4cbd1c) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:59:08 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (05bb19e4-e525-42d4-ad48-90f4fc4cbd1c) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': 'b29a7592-128a-41ee-a8e2-57780dee33e7', 'volume_name': 'volume-b29a7592-128a-41ee-a8e2-57780dee33e7', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:59:08 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (7c46364d-98d6-442f-b104-6b39969ad9e9) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:59:08 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (7c46364d-98d6-442f-b104-6b39969ad9e9) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:59:08 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (1c9ae51c-3231-4186-bea9-3e2d1b20072f) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:59:08 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Volume b29a7592-128a-41ee-a8e2-57780dee33e7: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-b29a7592-128a-41ee-a8e2-57780dee33e7', 'volume_size': 1} Apr 17 15:59:08 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] creating volume v_b29a7592-128a-41ee-a8e2-57780dee33e7. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 15:59:08 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] create volume {'name': 'v_b29a7592-128a-41ee-a8e2-57780dee33e7', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 15:59:08 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:59:08 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:59:11 user cinder-volume[84696]: DEBUG cinder.utils [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.953(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:59:11 user cinder-volume[84696]: DEBUG cinder.utils [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_b29a7592-128a-41ee-a8e2-57780dee33e7/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:59:11 user cinder-volume[84696]: DEBUG cinder.utils [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.582(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:59:11 user cinder-volume[84696]: DEBUG cinder.utils [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:59:14 user cinder-volume[84696]: DEBUG cinder.utils [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 140.038(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:59:14 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:59:14 user cinder-volume[84696]: WARNING cinder.volume.manager [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (994f4aa9-d679-40ca-ab50-69c4b240fe33) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 15:59:14 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 15:59:14 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:54:35Z,deleted=False,deleted_at=None,display_description='',display_name='',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=75163b78-31b2-4676-b93b-542a8ea784c1,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='540826be4baf46dba9617b2fd47821a3',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:56:54Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:56:54Z,use_quota=True,user_id='d819e3f51368413c99691d1b45bbcfb1',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 15:59:14 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:54:35Z,deleted=False,deleted_at=None,display_description='',display_name='',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=75163b78-31b2-4676-b93b-542a8ea784c1,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='540826be4baf46dba9617b2fd47821a3',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:56:54Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:56:54Z,use_quota=True,user_id='d819e3f51368413c99691d1b45bbcfb1',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=b3f0dec1-7758-44a8-82e0-76ada8837428,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(75163b78-31b2-4676-b93b-542a8ea784c1),volume_id=75163b78-31b2-4676-b93b-542a8ea784c1,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'image', 'volume_id': '75163b78-31b2-4676-b93b-542a8ea784c1', 'volume_name': 'volume-75163b78-31b2-4676-b93b-542a8ea784c1', 'volume_size': 1, 'image_id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'image_location': (None, None), 'image_meta': {'name': 'cirros-0.5.2-x86_64-disk', 'disk_format': 'qcow2', 'container_format': 'bare', 'visibility': 'public', 'size': 16300544, 'virtual_size': 117440512, 'status': 'active', 'checksum': 'b874c39491a2377b8490f5f1e89761a4', 'protected': False, 'min_ram': 0, 'min_disk': 0, 'owner': 'a43c211b4ed542538799349489b1d884', 'os_hidden': False, 'os_hash_algo': 'sha512', 'os_hash_value': '6b813aa46bb90b4da216a4d19376593fa3f4fc7e617f03a92b7fe11e9a3981cbe8f0959dbebe36225e5f53dc4492341a4863cac4ed1ee0909f3fc78ef9c3e869', 'id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'created_at': datetime.datetime(2023, 4, 17, 15, 19, 53, tzinfo=datetime.timezone.utc), 'updated_at': datetime.datetime(2023, 4, 17, 15, 19, 55, tzinfo=datetime.timezone.utc), 'tags': [], 'file': '/v2/images/b3f0dec1-7758-44a8-82e0-76ada8837428/file', 'properties': {'hw_rng_model': 'virtio', 'owner_specified.openstack.md5': '', 'owner_specified.openstack.object': 'images/cirros-0.5.2-x86_64-disk', 'owner_specified.openstack.sha256': ''}}, 'image_service': }} Apr 17 15:59:14 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'snapshot_id': None, 'source_volid': None, 'volume_id': '75163b78-31b2-4676-b93b-542a8ea784c1', 'volume': {'id': '75163b78-31b2-4676-b93b-542a8ea784c1', '_name_id': None, 'ec2_id': None, 'user_id': 'd819e3f51368413c99691d1b45bbcfb1', 'project_id': '540826be4baf46dba9617b2fd47821a3', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': '', 'display_description': '', 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'admin_metadata': {}, 'glance_metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T15:54:35.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-75163b78-31b2-4676-b93b-542a8ea784c1', 'name_id': '75163b78-31b2-4676-b93b-542a8ea784c1', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': '', 'display_name': '', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '540826be4baf46dba9617b2fd47821a3', 'qos_specs': None, 'reservations': ['351bb8e5-5094-4129-bd6a-8f086342ae4f', '4dc05382-d603-486b-ab14-e1caee8118dd', 'b04f0e6c-aaea-44ea-b237-e337edc22236', 'e402f3ac-509b-4dac-8852-9c619b150ea3'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'd819e3f51368413c99691d1b45bbcfb1', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': '', 'display_name': '', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '540826be4baf46dba9617b2fd47821a3', 'qos_specs': None, 'reservations': ['351bb8e5-5094-4129-bd6a-8f086342ae4f', '4dc05382-d603-486b-ab14-e1caee8118dd', 'b04f0e6c-aaea-44ea-b237-e337edc22236', 'e402f3ac-509b-4dac-8852-9c619b150ea3'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'd819e3f51368413c99691d1b45bbcfb1', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': 'd819e3f51368413c99691d1b45bbcfb1', 'metadata': {}, 'qos_specs': None, 'retry': {'num_attempts': 2, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']}}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=b3f0dec1-7758-44a8-82e0-76ada8837428,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(75163b78-31b2-4676-b93b-542a8ea784c1),volume_id=75163b78-31b2-4676-b93b-542a8ea784c1,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:54:35Z,deleted=False,deleted_at=None,display_description='',display_name='',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=75163b78-31b2-4676-b93b-542a8ea784c1,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='540826be4baf46dba9617b2fd47821a3',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:56:54Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:56:54Z,use_quota=True,user_id='d819e3f51368413c99691d1b45bbcfb1',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 15:59:14 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:54:35Z,deleted=False,deleted_at=None,display_description='',display_name='',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=75163b78-31b2-4676-b93b-542a8ea784c1,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='540826be4baf46dba9617b2fd47821a3',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:56:54Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:56:54Z,use_quota=True,user_id='d819e3f51368413c99691d1b45bbcfb1',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:54:35Z,deleted=False,deleted_at=None,display_description='',display_name='',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=75163b78-31b2-4676-b93b-542a8ea784c1,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='540826be4baf46dba9617b2fd47821a3',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:56:54Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:56:54Z,use_quota=True,user_id='d819e3f51368413c99691d1b45bbcfb1',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 15:59:14 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 15:59:14 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 15:59:14 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:59:14 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 15:59:14 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1237, in execute Apr 17 15:59:14 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_from_image(context, Apr 17 15:59:14 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/utils.py", line 670, in _wrapper Apr 17 15:59:14 user cinder-volume[84696]: ERROR cinder.volume.manager return r(f, *args, **kwargs) Apr 17 15:59:14 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/tenacity/__init__.py", line 423, in __call__ Apr 17 15:59:14 user cinder-volume[84696]: ERROR cinder.volume.manager do = self.iter(retry_state=retry_state) Apr 17 15:59:14 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/tenacity/__init__.py", line 360, in iter Apr 17 15:59:14 user cinder-volume[84696]: ERROR cinder.volume.manager return fut.result() Apr 17 15:59:14 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/lib/python3.10/concurrent/futures/_base.py", line 451, in result Apr 17 15:59:14 user cinder-volume[84696]: ERROR cinder.volume.manager return self.__get_result() Apr 17 15:59:14 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result Apr 17 15:59:14 user cinder-volume[84696]: ERROR cinder.volume.manager raise self._exception Apr 17 15:59:14 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/tenacity/__init__.py", line 426, in __call__ Apr 17 15:59:14 user cinder-volume[84696]: ERROR cinder.volume.manager result = fn(*args, **kwargs) Apr 17 15:59:14 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1125, in _create_from_image Apr 17 15:59:14 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_from_image_cache_or_download( Apr 17 15:59:14 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 944, in _create_from_image_cache_or_download Apr 17 15:59:14 user cinder-volume[84696]: ERROR cinder.volume.manager model_update, cloned = self._create_from_image_cache( Apr 17 15:59:14 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 843, in _create_from_image_cache Apr 17 15:59:14 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_from_source_volume( Apr 17 15:59:14 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 659, in _create_from_source_volume Apr 17 15:59:14 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self.driver.create_cloned_volume(volume, srcvol_ref) Apr 17 15:59:14 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 525, in create_cloned_volume Apr 17 15:59:14 user cinder-volume[84696]: ERROR cinder.volume.manager self._clone_object(vname, cvname) Apr 17 15:59:14 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 461, in _clone_object Apr 17 15:59:14 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_snapshot(oname, coname) Apr 17 15:59:14 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 639, in create_snapshot Apr 17 15:59:14 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201, 204): Apr 17 15:59:14 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 15:59:14 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 15:59:14 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (994f4aa9-d679-40ca-ab50-69c4b240fe33) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:59:14 user cinder-volume[84696]: WARNING cinder.volume.manager [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (994f4aa9-d679-40ca-ab50-69c4b240fe33) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:59:14 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (c93c4cc0-9f18-4132-9b7a-2f91ee796536) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:59:14 user cinder-volume[84696]: WARNING cinder.volume.manager [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (c93c4cc0-9f18-4132-9b7a-2f91ee796536) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:59:14 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (d4b29370-88cf-4816-b75b-7ce60e9dd025) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:59:14 user cinder-volume[84696]: WARNING cinder.volume.manager [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (d4b29370-88cf-4816-b75b-7ce60e9dd025) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:59:14 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (9ebb7d34-65cd-4f48-a5e2-e36a75702ea7) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:59:14 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Updating volume 75163b78-31b2-4676-b93b-542a8ea784c1 with {'scheduled_at': datetime.datetime(2023, 4, 17, 15, 59, 14, 473621), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 15:59:14 user cinder-volume[84696]: DEBUG cinder.utils [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 139.068(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:59:14 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:59:14 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (e424f74d-587b-48a5-95ce-cb5f915cf222) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 15:59:14 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 15:59:14 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:52:20Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TestVolumeBootPattern-volume-origin-952290434',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=60979eba-96b1-4cb3-bae1-8874a86cb411,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='c8a836b47d954cb9a7c1cd7c0f790426',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:56:54Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:56:54Z,use_quota=True,user_id='a80e753322224ec1a897598b5b9ed6ea',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 15:59:14 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:52:20Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TestVolumeBootPattern-volume-origin-952290434',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=60979eba-96b1-4cb3-bae1-8874a86cb411,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='c8a836b47d954cb9a7c1cd7c0f790426',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:56:54Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:56:54Z,use_quota=True,user_id='a80e753322224ec1a897598b5b9ed6ea',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=b3f0dec1-7758-44a8-82e0-76ada8837428,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(60979eba-96b1-4cb3-bae1-8874a86cb411),volume_id=60979eba-96b1-4cb3-bae1-8874a86cb411,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'image', 'volume_id': '60979eba-96b1-4cb3-bae1-8874a86cb411', 'volume_name': 'volume-60979eba-96b1-4cb3-bae1-8874a86cb411', 'volume_size': 1, 'image_id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'image_location': (None, None), 'image_meta': {'name': 'cirros-0.5.2-x86_64-disk', 'disk_format': 'qcow2', 'container_format': 'bare', 'visibility': 'public', 'size': 16300544, 'virtual_size': 117440512, 'status': 'active', 'checksum': 'b874c39491a2377b8490f5f1e89761a4', 'protected': False, 'min_ram': 0, 'min_disk': 0, 'owner': 'a43c211b4ed542538799349489b1d884', 'os_hidden': False, 'os_hash_algo': 'sha512', 'os_hash_value': '6b813aa46bb90b4da216a4d19376593fa3f4fc7e617f03a92b7fe11e9a3981cbe8f0959dbebe36225e5f53dc4492341a4863cac4ed1ee0909f3fc78ef9c3e869', 'id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'created_at': datetime.datetime(2023, 4, 17, 15, 19, 53, tzinfo=datetime.timezone.utc), 'updated_at': datetime.datetime(2023, 4, 17, 15, 19, 55, tzinfo=datetime.timezone.utc), 'tags': [], 'file': '/v2/images/b3f0dec1-7758-44a8-82e0-76ada8837428/file', 'properties': {'hw_rng_model': 'virtio', 'owner_specified.openstack.md5': '', 'owner_specified.openstack.object': 'images/cirros-0.5.2-x86_64-disk', 'owner_specified.openstack.sha256': ''}}, 'image_service': }} Apr 17 15:59:14 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'snapshot_id': None, 'source_volid': None, 'volume_id': '60979eba-96b1-4cb3-bae1-8874a86cb411', 'volume': {'id': '60979eba-96b1-4cb3-bae1-8874a86cb411', '_name_id': None, 'ec2_id': None, 'user_id': 'a80e753322224ec1a897598b5b9ed6ea', 'project_id': 'c8a836b47d954cb9a7c1cd7c0f790426', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-TestVolumeBootPattern-volume-origin-952290434', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'admin_metadata': {}, 'glance_metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T15:52:19.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-60979eba-96b1-4cb3-bae1-8874a86cb411', 'name_id': '60979eba-96b1-4cb3-bae1-8874a86cb411', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-TestVolumeBootPattern-volume-origin-952290434', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': 'c8a836b47d954cb9a7c1cd7c0f790426', 'qos_specs': None, 'reservations': ['665fbb3e-6435-4ad5-b020-211f010efcfd', '6df9e628-e8f9-4f25-b1cc-f04ca7f0e972', '0f4da886-833a-4bdb-a9a5-4937f5294c9a', 'f63e6f69-e592-4e3a-8f7a-79eaf32aacf0'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'a80e753322224ec1a897598b5b9ed6ea', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-TestVolumeBootPattern-volume-origin-952290434', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': 'c8a836b47d954cb9a7c1cd7c0f790426', 'qos_specs': None, 'reservations': ['665fbb3e-6435-4ad5-b020-211f010efcfd', '6df9e628-e8f9-4f25-b1cc-f04ca7f0e972', '0f4da886-833a-4bdb-a9a5-4937f5294c9a', 'f63e6f69-e592-4e3a-8f7a-79eaf32aacf0'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'a80e753322224ec1a897598b5b9ed6ea', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': 'a80e753322224ec1a897598b5b9ed6ea', 'metadata': {}, 'qos_specs': None, 'retry': {'num_attempts': 3, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']}}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=b3f0dec1-7758-44a8-82e0-76ada8837428,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(60979eba-96b1-4cb3-bae1-8874a86cb411),volume_id=60979eba-96b1-4cb3-bae1-8874a86cb411,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:52:20Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TestVolumeBootPattern-volume-origin-952290434',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=60979eba-96b1-4cb3-bae1-8874a86cb411,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='c8a836b47d954cb9a7c1cd7c0f790426',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:56:54Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:56:54Z,use_quota=True,user_id='a80e753322224ec1a897598b5b9ed6ea',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 15:59:14 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:52:20Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TestVolumeBootPattern-volume-origin-952290434',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=60979eba-96b1-4cb3-bae1-8874a86cb411,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='c8a836b47d954cb9a7c1cd7c0f790426',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:56:54Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:56:54Z,use_quota=True,user_id='a80e753322224ec1a897598b5b9ed6ea',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:52:20Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TestVolumeBootPattern-volume-origin-952290434',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=60979eba-96b1-4cb3-bae1-8874a86cb411,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='c8a836b47d954cb9a7c1cd7c0f790426',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:56:54Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:56:54Z,use_quota=True,user_id='a80e753322224ec1a897598b5b9ed6ea',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 15:59:14 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 15:59:14 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 15:59:14 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:59:14 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 15:59:14 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1237, in execute Apr 17 15:59:14 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_from_image(context, Apr 17 15:59:14 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/utils.py", line 670, in _wrapper Apr 17 15:59:14 user cinder-volume[84696]: ERROR cinder.volume.manager return r(f, *args, **kwargs) Apr 17 15:59:14 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/tenacity/__init__.py", line 423, in __call__ Apr 17 15:59:14 user cinder-volume[84696]: ERROR cinder.volume.manager do = self.iter(retry_state=retry_state) Apr 17 15:59:14 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/tenacity/__init__.py", line 360, in iter Apr 17 15:59:14 user cinder-volume[84696]: ERROR cinder.volume.manager return fut.result() Apr 17 15:59:14 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/lib/python3.10/concurrent/futures/_base.py", line 451, in result Apr 17 15:59:14 user cinder-volume[84696]: ERROR cinder.volume.manager return self.__get_result() Apr 17 15:59:14 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result Apr 17 15:59:14 user cinder-volume[84696]: ERROR cinder.volume.manager raise self._exception Apr 17 15:59:14 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/tenacity/__init__.py", line 426, in __call__ Apr 17 15:59:14 user cinder-volume[84696]: ERROR cinder.volume.manager result = fn(*args, **kwargs) Apr 17 15:59:14 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1125, in _create_from_image Apr 17 15:59:14 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_from_image_cache_or_download( Apr 17 15:59:14 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 944, in _create_from_image_cache_or_download Apr 17 15:59:14 user cinder-volume[84696]: ERROR cinder.volume.manager model_update, cloned = self._create_from_image_cache( Apr 17 15:59:14 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 843, in _create_from_image_cache Apr 17 15:59:14 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_from_source_volume( Apr 17 15:59:14 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 659, in _create_from_source_volume Apr 17 15:59:14 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self.driver.create_cloned_volume(volume, srcvol_ref) Apr 17 15:59:14 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 525, in create_cloned_volume Apr 17 15:59:14 user cinder-volume[84696]: ERROR cinder.volume.manager self._clone_object(vname, cvname) Apr 17 15:59:14 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 461, in _clone_object Apr 17 15:59:14 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_snapshot(oname, coname) Apr 17 15:59:14 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 639, in create_snapshot Apr 17 15:59:14 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201, 204): Apr 17 15:59:14 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 15:59:14 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 15:59:14 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (e424f74d-587b-48a5-95ce-cb5f915cf222) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:59:14 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (e424f74d-587b-48a5-95ce-cb5f915cf222) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:59:14 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (2a119d5d-67ee-4e23-aab1-ba8763a728e7) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:59:14 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (2a119d5d-67ee-4e23-aab1-ba8763a728e7) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:59:14 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (6646d7ca-1d1c-4a96-8aa3-70fff100ad4f) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:59:14 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (6646d7ca-1d1c-4a96-8aa3-70fff100ad4f) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:59:14 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (4ff6d4e8-85fa-467e-acdc-870b80c1f4fc) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:59:14 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Updating volume 60979eba-96b1-4cb3-bae1-8874a86cb411 with {'scheduled_at': datetime.datetime(2023, 4, 17, 15, 59, 14, 518211), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 15:59:14 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=16,cinder:UPDATE=11,cinder:INSERT=1 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:59:14 user cinder-volume[84696]: DEBUG cinder.utils [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 140.559(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:59:14 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_75163b78-31b2-4676-b93b-542a8ea784c1/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:59:14 user cinder-volume[84696]: WARNING cinder.volume.manager [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (6f50d91f-d980-44f9-8b14-89e1707bab15) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 15:59:14 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (5806b395-60a7-4762-a8a5-a5aa0b077236) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:59:14 user cinder-volume[84696]: WARNING cinder.volume.manager [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (5806b395-60a7-4762-a8a5-a5aa0b077236) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:59:14 user cinder-volume[84696]: WARNING cinder.volume.manager [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Flow 'volume_create_manager' (64bc9ae2-f8ff-4daa-8349-c4488d54df95) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 15:59:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 15:59:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 15:59:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 15:59:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 15:59:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 15:59:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 15:59:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 15:59:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 15:59:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 15:59:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 15:59:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 15:59:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 15:59:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 15:59:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 15:59:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 15:59:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 15:59:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 15:59:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 15:59:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 15:59:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 15:59:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 15:59:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 15:59:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 15:59:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 15:59:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:59:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 15:59:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1237, in execute Apr 17 15:59:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_from_image(context, Apr 17 15:59:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/utils.py", line 670, in _wrapper Apr 17 15:59:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return r(f, *args, **kwargs) Apr 17 15:59:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/tenacity/__init__.py", line 423, in __call__ Apr 17 15:59:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server do = self.iter(retry_state=retry_state) Apr 17 15:59:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/tenacity/__init__.py", line 360, in iter Apr 17 15:59:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return fut.result() Apr 17 15:59:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/lib/python3.10/concurrent/futures/_base.py", line 451, in result Apr 17 15:59:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self.__get_result() Apr 17 15:59:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result Apr 17 15:59:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self._exception Apr 17 15:59:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/tenacity/__init__.py", line 426, in __call__ Apr 17 15:59:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = fn(*args, **kwargs) Apr 17 15:59:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1125, in _create_from_image Apr 17 15:59:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_from_image_cache_or_download( Apr 17 15:59:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 944, in _create_from_image_cache_or_download Apr 17 15:59:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update, cloned = self._create_from_image_cache( Apr 17 15:59:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 843, in _create_from_image_cache Apr 17 15:59:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_from_source_volume( Apr 17 15:59:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 659, in _create_from_source_volume Apr 17 15:59:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self.driver.create_cloned_volume(volume, srcvol_ref) Apr 17 15:59:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 525, in create_cloned_volume Apr 17 15:59:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self._clone_object(vname, cvname) Apr 17 15:59:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 461, in _clone_object Apr 17 15:59:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_snapshot(oname, coname) Apr 17 15:59:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 639, in create_snapshot Apr 17 15:59:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201, 204): Apr 17 15:59:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 15:59:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 15:59:14 user cinder-volume[84696]: DEBUG cinder.utils [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 20.522(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:59:14 user cinder-volume[84696]: DEBUG cinder.utils [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:59:14 user cinder-volume[84696]: DEBUG cinder.utils [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 140.568(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:59:14 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_60979eba-96b1-4cb3-bae1-8874a86cb411/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:59:14 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (19719143-3e90-493d-82f6-93075aeceb7d) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 15:59:14 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (0e29ba6a-c3ac-42a8-bfd2-4a9b9ce89b60) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:59:14 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (0e29ba6a-c3ac-42a8-bfd2-4a9b9ce89b60) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:59:14 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Flow 'volume_create_manager' (cab37801-3312-4efa-820f-bcf7a206953d) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 15:59:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 15:59:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 15:59:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 15:59:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 15:59:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 15:59:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 15:59:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 15:59:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 15:59:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 15:59:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 15:59:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 15:59:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 15:59:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 15:59:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 15:59:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 15:59:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 15:59:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 15:59:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 15:59:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 15:59:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 15:59:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 15:59:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 15:59:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 15:59:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 15:59:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:59:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 15:59:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1237, in execute Apr 17 15:59:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_from_image(context, Apr 17 15:59:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/utils.py", line 670, in _wrapper Apr 17 15:59:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return r(f, *args, **kwargs) Apr 17 15:59:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/tenacity/__init__.py", line 423, in __call__ Apr 17 15:59:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server do = self.iter(retry_state=retry_state) Apr 17 15:59:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/tenacity/__init__.py", line 360, in iter Apr 17 15:59:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return fut.result() Apr 17 15:59:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/lib/python3.10/concurrent/futures/_base.py", line 451, in result Apr 17 15:59:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self.__get_result() Apr 17 15:59:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result Apr 17 15:59:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self._exception Apr 17 15:59:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/tenacity/__init__.py", line 426, in __call__ Apr 17 15:59:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = fn(*args, **kwargs) Apr 17 15:59:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1125, in _create_from_image Apr 17 15:59:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_from_image_cache_or_download( Apr 17 15:59:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 944, in _create_from_image_cache_or_download Apr 17 15:59:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update, cloned = self._create_from_image_cache( Apr 17 15:59:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 843, in _create_from_image_cache Apr 17 15:59:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_from_source_volume( Apr 17 15:59:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 659, in _create_from_source_volume Apr 17 15:59:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self.driver.create_cloned_volume(volume, srcvol_ref) Apr 17 15:59:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 525, in create_cloned_volume Apr 17 15:59:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self._clone_object(vname, cvname) Apr 17 15:59:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 461, in _clone_object Apr 17 15:59:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_snapshot(oname, coname) Apr 17 15:59:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 639, in create_snapshot Apr 17 15:59:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201, 204): Apr 17 15:59:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 15:59:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 15:59:14 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.624(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:59:14 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:59:14 user cinder-volume[84696]: DEBUG cinder.utils [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.154(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:59:14 user cinder-volume[84696]: DEBUG cinder.utils [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_b29a7592-128a-41ee-a8e2-57780dee33e7/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:59:14 user cinder-volume[84696]: DEBUG cinder.utils [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.783(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:59:14 user cinder-volume[84696]: DEBUG cinder.utils [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:59:14 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=26,cinder:INSERT=3,cinder:UPDATE=3 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:59:14 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Volume 75163b78-31b2-4676-b93b-542a8ea784c1: re-scheduling SchedulerAPI.create_volume attempt 2 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 15:59:14 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Volume 60979eba-96b1-4cb3-bae1-8874a86cb411: re-scheduling SchedulerAPI.create_volume attempt 3 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 15:59:14 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Volume 75163b78-31b2-4676-b93b-542a8ea784c1: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 15:59:14 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] deleating volume v_75163b78-31b2-4676-b93b-542a8ea784c1 {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 15:59:14 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] get snapshots for volume v_75163b78-31b2-4676-b93b-542a8ea784c1 {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 15:59:14 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_75163b78-31b2-4676-b93b-542a8ea784c1/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:59:14 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_75163b78-31b2-4676-b93b-542a8ea784c1/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:59:14 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Volume 60979eba-96b1-4cb3-bae1-8874a86cb411: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 15:59:14 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] deleating volume v_60979eba-96b1-4cb3-bae1-8874a86cb411 {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 15:59:14 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] get snapshots for volume v_60979eba-96b1-4cb3-bae1-8874a86cb411 {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 15:59:14 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_60979eba-96b1-4cb3-bae1-8874a86cb411/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:59:14 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_60979eba-96b1-4cb3-bae1-8874a86cb411/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:59:14 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Volume reschedule parameters: True retry: {'num_attempts': 3, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 15:59:14 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Flow 'volume_create_manager' (b30c9810-66a7-4121-a690-a2f9ca833f5a) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:59:14 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (ac918f7e-a49c-4935-abb8-d89e944e8841) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:59:14 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (ac918f7e-a49c-4935-abb8-d89e944e8841) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:54:35Z,deleted=False,deleted_at=None,display_description='',display_name='',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=75163b78-31b2-4676-b93b-542a8ea784c1,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='540826be4baf46dba9617b2fd47821a3',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:59:15Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:59:15Z,use_quota=True,user_id='d819e3f51368413c99691d1b45bbcfb1',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:59:14 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (c46062ad-beb8-4f66-9e1e-d87f80e69bbb) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:59:14 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (c46062ad-beb8-4f66-9e1e-d87f80e69bbb) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:59:14 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (619e164b-5a0a-4ce1-8b8b-ebcf68c81389) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:59:14 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (619e164b-5a0a-4ce1-8b8b-ebcf68c81389) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'image', 'volume_id': '75163b78-31b2-4676-b93b-542a8ea784c1', 'volume_name': 'volume-75163b78-31b2-4676-b93b-542a8ea784c1', 'volume_size': 1, 'image_id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'image_location': (None, None), 'image_meta': {'name': 'cirros-0.5.2-x86_64-disk', 'disk_format': 'qcow2', 'container_format': 'bare', 'visibility': 'public', 'size': 16300544, 'virtual_size': 117440512, 'status': 'active', 'checksum': 'b874c39491a2377b8490f5f1e89761a4', 'protected': False, 'min_ram': 0, 'min_disk': 0, 'owner': 'a43c211b4ed542538799349489b1d884', 'os_hidden': False, 'os_hash_algo': 'sha512', 'os_hash_value': '6b813aa46bb90b4da216a4d19376593fa3f4fc7e617f03a92b7fe11e9a3981cbe8f0959dbebe36225e5f53dc4492341a4863cac4ed1ee0909f3fc78ef9c3e869', 'id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'created_at': datetime.datetime(2023, 4, 17, 15, 19, 53, tzinfo=datetime.timezone.utc), 'updated_at': datetime.datetime(2023, 4, 17, 15, 19, 55, tzinfo=datetime.timezone.utc), 'tags': [], 'file': '/v2/images/b3f0dec1-7758-44a8-82e0-76ada8837428/file', 'properties': {'hw_rng_model': 'virtio', 'owner_specified.openstack.md5': '', 'owner_specified.openstack.object': 'images/cirros-0.5.2-x86_64-disk', 'owner_specified.openstack.sha256': ''}}, 'image_service': }' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:59:14 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (0ec43a89-b097-4912-bab1-f269151f787a) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:59:14 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (0ec43a89-b097-4912-bab1-f269151f787a) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:59:14 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (91febb26-0692-4dda-bb8a-1ac82a6371e0) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:59:14 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Volume 75163b78-31b2-4676-b93b-542a8ea784c1: being created as image with specification: {'status': 'creating', 'volume_name': 'volume-75163b78-31b2-4676-b93b-542a8ea784c1', 'volume_size': 1, 'image_id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'image_location': (None, None), 'image_meta': {'name': 'cirros-0.5.2-x86_64-disk', 'disk_format': 'qcow2', 'container_format': 'bare', 'visibility': 'public', 'size': 16300544, 'virtual_size': 117440512, 'status': 'active', 'checksum': 'b874c39491a2377b8490f5f1e89761a4', 'protected': False, 'min_ram': 0, 'min_disk': 0, 'owner': 'a43c211b4ed542538799349489b1d884', 'os_hidden': False, 'os_hash_algo': 'sha512', 'os_hash_value': '6b813aa46bb90b4da216a4d19376593fa3f4fc7e617f03a92b7fe11e9a3981cbe8f0959dbebe36225e5f53dc4492341a4863cac4ed1ee0909f3fc78ef9c3e869', 'id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'created_at': datetime.datetime(2023, 4, 17, 15, 19, 53, tzinfo=datetime.timezone.utc), 'updated_at': datetime.datetime(2023, 4, 17, 15, 19, 55, tzinfo=datetime.timezone.utc), 'tags': [], 'file': '/v2/images/b3f0dec1-7758-44a8-82e0-76ada8837428/file', 'properties': {'hw_rng_model': 'virtio', 'owner_specified.openstack.md5': '', 'owner_specified.openstack.object': 'images/cirros-0.5.2-x86_64-disk', 'owner_specified.openstack.sha256': ''}}, 'image_service': } Apr 17 15:59:14 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Cloning 75163b78-31b2-4676-b93b-542a8ea784c1 from image b3f0dec1-7758-44a8-82e0-76ada8837428 at location (None, None). {{(pid=84696) _create_from_image /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1071}} Apr 17 15:59:14 user cinder-volume[84696]: DEBUG cinder.coordination [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Acquiring lock "cinder-b3f0dec1-7758-44a8-82e0-76ada8837428" by "_prepare_image_cache_entry" {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:192}} Apr 17 15:59:14 user cinder-volume[84696]: DEBUG cinder.coordination [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Lock "cinder-b3f0dec1-7758-44a8-82e0-76ada8837428" acquired by "_prepare_image_cache_entry" :: waited 0.008s {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:196}} Apr 17 15:59:14 user cinder-volume[84696]: DEBUG cinder.image.cache [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Found image-volume cache entry: {'id': 1, 'image_id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'volume_id': '006f20ae-5306-44fe-afdf-e433f79284e8', 'host': 'user@joviandss-2#joviandss-2', 'size': 1, 'image_updated_at': datetime.datetime(2023, 4, 17, 15, 19, 55), 'last_used': datetime.datetime(2023, 4, 17, 15, 59, 14, 899306)}. {{(pid=84696) get_entry /opt/stack/cinder/cinder/image/cache.py:76}} Apr 17 15:59:14 user cinder-volume[84696]: DEBUG cinder.image.cache [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Image-volume cache entry image_update_at = 2023-04-17 15:19:55+00:00, requested image updated_at = 2023-04-17 15:19:55+00:00. {{(pid=84696) _should_update_entry /opt/stack/cinder/cinder/image/cache.py:259}} Apr 17 15:59:14 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Found cache entry for image = b3f0dec1-7758-44a8-82e0-76ada8837428 on host user@joviandss-2#joviandss-2. {{(pid=84696) _prepare_image_cache_entry /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:886}} Apr 17 15:59:14 user cinder-volume[84696]: DEBUG cinder.coordination [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Lock "cinder-b3f0dec1-7758-44a8-82e0-76ada8837428" released by "_prepare_image_cache_entry" :: held 0.014s {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:207}} Apr 17 15:59:14 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Attempting to retrieve cache entry for image = b3f0dec1-7758-44a8-82e0-76ada8837428 on host user@joviandss-2#joviandss-2. {{(pid=84696) _create_from_image_cache /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:827}} Apr 17 15:59:14 user cinder-volume[84696]: DEBUG cinder.image.cache [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Found image-volume cache entry: {'id': 1, 'image_id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'volume_id': '006f20ae-5306-44fe-afdf-e433f79284e8', 'host': 'user@joviandss-2#joviandss-2', 'size': 1, 'image_updated_at': datetime.datetime(2023, 4, 17, 15, 19, 55), 'last_used': datetime.datetime(2023, 4, 17, 15, 59, 14, 913206)}. {{(pid=84696) get_entry /opt/stack/cinder/cinder/image/cache.py:76}} Apr 17 15:59:14 user cinder-volume[84696]: DEBUG cinder.image.cache [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Image-volume cache entry image_update_at = 2023-04-17 15:19:55+00:00, requested image updated_at = 2023-04-17 15:19:55+00:00. {{(pid=84696) _should_update_entry /opt/stack/cinder/cinder/image/cache.py:259}} Apr 17 15:59:14 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Creating from source image-volume 006f20ae-5306-44fe-afdf-e433f79284e8 {{(pid=84696) _create_from_image_cache /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:841}} Apr 17 15:59:14 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] cloned volume 006f20ae-5306-44fe-afdf-e433f79284e8 to 75163b78-31b2-4676-b93b-542a8ea784c1 {{(pid=84696) create_cloned_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:521}} Apr 17 15:59:14 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] cloning v_006f20ae-5306-44fe-afdf-e433f79284e8 to v_75163b78-31b2-4676-b93b-542a8ea784c1 {{(pid=84696) _clone_object /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:456}} Apr 17 15:59:14 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] create snapshot v_75163b78-31b2-4676-b93b-542a8ea784c1 {{(pid=84696) create_snapshot /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:635}} Apr 17 15:59:14 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:59:14 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:59:17 user cinder-volume[84696]: DEBUG cinder.utils [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.913(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:59:17 user cinder-volume[84696]: DEBUG cinder.utils [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_75163b78-31b2-4676-b93b-542a8ea784c1/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:59:17 user cinder-volume[84696]: DEBUG cinder.utils [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.574(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:59:17 user cinder-volume[84696]: DEBUG cinder.utils [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:59:17 user cinder-volume[84696]: DEBUG cinder.utils [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.909(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:59:17 user cinder-volume[84696]: DEBUG cinder.utils [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_60979eba-96b1-4cb3-bae1-8874a86cb411/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:59:20 user cinder-volume[84696]: DEBUG cinder.utils [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 11.797(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:59:20 user cinder-volume[84696]: DEBUG cinder.utils [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:59:20 user cinder-volume[84696]: DEBUG cinder.utils [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.171(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:59:20 user cinder-volume[84696]: DEBUG cinder.utils [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_b29a7592-128a-41ee-a8e2-57780dee33e7/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:59:20 user cinder-volume[84696]: DEBUG cinder.utils [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 140.483(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:59:20 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_2ed5b8aa-fb48-4255-b6d5-f0f84f733834/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:59:20 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (4dc5a06a-4105-45c6-b762-a06c80fed045) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 15:59:20 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (8967ebb0-115e-4d82-baad-be32a1566790) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:59:20 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (8967ebb0-115e-4d82-baad-be32a1566790) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:59:20 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Flow 'volume_create_manager' (822d2eb8-53f2-47c4-9634-d531c3e46104) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 15:59:20 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-4381eeaf-f88c-49c3-a64f-a647fc1596cd tempest-ServerActionsTestJSON-645547919 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 15:59:20 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 15:59:20 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 15:59:20 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 15:59:20 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 15:59:20 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 15:59:20 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 15:59:20 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 15:59:20 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 15:59:20 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 15:59:20 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 15:59:20 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 15:59:20 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 15:59:20 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 15:59:20 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 15:59:20 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 15:59:20 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 15:59:20 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 15:59:20 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 15:59:20 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 15:59:20 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 15:59:20 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 15:59:20 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 15:59:20 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 15:59:20 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:59:20 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 15:59:20 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 15:59:20 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 15:59:20 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 15:59:20 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 15:59:20 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 15:59:20 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 15:59:20 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 15:59:20 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 15:59:20 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 15:59:20 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 15:59:20 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 15:59:20 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 15:59:20 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 15:59:20 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 15:59:20 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 15:59:20 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 15:59:20 user cinder-volume[84696]: DEBUG cinder.utils [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.663(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:59:20 user cinder-volume[84696]: DEBUG cinder.utils [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:59:20 user cinder-volume[84696]: DEBUG cinder.utils [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.005(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:59:20 user cinder-volume[84696]: DEBUG cinder.utils [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_75163b78-31b2-4676-b93b-542a8ea784c1/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:59:20 user cinder-volume[84696]: DEBUG cinder.utils [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.997(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:59:20 user cinder-volume[84696]: DEBUG cinder.utils [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_60979eba-96b1-4cb3-bae1-8874a86cb411/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:59:26 user cinder-volume[84696]: DEBUG cinder.utils [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 11.816(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:59:26 user cinder-volume[84696]: DEBUG cinder.utils [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:59:26 user cinder-volume[84696]: DEBUG cinder.utils [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.867(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:59:26 user cinder-volume[84696]: DEBUG cinder.utils [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_bff6fdbb-a1fb-4b48-bca8-2f7efbc3f0ff/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:59:26 user cinder-volume[84696]: DEBUG cinder.utils [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.696(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:59:26 user cinder-volume[84696]: DEBUG cinder.utils [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:59:26 user cinder-volume[84696]: DEBUG cinder.utils [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.162(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:59:26 user cinder-volume[84696]: DEBUG cinder.utils [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_75163b78-31b2-4676-b93b-542a8ea784c1/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:59:26 user cinder-volume[84696]: DEBUG cinder.utils [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.155(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:59:26 user cinder-volume[84696]: DEBUG cinder.utils [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_60979eba-96b1-4cb3-bae1-8874a86cb411/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:59:29 user cinder-volume[84696]: DEBUG cinder.utils [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.421(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:59:29 user cinder-volume[84696]: DEBUG cinder.utils [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_b29a7592-128a-41ee-a8e2-57780dee33e7/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:59:29 user cinder-volume[84696]: DEBUG cinder.utils [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.051(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:59:29 user cinder-volume[84696]: DEBUG cinder.utils [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:59:32 user cinder-volume[84696]: DEBUG cinder.utils [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 38.875(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:59:32 user cinder-volume[84696]: DEBUG cinder.utils [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:59:32 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.961(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:59:32 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:59:35 user cinder-volume[84696]: DEBUG cinder.utils [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.032(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:59:35 user cinder-volume[84696]: DEBUG cinder.utils [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:59:35 user cinder-volume[84696]: DEBUG cinder.utils [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.374(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:59:35 user cinder-volume[84696]: DEBUG cinder.utils [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_75163b78-31b2-4676-b93b-542a8ea784c1/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:59:35 user cinder-volume[84696]: DEBUG cinder.utils [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.367(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:59:35 user cinder-volume[84696]: DEBUG cinder.utils [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_60979eba-96b1-4cb3-bae1-8874a86cb411/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:59:45 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=16,cinder:UPDATE=5,cinder:INSERT=2 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:59:45 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:UPDATE=4,cinder:SELECT=11 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 15:59:45 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Volume reschedule parameters: True retry: {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 15:59:45 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Flow 'volume_create_manager' (e5bae52d-cd04-489c-b1af-1788f6c077fe) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:59:45 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (bcf67fd7-208b-4833-90ae-38b887220015) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:59:45 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (bcf67fd7-208b-4833-90ae-38b887220015) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:59:46Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesListAdminTestJSON-Volume-237491364',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=26014b4e-ec84-4011-a69f-ed42c3424a24,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='73222838c1d4451281d075e1f9c092b3',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:59:46Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:59:46Z,use_quota=True,user_id='6a809fbfe4264147a2716c1cc5914a28',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:59:45 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (0017c1da-bbee-4554-b557-920615d6357f) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:59:45 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (0017c1da-bbee-4554-b557-920615d6357f) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:59:45 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (e3ba53fc-7705-49ab-876c-a0a6c41eac57) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:59:45 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (e3ba53fc-7705-49ab-876c-a0a6c41eac57) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': '26014b4e-ec84-4011-a69f-ed42c3424a24', 'volume_name': 'volume-26014b4e-ec84-4011-a69f-ed42c3424a24', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:59:45 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (efdc759c-5d32-4b0c-870a-eb83a7367157) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:59:45 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (efdc759c-5d32-4b0c-870a-eb83a7367157) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:59:45 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (42fe441f-f4b2-4f07-bfd6-ab8c0d9aebac) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:59:45 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Volume 26014b4e-ec84-4011-a69f-ed42c3424a24: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-26014b4e-ec84-4011-a69f-ed42c3424a24', 'volume_size': 1} Apr 17 15:59:45 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] creating volume v_26014b4e-ec84-4011-a69f-ed42c3424a24. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 15:59:45 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] create volume {'name': 'v_26014b4e-ec84-4011-a69f-ed42c3424a24', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 15:59:45 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:59:45 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:59:48 user cinder-volume[84696]: DEBUG cinder.utils [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 3.018(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:59:48 user cinder-volume[84696]: DEBUG cinder.utils [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:59:48 user cinder-volume[84696]: DEBUG cinder.utils [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 40.426(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:59:48 user cinder-volume[84696]: DEBUG cinder.utils [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_b29a7592-128a-41ee-a8e2-57780dee33e7/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:59:48 user cinder-volume[84696]: DEBUG cinder.utils [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 40.055(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:59:48 user cinder-volume[84696]: DEBUG cinder.utils [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:59:51 user cinder-volume[84696]: DEBUG cinder.utils [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.090(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:59:51 user cinder-volume[84696]: DEBUG cinder.utils [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:59:54 user cinder-volume[84696]: DEBUG cinder.utils [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 40.381(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:59:54 user cinder-volume[84696]: DEBUG cinder.utils [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_75163b78-31b2-4676-b93b-542a8ea784c1/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:59:54 user cinder-volume[84696]: DEBUG cinder.utils [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 40.041(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:59:54 user cinder-volume[84696]: DEBUG cinder.utils [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:59:54 user cinder-volume[84696]: DEBUG cinder.utils [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 40.374(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:59:54 user cinder-volume[84696]: DEBUG cinder.utils [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_60979eba-96b1-4cb3-bae1-8874a86cb411/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:59:54 user cinder-volume[84696]: DEBUG cinder.utils [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 137.136(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:59:54 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:59:54 user cinder-volume[84696]: INFO cinder.message.api [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Creating message record for request_id = req-5a40883b-32f9-400a-a3f7-129f4d4677bb Apr 17 15:59:54 user cinder-volume[84696]: DEBUG cinder.utils [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 137.321(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:59:54 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_87514055-5a0f-459f-b10b-438d7a13d39d/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:59:54 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (53008c4f-ab86-4329-87aa-c158c3e48eeb) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 15:59:54 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (10fe12ae-19f3-41f4-a6f0-8a4a10b57b9e) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:59:54 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (10fe12ae-19f3-41f4-a6f0-8a4a10b57b9e) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:59:54 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Flow 'volume_create_manager' (3abee37b-a727-4bad-9026-53191fc8d5cc) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 15:59:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 15:59:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 15:59:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 15:59:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 15:59:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 15:59:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 15:59:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 15:59:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 15:59:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 15:59:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 15:59:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 15:59:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 15:59:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 15:59:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 15:59:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 15:59:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 15:59:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 15:59:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 15:59:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 15:59:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 15:59:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 15:59:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 15:59:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 15:59:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 15:59:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:59:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 15:59:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 15:59:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 15:59:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 15:59:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 15:59:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 15:59:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 15:59:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 15:59:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 15:59:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 15:59:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 15:59:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 15:59:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 15:59:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 15:59:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 15:59:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 15:59:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 15:59:55 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 15:59:55 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (1079e44a-eaf6-401d-adfc-6a2732d9bf6c) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 15:59:55 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 15:59:55 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:55:17Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-AttachVolumeNegativeTest-volume-1944200504',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=87514055-5a0f-459f-b10b-438d7a13d39d,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='8586cc10142d4743874f43cdb0f37aba',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:57:38Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:57:38Z,use_quota=True,user_id='9fdc7c80b4654692a52c8a4c4149324f',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 15:59:55 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:55:17Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-AttachVolumeNegativeTest-volume-1944200504',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=87514055-5a0f-459f-b10b-438d7a13d39d,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='8586cc10142d4743874f43cdb0f37aba',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:57:38Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:57:38Z,use_quota=True,user_id='9fdc7c80b4654692a52c8a4c4149324f',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(87514055-5a0f-459f-b10b-438d7a13d39d),volume_id=87514055-5a0f-459f-b10b-438d7a13d39d,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': '87514055-5a0f-459f-b10b-438d7a13d39d', 'volume_name': 'volume-87514055-5a0f-459f-b10b-438d7a13d39d', 'volume_size': 1}} Apr 17 15:59:55 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': '87514055-5a0f-459f-b10b-438d7a13d39d', 'volume': {'id': '87514055-5a0f-459f-b10b-438d7a13d39d', '_name_id': None, 'ec2_id': None, 'user_id': '9fdc7c80b4654692a52c8a4c4149324f', 'project_id': '8586cc10142d4743874f43cdb0f37aba', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-AttachVolumeNegativeTest-volume-1944200504', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'admin_metadata': {}, 'glance_metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T15:55:17.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-87514055-5a0f-459f-b10b-438d7a13d39d', 'name_id': '87514055-5a0f-459f-b10b-438d7a13d39d', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-AttachVolumeNegativeTest-volume-1944200504', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '8586cc10142d4743874f43cdb0f37aba', 'qos_specs': None, 'reservations': ['1c53c030-381a-4511-90a6-073111ef116f', '7bbd5e8b-d708-4476-a281-0ce37fa87819', '438a4611-10fd-4bd4-a60e-52564e4a0b82', 'a28101d3-8126-4720-b09f-75c7afd4c0dd'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': '9fdc7c80b4654692a52c8a4c4149324f', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-AttachVolumeNegativeTest-volume-1944200504', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '8586cc10142d4743874f43cdb0f37aba', 'qos_specs': None, 'reservations': ['1c53c030-381a-4511-90a6-073111ef116f', '7bbd5e8b-d708-4476-a281-0ce37fa87819', '438a4611-10fd-4bd4-a60e-52564e4a0b82', 'a28101d3-8126-4720-b09f-75c7afd4c0dd'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': '9fdc7c80b4654692a52c8a4c4149324f', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': '9fdc7c80b4654692a52c8a4c4149324f', 'metadata': {}, 'qos_specs': None, 'retry': {'num_attempts': 2, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']}}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(87514055-5a0f-459f-b10b-438d7a13d39d),volume_id=87514055-5a0f-459f-b10b-438d7a13d39d,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:55:17Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-AttachVolumeNegativeTest-volume-1944200504',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=87514055-5a0f-459f-b10b-438d7a13d39d,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='8586cc10142d4743874f43cdb0f37aba',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:57:38Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:57:38Z,use_quota=True,user_id='9fdc7c80b4654692a52c8a4c4149324f',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 15:59:55 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:55:17Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-AttachVolumeNegativeTest-volume-1944200504',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=87514055-5a0f-459f-b10b-438d7a13d39d,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='8586cc10142d4743874f43cdb0f37aba',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:57:38Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:57:38Z,use_quota=True,user_id='9fdc7c80b4654692a52c8a4c4149324f',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:55:17Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-AttachVolumeNegativeTest-volume-1944200504',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=87514055-5a0f-459f-b10b-438d7a13d39d,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='8586cc10142d4743874f43cdb0f37aba',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:57:38Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:57:38Z,use_quota=True,user_id='9fdc7c80b4654692a52c8a4c4149324f',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 15:59:55 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 15:59:55 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 15:59:55 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:59:55 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 15:59:55 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 15:59:55 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 15:59:55 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 15:59:55 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 15:59:55 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 15:59:55 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 15:59:55 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 15:59:55 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 15:59:55 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 15:59:55 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 15:59:55 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 15:59:55 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 15:59:55 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 15:59:55 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 15:59:55 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 15:59:55 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 15:59:55 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (1079e44a-eaf6-401d-adfc-6a2732d9bf6c) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:59:55 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (1079e44a-eaf6-401d-adfc-6a2732d9bf6c) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:59:55 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (47f88e3e-d4e7-46dc-8d02-0198752e12f5) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:59:55 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (47f88e3e-d4e7-46dc-8d02-0198752e12f5) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:59:55 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (1fb0c416-a2fa-4f26-aa3f-5597a03192af) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:59:55 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (1fb0c416-a2fa-4f26-aa3f-5597a03192af) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:59:55 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (9db71fee-70ac-4ddc-bef2-8b370efd6663) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:59:55 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Updating volume 87514055-5a0f-459f-b10b-438d7a13d39d with {'scheduled_at': datetime.datetime(2023, 4, 17, 15, 59, 55, 57150), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 15:59:55 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Volume 87514055-5a0f-459f-b10b-438d7a13d39d: re-scheduling SchedulerAPI.create_volume attempt 2 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 15:59:55 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Volume 87514055-5a0f-459f-b10b-438d7a13d39d: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 15:59:55 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] deleating volume v_87514055-5a0f-459f-b10b-438d7a13d39d {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 15:59:55 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] get snapshots for volume v_87514055-5a0f-459f-b10b-438d7a13d39d {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 15:59:55 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_87514055-5a0f-459f-b10b-438d7a13d39d/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:59:55 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_87514055-5a0f-459f-b10b-438d7a13d39d/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:59:55 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Volume reschedule parameters: True retry: {'num_attempts': 3, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 15:59:55 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Flow 'volume_create_manager' (69a56206-5eb3-46ac-aec1-c0693642611c) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:59:55 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (afae940a-47b8-4175-a547-e2ad29901639) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:59:55 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (afae940a-47b8-4175-a547-e2ad29901639) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:55:17Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-AttachVolumeNegativeTest-volume-1944200504',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=87514055-5a0f-459f-b10b-438d7a13d39d,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='8586cc10142d4743874f43cdb0f37aba',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:59:55Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:59:55Z,use_quota=True,user_id='9fdc7c80b4654692a52c8a4c4149324f',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:59:55 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (43048501-f203-43eb-8c24-422c165f661a) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:59:55 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (43048501-f203-43eb-8c24-422c165f661a) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:59:55 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (eccf2cdf-735f-48be-901e-f224f342f7c5) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:59:55 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (eccf2cdf-735f-48be-901e-f224f342f7c5) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': '87514055-5a0f-459f-b10b-438d7a13d39d', 'volume_name': 'volume-87514055-5a0f-459f-b10b-438d7a13d39d', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:59:55 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (7db4611c-4af8-4a0e-84d0-58c0123e236c) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:59:55 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (7db4611c-4af8-4a0e-84d0-58c0123e236c) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:59:55 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (ba0a9923-7993-4363-a092-9649d52e75ec) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:59:55 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Volume 87514055-5a0f-459f-b10b-438d7a13d39d: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-87514055-5a0f-459f-b10b-438d7a13d39d', 'volume_size': 1} Apr 17 15:59:55 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] creating volume v_87514055-5a0f-459f-b10b-438d7a13d39d. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 15:59:55 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] create volume {'name': 'v_87514055-5a0f-459f-b10b-438d7a13d39d', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 15:59:55 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:59:55 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:59:58 user cinder-volume[84696]: DEBUG cinder.utils [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.960(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:59:58 user cinder-volume[84696]: DEBUG cinder.utils [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_87514055-5a0f-459f-b10b-438d7a13d39d/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:59:58 user cinder-volume[84696]: DEBUG cinder.utils [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.792(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:59:58 user cinder-volume[84696]: DEBUG cinder.utils [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:59:58 user cinder-volume[84696]: DEBUG cinder.utils [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 137.336(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:59:58 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_14aa38ea-7cbb-48f7-a9aa-ffcc7b66faa0/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:59:58 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (a72c6b3f-14ea-4ea4-bded-4d9494ce46f3) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 15:59:58 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (39d93b4d-287d-4baa-ba09-b31adfd31748) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:59:58 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (39d93b4d-287d-4baa-ba09-b31adfd31748) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:59:58 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Flow 'volume_create_manager' (d584aec2-1bcc-4af1-ab18-d1d5c82b4806) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 15:59:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 15:59:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 15:59:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 15:59:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 15:59:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 15:59:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 15:59:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 15:59:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 15:59:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 15:59:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 15:59:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 15:59:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 15:59:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 15:59:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 15:59:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 15:59:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 15:59:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 15:59:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 15:59:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 15:59:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 15:59:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 15:59:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 15:59:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 15:59:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 15:59:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:59:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 15:59:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1237, in execute Apr 17 15:59:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_from_image(context, Apr 17 15:59:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/utils.py", line 670, in _wrapper Apr 17 15:59:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return r(f, *args, **kwargs) Apr 17 15:59:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/tenacity/__init__.py", line 423, in __call__ Apr 17 15:59:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server do = self.iter(retry_state=retry_state) Apr 17 15:59:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/tenacity/__init__.py", line 360, in iter Apr 17 15:59:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return fut.result() Apr 17 15:59:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/lib/python3.10/concurrent/futures/_base.py", line 451, in result Apr 17 15:59:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self.__get_result() Apr 17 15:59:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result Apr 17 15:59:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self._exception Apr 17 15:59:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/tenacity/__init__.py", line 426, in __call__ Apr 17 15:59:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = fn(*args, **kwargs) Apr 17 15:59:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1125, in _create_from_image Apr 17 15:59:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_from_image_cache_or_download( Apr 17 15:59:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 944, in _create_from_image_cache_or_download Apr 17 15:59:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update, cloned = self._create_from_image_cache( Apr 17 15:59:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 843, in _create_from_image_cache Apr 17 15:59:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_from_source_volume( Apr 17 15:59:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 659, in _create_from_source_volume Apr 17 15:59:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self.driver.create_cloned_volume(volume, srcvol_ref) Apr 17 15:59:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 525, in create_cloned_volume Apr 17 15:59:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self._clone_object(vname, cvname) Apr 17 15:59:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 461, in _clone_object Apr 17 15:59:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_snapshot(oname, coname) Apr 17 15:59:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 639, in create_snapshot Apr 17 15:59:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201, 204): Apr 17 15:59:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 15:59:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 15:59:58 user cinder-volume[84696]: DEBUG cinder.utils [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.259(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:59:58 user cinder-volume[84696]: DEBUG cinder.utils [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 15:59:58 user cinder-volume[84696]: DEBUG cinder.utils [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 137.034(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 15:59:58 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 15:59:58 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (6293c97f-7299-43fd-99bf-ee8e029e0520) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 15:59:58 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 15:59:58 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:55:22Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-tempest.common.compute-volume-826817831',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=14aa38ea-7cbb-48f7-a9aa-ffcc7b66faa0,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='cb5f6b5ed2fc43b69c47e1fc18646545',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:57:41Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:57:41Z,use_quota=True,user_id='638a4f535e9740fc9d810639a5eb24d7',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 15:59:58 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:55:22Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-tempest.common.compute-volume-826817831',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=14aa38ea-7cbb-48f7-a9aa-ffcc7b66faa0,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='cb5f6b5ed2fc43b69c47e1fc18646545',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:57:41Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:57:41Z,use_quota=True,user_id='638a4f535e9740fc9d810639a5eb24d7',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=b3f0dec1-7758-44a8-82e0-76ada8837428,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(14aa38ea-7cbb-48f7-a9aa-ffcc7b66faa0),volume_id=14aa38ea-7cbb-48f7-a9aa-ffcc7b66faa0,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'image', 'volume_id': '14aa38ea-7cbb-48f7-a9aa-ffcc7b66faa0', 'volume_name': 'volume-14aa38ea-7cbb-48f7-a9aa-ffcc7b66faa0', 'volume_size': 1, 'image_id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'image_location': (None, None), 'image_meta': {'name': 'cirros-0.5.2-x86_64-disk', 'disk_format': 'qcow2', 'container_format': 'bare', 'visibility': 'public', 'size': 16300544, 'virtual_size': 117440512, 'status': 'active', 'checksum': 'b874c39491a2377b8490f5f1e89761a4', 'protected': False, 'min_ram': 0, 'min_disk': 0, 'owner': 'a43c211b4ed542538799349489b1d884', 'os_hidden': False, 'os_hash_algo': 'sha512', 'os_hash_value': '6b813aa46bb90b4da216a4d19376593fa3f4fc7e617f03a92b7fe11e9a3981cbe8f0959dbebe36225e5f53dc4492341a4863cac4ed1ee0909f3fc78ef9c3e869', 'id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'created_at': datetime.datetime(2023, 4, 17, 15, 19, 53, tzinfo=datetime.timezone.utc), 'updated_at': datetime.datetime(2023, 4, 17, 15, 19, 55, tzinfo=datetime.timezone.utc), 'tags': [], 'file': '/v2/images/b3f0dec1-7758-44a8-82e0-76ada8837428/file', 'properties': {'hw_rng_model': 'virtio', 'owner_specified.openstack.md5': '', 'owner_specified.openstack.object': 'images/cirros-0.5.2-x86_64-disk', 'owner_specified.openstack.sha256': ''}}, 'image_service': }} Apr 17 15:59:58 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'snapshot_id': None, 'source_volid': None, 'volume_id': '14aa38ea-7cbb-48f7-a9aa-ffcc7b66faa0', 'volume': {'id': '14aa38ea-7cbb-48f7-a9aa-ffcc7b66faa0', '_name_id': None, 'ec2_id': None, 'user_id': '638a4f535e9740fc9d810639a5eb24d7', 'project_id': 'cb5f6b5ed2fc43b69c47e1fc18646545', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-tempest.common.compute-volume-826817831', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'admin_metadata': {}, 'glance_metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T15:55:22.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-14aa38ea-7cbb-48f7-a9aa-ffcc7b66faa0', 'name_id': '14aa38ea-7cbb-48f7-a9aa-ffcc7b66faa0', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-tempest.common.compute-volume-826817831', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': 'cb5f6b5ed2fc43b69c47e1fc18646545', 'qos_specs': None, 'reservations': ['c3b21a3e-ae73-41f9-8d32-500bd32a5696', '477b1752-2fec-4e18-835a-e7ca57617158', '2e94e8e6-37cc-46c3-b87c-ae7717df65c2', 'a5c6c20e-c7ee-491f-b333-35f64711994a'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': '638a4f535e9740fc9d810639a5eb24d7', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-tempest.common.compute-volume-826817831', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': 'cb5f6b5ed2fc43b69c47e1fc18646545', 'qos_specs': None, 'reservations': ['c3b21a3e-ae73-41f9-8d32-500bd32a5696', '477b1752-2fec-4e18-835a-e7ca57617158', '2e94e8e6-37cc-46c3-b87c-ae7717df65c2', 'a5c6c20e-c7ee-491f-b333-35f64711994a'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': '638a4f535e9740fc9d810639a5eb24d7', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': '638a4f535e9740fc9d810639a5eb24d7', 'metadata': {}, 'qos_specs': None, 'retry': {'num_attempts': 2, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']}}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=b3f0dec1-7758-44a8-82e0-76ada8837428,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(14aa38ea-7cbb-48f7-a9aa-ffcc7b66faa0),volume_id=14aa38ea-7cbb-48f7-a9aa-ffcc7b66faa0,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:55:22Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-tempest.common.compute-volume-826817831',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=14aa38ea-7cbb-48f7-a9aa-ffcc7b66faa0,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='cb5f6b5ed2fc43b69c47e1fc18646545',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:57:41Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:57:41Z,use_quota=True,user_id='638a4f535e9740fc9d810639a5eb24d7',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 15:59:58 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:55:22Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-tempest.common.compute-volume-826817831',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=14aa38ea-7cbb-48f7-a9aa-ffcc7b66faa0,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='cb5f6b5ed2fc43b69c47e1fc18646545',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:57:41Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:57:41Z,use_quota=True,user_id='638a4f535e9740fc9d810639a5eb24d7',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:55:22Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-tempest.common.compute-volume-826817831',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=14aa38ea-7cbb-48f7-a9aa-ffcc7b66faa0,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='cb5f6b5ed2fc43b69c47e1fc18646545',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:57:41Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:57:41Z,use_quota=True,user_id='638a4f535e9740fc9d810639a5eb24d7',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 15:59:58 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 15:59:58 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 15:59:58 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 15:59:58 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 15:59:58 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1237, in execute Apr 17 15:59:58 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_from_image(context, Apr 17 15:59:58 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/utils.py", line 670, in _wrapper Apr 17 15:59:58 user cinder-volume[84696]: ERROR cinder.volume.manager return r(f, *args, **kwargs) Apr 17 15:59:58 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/tenacity/__init__.py", line 423, in __call__ Apr 17 15:59:58 user cinder-volume[84696]: ERROR cinder.volume.manager do = self.iter(retry_state=retry_state) Apr 17 15:59:58 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/tenacity/__init__.py", line 360, in iter Apr 17 15:59:58 user cinder-volume[84696]: ERROR cinder.volume.manager return fut.result() Apr 17 15:59:58 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/lib/python3.10/concurrent/futures/_base.py", line 451, in result Apr 17 15:59:58 user cinder-volume[84696]: ERROR cinder.volume.manager return self.__get_result() Apr 17 15:59:58 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result Apr 17 15:59:58 user cinder-volume[84696]: ERROR cinder.volume.manager raise self._exception Apr 17 15:59:58 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/tenacity/__init__.py", line 426, in __call__ Apr 17 15:59:58 user cinder-volume[84696]: ERROR cinder.volume.manager result = fn(*args, **kwargs) Apr 17 15:59:58 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1125, in _create_from_image Apr 17 15:59:58 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_from_image_cache_or_download( Apr 17 15:59:58 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 944, in _create_from_image_cache_or_download Apr 17 15:59:58 user cinder-volume[84696]: ERROR cinder.volume.manager model_update, cloned = self._create_from_image_cache( Apr 17 15:59:58 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 843, in _create_from_image_cache Apr 17 15:59:58 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_from_source_volume( Apr 17 15:59:58 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 659, in _create_from_source_volume Apr 17 15:59:58 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self.driver.create_cloned_volume(volume, srcvol_ref) Apr 17 15:59:58 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 525, in create_cloned_volume Apr 17 15:59:58 user cinder-volume[84696]: ERROR cinder.volume.manager self._clone_object(vname, cvname) Apr 17 15:59:58 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 461, in _clone_object Apr 17 15:59:58 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_snapshot(oname, coname) Apr 17 15:59:58 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 639, in create_snapshot Apr 17 15:59:58 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201, 204): Apr 17 15:59:58 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 15:59:58 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 15:59:58 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (6293c97f-7299-43fd-99bf-ee8e029e0520) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:59:58 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (6293c97f-7299-43fd-99bf-ee8e029e0520) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:59:58 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (7169e5d6-8d5d-4c85-b533-c2de1fe81112) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:59:58 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (7169e5d6-8d5d-4c85-b533-c2de1fe81112) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:59:58 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (7f53bd87-cac2-4a04-b3b9-6350e800247b) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:59:58 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (7f53bd87-cac2-4a04-b3b9-6350e800247b) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 15:59:58 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (7ad071b2-5c5a-4e67-a1e3-1c6ca780c749) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:59:58 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Updating volume 14aa38ea-7cbb-48f7-a9aa-ffcc7b66faa0 with {'scheduled_at': datetime.datetime(2023, 4, 17, 15, 59, 58, 114086), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 15:59:58 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Volume 14aa38ea-7cbb-48f7-a9aa-ffcc7b66faa0: re-scheduling SchedulerAPI.create_volume attempt 2 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 15:59:58 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Volume 14aa38ea-7cbb-48f7-a9aa-ffcc7b66faa0: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 15:59:58 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] deleating volume v_14aa38ea-7cbb-48f7-a9aa-ffcc7b66faa0 {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 15:59:58 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] get snapshots for volume v_14aa38ea-7cbb-48f7-a9aa-ffcc7b66faa0 {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 15:59:58 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_14aa38ea-7cbb-48f7-a9aa-ffcc7b66faa0/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:59:58 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_14aa38ea-7cbb-48f7-a9aa-ffcc7b66faa0/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 15:59:58 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Volume reschedule parameters: True retry: {'num_attempts': 3, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 15:59:58 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Flow 'volume_create_manager' (2f00e9a7-d1a6-4c53-923c-c7a818782e35) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 15:59:58 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (cb21c6b9-8d0f-4c51-a6ff-25ffcc1a0545) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:59:58 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (cb21c6b9-8d0f-4c51-a6ff-25ffcc1a0545) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:55:22Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-tempest.common.compute-volume-826817831',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=14aa38ea-7cbb-48f7-a9aa-ffcc7b66faa0,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='cb5f6b5ed2fc43b69c47e1fc18646545',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:59:58Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:59:58Z,use_quota=True,user_id='638a4f535e9740fc9d810639a5eb24d7',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:59:58 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (5faf14b4-e033-41b8-84fb-8b4d162dd650) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:59:58 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (5faf14b4-e033-41b8-84fb-8b4d162dd650) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:59:58 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (1cbc4a98-22d3-4eab-9959-b94cc8182263) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:59:58 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (1cbc4a98-22d3-4eab-9959-b94cc8182263) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'image', 'volume_id': '14aa38ea-7cbb-48f7-a9aa-ffcc7b66faa0', 'volume_name': 'volume-14aa38ea-7cbb-48f7-a9aa-ffcc7b66faa0', 'volume_size': 1, 'image_id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'image_location': (None, None), 'image_meta': {'name': 'cirros-0.5.2-x86_64-disk', 'disk_format': 'qcow2', 'container_format': 'bare', 'visibility': 'public', 'size': 16300544, 'virtual_size': 117440512, 'status': 'active', 'checksum': 'b874c39491a2377b8490f5f1e89761a4', 'protected': False, 'min_ram': 0, 'min_disk': 0, 'owner': 'a43c211b4ed542538799349489b1d884', 'os_hidden': False, 'os_hash_algo': 'sha512', 'os_hash_value': '6b813aa46bb90b4da216a4d19376593fa3f4fc7e617f03a92b7fe11e9a3981cbe8f0959dbebe36225e5f53dc4492341a4863cac4ed1ee0909f3fc78ef9c3e869', 'id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'created_at': datetime.datetime(2023, 4, 17, 15, 19, 53, tzinfo=datetime.timezone.utc), 'updated_at': datetime.datetime(2023, 4, 17, 15, 19, 55, tzinfo=datetime.timezone.utc), 'tags': [], 'file': '/v2/images/b3f0dec1-7758-44a8-82e0-76ada8837428/file', 'properties': {'hw_rng_model': 'virtio', 'owner_specified.openstack.md5': '', 'owner_specified.openstack.object': 'images/cirros-0.5.2-x86_64-disk', 'owner_specified.openstack.sha256': ''}}, 'image_service': }' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:59:58 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (cd0c89a4-d7ed-438c-99d0-156bd9bd99bd) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:59:58 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (cd0c89a4-d7ed-438c-99d0-156bd9bd99bd) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 15:59:58 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (ff78fa8d-6ac1-4704-8c0c-3b932fe191d3) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 15:59:58 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Volume 14aa38ea-7cbb-48f7-a9aa-ffcc7b66faa0: being created as image with specification: {'status': 'creating', 'volume_name': 'volume-14aa38ea-7cbb-48f7-a9aa-ffcc7b66faa0', 'volume_size': 1, 'image_id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'image_location': (None, None), 'image_meta': {'name': 'cirros-0.5.2-x86_64-disk', 'disk_format': 'qcow2', 'container_format': 'bare', 'visibility': 'public', 'size': 16300544, 'virtual_size': 117440512, 'status': 'active', 'checksum': 'b874c39491a2377b8490f5f1e89761a4', 'protected': False, 'min_ram': 0, 'min_disk': 0, 'owner': 'a43c211b4ed542538799349489b1d884', 'os_hidden': False, 'os_hash_algo': 'sha512', 'os_hash_value': '6b813aa46bb90b4da216a4d19376593fa3f4fc7e617f03a92b7fe11e9a3981cbe8f0959dbebe36225e5f53dc4492341a4863cac4ed1ee0909f3fc78ef9c3e869', 'id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'created_at': datetime.datetime(2023, 4, 17, 15, 19, 53, tzinfo=datetime.timezone.utc), 'updated_at': datetime.datetime(2023, 4, 17, 15, 19, 55, tzinfo=datetime.timezone.utc), 'tags': [], 'file': '/v2/images/b3f0dec1-7758-44a8-82e0-76ada8837428/file', 'properties': {'hw_rng_model': 'virtio', 'owner_specified.openstack.md5': '', 'owner_specified.openstack.object': 'images/cirros-0.5.2-x86_64-disk', 'owner_specified.openstack.sha256': ''}}, 'image_service': } Apr 17 15:59:58 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Cloning 14aa38ea-7cbb-48f7-a9aa-ffcc7b66faa0 from image b3f0dec1-7758-44a8-82e0-76ada8837428 at location (None, None). {{(pid=84696) _create_from_image /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1071}} Apr 17 15:59:58 user cinder-volume[84696]: DEBUG cinder.coordination [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Acquiring lock "cinder-b3f0dec1-7758-44a8-82e0-76ada8837428" by "_prepare_image_cache_entry" {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:192}} Apr 17 15:59:58 user cinder-volume[84696]: DEBUG cinder.coordination [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Lock "cinder-b3f0dec1-7758-44a8-82e0-76ada8837428" acquired by "_prepare_image_cache_entry" :: waited 0.011s {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:196}} Apr 17 15:59:58 user cinder-volume[84696]: DEBUG cinder.image.cache [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Found image-volume cache entry: {'id': 1, 'image_id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'volume_id': '006f20ae-5306-44fe-afdf-e433f79284e8', 'host': 'user@joviandss-2#joviandss-2', 'size': 1, 'image_updated_at': datetime.datetime(2023, 4, 17, 15, 19, 55), 'last_used': datetime.datetime(2023, 4, 17, 15, 59, 58, 470398)}. {{(pid=84696) get_entry /opt/stack/cinder/cinder/image/cache.py:76}} Apr 17 15:59:58 user cinder-volume[84696]: DEBUG cinder.image.cache [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Image-volume cache entry image_update_at = 2023-04-17 15:19:55+00:00, requested image updated_at = 2023-04-17 15:19:55+00:00. {{(pid=84696) _should_update_entry /opt/stack/cinder/cinder/image/cache.py:259}} Apr 17 15:59:58 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Found cache entry for image = b3f0dec1-7758-44a8-82e0-76ada8837428 on host user@joviandss-2#joviandss-2. {{(pid=84696) _prepare_image_cache_entry /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:886}} Apr 17 15:59:58 user cinder-volume[84696]: DEBUG cinder.coordination [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Lock "cinder-b3f0dec1-7758-44a8-82e0-76ada8837428" released by "_prepare_image_cache_entry" :: held 0.014s {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:207}} Apr 17 15:59:58 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Attempting to retrieve cache entry for image = b3f0dec1-7758-44a8-82e0-76ada8837428 on host user@joviandss-2#joviandss-2. {{(pid=84696) _create_from_image_cache /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:827}} Apr 17 15:59:58 user cinder-volume[84696]: DEBUG cinder.image.cache [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Found image-volume cache entry: {'id': 1, 'image_id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'volume_id': '006f20ae-5306-44fe-afdf-e433f79284e8', 'host': 'user@joviandss-2#joviandss-2', 'size': 1, 'image_updated_at': datetime.datetime(2023, 4, 17, 15, 19, 55), 'last_used': datetime.datetime(2023, 4, 17, 15, 59, 58, 483041)}. {{(pid=84696) get_entry /opt/stack/cinder/cinder/image/cache.py:76}} Apr 17 15:59:58 user cinder-volume[84696]: DEBUG cinder.image.cache [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Image-volume cache entry image_update_at = 2023-04-17 15:19:55+00:00, requested image updated_at = 2023-04-17 15:19:55+00:00. {{(pid=84696) _should_update_entry /opt/stack/cinder/cinder/image/cache.py:259}} Apr 17 15:59:58 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Creating from source image-volume 006f20ae-5306-44fe-afdf-e433f79284e8 {{(pid=84696) _create_from_image_cache /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:841}} Apr 17 15:59:58 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] cloned volume 006f20ae-5306-44fe-afdf-e433f79284e8 to 14aa38ea-7cbb-48f7-a9aa-ffcc7b66faa0 {{(pid=84696) create_cloned_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:521}} Apr 17 15:59:58 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] cloning v_006f20ae-5306-44fe-afdf-e433f79284e8 to v_14aa38ea-7cbb-48f7-a9aa-ffcc7b66faa0 {{(pid=84696) _clone_object /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:456}} Apr 17 15:59:58 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] create snapshot v_14aa38ea-7cbb-48f7-a9aa-ffcc7b66faa0 {{(pid=84696) create_snapshot /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:635}} Apr 17 15:59:58 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 15:59:58 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:00:01 user cinder-volume[84696]: DEBUG cinder.utils [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 137.287(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:00:01 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_45e5cc09-6c97-438a-a6a2-c89e7d725ce5/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:00:01 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (cfe42e67-3aa3-4197-8aa9-61ca5abb7fb3) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 16:00:01 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (a276e14b-58c4-46c3-a804-e11638a58ecc) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:00:01 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (a276e14b-58c4-46c3-a804-e11638a58ecc) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:00:01 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Flow 'volume_create_manager' (cb5e75a4-dba2-45e9-85bb-081cc2ee2604) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 16:00:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 16:00:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 16:00:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 16:00:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 16:00:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 16:00:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 16:00:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 16:00:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 16:00:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 16:00:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 16:00:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 16:00:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 16:00:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 16:00:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 16:00:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 16:00:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 16:00:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 16:00:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 16:00:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 16:00:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 16:00:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 16:00:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 16:00:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 16:00:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 16:00:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:00:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 16:00:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:00:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 16:00:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:00:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 16:00:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:00:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 16:00:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:00:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 16:00:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:00:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 16:00:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:00:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 16:00:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:00:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:00:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 16:00:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 16:00:01 user cinder-volume[84696]: DEBUG cinder.utils [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 74.082(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:00:01 user cinder-volume[84696]: DEBUG cinder.utils [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:00:01 user cinder-volume[84696]: DEBUG cinder.utils [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 74.255(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:00:01 user cinder-volume[84696]: DEBUG cinder.utils [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_bff6fdbb-a1fb-4b48-bca8-2f7efbc3f0ff/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:00:01 user cinder-volume[84696]: DEBUG cinder.utils [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.643(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:00:01 user cinder-volume[84696]: DEBUG cinder.utils [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:00:01 user cinder-volume[84696]: DEBUG cinder.utils [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.985(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:00:01 user cinder-volume[84696]: DEBUG cinder.utils [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_14aa38ea-7cbb-48f7-a9aa-ffcc7b66faa0/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:00:01 user cinder-volume[84696]: DEBUG cinder.utils [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.055(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:00:01 user cinder-volume[84696]: DEBUG cinder.utils [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_87514055-5a0f-459f-b10b-438d7a13d39d/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:00:01 user cinder-volume[84696]: DEBUG cinder.utils [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.886(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:00:01 user cinder-volume[84696]: DEBUG cinder.utils [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:00:01 user cinder-volume[84696]: DEBUG cinder.utils [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 137.140(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:00:01 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:00:01 user cinder-volume[84696]: INFO cinder.message.api [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Creating message record for request_id = req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 Apr 17 16:00:01 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 16:00:01 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (691f22bc-e2c1-43bc-b87e-457f041d6b10) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 16:00:01 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 16:00:01 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:53:03Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TestMinimumBasicScenario-volume-1479398066',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=45e5cc09-6c97-438a-a6a2-c89e7d725ce5,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='17d1dae0ad7149bbbd160525d64c0c20',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:57:44Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:57:44Z,use_quota=True,user_id='3c407d9d494d485c9c95dc3981284fba',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 16:00:01 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:53:03Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TestMinimumBasicScenario-volume-1479398066',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=45e5cc09-6c97-438a-a6a2-c89e7d725ce5,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='17d1dae0ad7149bbbd160525d64c0c20',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:57:44Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:57:44Z,use_quota=True,user_id='3c407d9d494d485c9c95dc3981284fba',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(45e5cc09-6c97-438a-a6a2-c89e7d725ce5),volume_id=45e5cc09-6c97-438a-a6a2-c89e7d725ce5,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': '45e5cc09-6c97-438a-a6a2-c89e7d725ce5', 'volume_name': 'volume-45e5cc09-6c97-438a-a6a2-c89e7d725ce5', 'volume_size': 1}} Apr 17 16:00:01 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': '45e5cc09-6c97-438a-a6a2-c89e7d725ce5', 'volume': {'id': '45e5cc09-6c97-438a-a6a2-c89e7d725ce5', '_name_id': None, 'ec2_id': None, 'user_id': '3c407d9d494d485c9c95dc3981284fba', 'project_id': '17d1dae0ad7149bbbd160525d64c0c20', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-TestMinimumBasicScenario-volume-1479398066', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'admin_metadata': {}, 'glance_metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T15:53:02.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-45e5cc09-6c97-438a-a6a2-c89e7d725ce5', 'name_id': '45e5cc09-6c97-438a-a6a2-c89e7d725ce5', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-TestMinimumBasicScenario-volume-1479398066', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '17d1dae0ad7149bbbd160525d64c0c20', 'qos_specs': None, 'reservations': ['a105b386-4e4e-4dca-ae87-4d1b3715ef82', '1ede682d-d17c-4d21-8d44-1ad1c081125a', 'df0aa6c2-bf1a-436e-9684-88c24c928b51', '6d77c826-2ba0-4ea1-b791-09b8cf39c978'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': '3c407d9d494d485c9c95dc3981284fba', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-TestMinimumBasicScenario-volume-1479398066', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '17d1dae0ad7149bbbd160525d64c0c20', 'qos_specs': None, 'reservations': ['a105b386-4e4e-4dca-ae87-4d1b3715ef82', '1ede682d-d17c-4d21-8d44-1ad1c081125a', 'df0aa6c2-bf1a-436e-9684-88c24c928b51', '6d77c826-2ba0-4ea1-b791-09b8cf39c978'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': '3c407d9d494d485c9c95dc3981284fba', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': '3c407d9d494d485c9c95dc3981284fba', 'metadata': {}, 'qos_specs': None, 'retry': {'num_attempts': 3, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']}}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(45e5cc09-6c97-438a-a6a2-c89e7d725ce5),volume_id=45e5cc09-6c97-438a-a6a2-c89e7d725ce5,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:53:03Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TestMinimumBasicScenario-volume-1479398066',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=45e5cc09-6c97-438a-a6a2-c89e7d725ce5,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='17d1dae0ad7149bbbd160525d64c0c20',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:57:44Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:57:44Z,use_quota=True,user_id='3c407d9d494d485c9c95dc3981284fba',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 16:00:01 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:53:03Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TestMinimumBasicScenario-volume-1479398066',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=45e5cc09-6c97-438a-a6a2-c89e7d725ce5,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='17d1dae0ad7149bbbd160525d64c0c20',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:57:44Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:57:44Z,use_quota=True,user_id='3c407d9d494d485c9c95dc3981284fba',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:53:03Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TestMinimumBasicScenario-volume-1479398066',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=45e5cc09-6c97-438a-a6a2-c89e7d725ce5,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='17d1dae0ad7149bbbd160525d64c0c20',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:57:44Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:57:44Z,use_quota=True,user_id='3c407d9d494d485c9c95dc3981284fba',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 16:00:01 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 16:00:01 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 16:00:01 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:00:01 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 16:00:01 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:00:01 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 16:00:01 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:00:01 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 16:00:01 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:00:01 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 16:00:01 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:00:01 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 16:00:01 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:00:01 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 16:00:01 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:00:01 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 16:00:01 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:00:01 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:00:01 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 16:00:01 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 16:00:01 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (691f22bc-e2c1-43bc-b87e-457f041d6b10) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:00:01 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (691f22bc-e2c1-43bc-b87e-457f041d6b10) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:00:01 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (0a8f26a6-0581-42bd-8d58-5e224ca90391) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:00:01 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (0a8f26a6-0581-42bd-8d58-5e224ca90391) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:00:01 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (442b4e40-5a82-49b5-84d7-3ca093921162) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:00:01 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (442b4e40-5a82-49b5-84d7-3ca093921162) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:00:01 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (458dc8a3-8d60-43b2-a4a5-5d529293df10) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:00:01 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Updating volume 45e5cc09-6c97-438a-a6a2-c89e7d725ce5 with {'scheduled_at': datetime.datetime(2023, 4, 17, 16, 0, 1, 213448), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 16:00:01 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Volume 45e5cc09-6c97-438a-a6a2-c89e7d725ce5: re-scheduling SchedulerAPI.create_volume attempt 3 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 16:00:01 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Volume 45e5cc09-6c97-438a-a6a2-c89e7d725ce5: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 16:00:01 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] deleating volume v_45e5cc09-6c97-438a-a6a2-c89e7d725ce5 {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 16:00:01 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] get snapshots for volume v_45e5cc09-6c97-438a-a6a2-c89e7d725ce5 {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 16:00:01 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_45e5cc09-6c97-438a-a6a2-c89e7d725ce5/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:00:01 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_45e5cc09-6c97-438a-a6a2-c89e7d725ce5/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:00:04 user cinder-volume[84696]: DEBUG cinder.utils [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.945(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:00:04 user cinder-volume[84696]: DEBUG cinder.utils [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_45e5cc09-6c97-438a-a6a2-c89e7d725ce5/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:00:04 user cinder-volume[84696]: DEBUG cinder.utils [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 136.906(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:00:04 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:00:04 user cinder-volume[84696]: INFO cinder.message.api [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Creating message record for request_id = req-dc8764a6-903f-44ac-86a4-76a8d1914785 Apr 17 16:00:04 user cinder-volume[84696]: DEBUG cinder.utils [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.036(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:00:04 user cinder-volume[84696]: DEBUG cinder.utils [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_14aa38ea-7cbb-48f7-a9aa-ffcc7b66faa0/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:00:04 user cinder-volume[84696]: DEBUG cinder.utils [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.696(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:00:04 user cinder-volume[84696]: DEBUG cinder.utils [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:00:04 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 16:00:04 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (fa6c819a-c3c4-4f92-97d3-d0278e26aca5) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 16:00:04 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 16:00:04 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:57:47Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumeRevertTests-Volume-1536424582',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=9a399a62-6a8f-4b6d-8ecf-8dcdf57e5fbb,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='9c755d3b5d8a4d9c9918b6c4744f510e',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:57:47Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:57:47Z,use_quota=True,user_id='e362cfe4a31c444b9ab44a5d60f37ad9',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 16:00:04 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:57:47Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumeRevertTests-Volume-1536424582',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=9a399a62-6a8f-4b6d-8ecf-8dcdf57e5fbb,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='9c755d3b5d8a4d9c9918b6c4744f510e',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:57:47Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:57:47Z,use_quota=True,user_id='e362cfe4a31c444b9ab44a5d60f37ad9',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(9a399a62-6a8f-4b6d-8ecf-8dcdf57e5fbb),volume_id=9a399a62-6a8f-4b6d-8ecf-8dcdf57e5fbb,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': '9a399a62-6a8f-4b6d-8ecf-8dcdf57e5fbb', 'volume_name': 'volume-9a399a62-6a8f-4b6d-8ecf-8dcdf57e5fbb', 'volume_size': 1}} Apr 17 16:00:04 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'retry': {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']}, 'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': '9a399a62-6a8f-4b6d-8ecf-8dcdf57e5fbb', 'volume': {'id': '9a399a62-6a8f-4b6d-8ecf-8dcdf57e5fbb', '_name_id': None, 'ec2_id': None, 'user_id': 'e362cfe4a31c444b9ab44a5d60f37ad9', 'project_id': '9c755d3b5d8a4d9c9918b6c4744f510e', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-VolumeRevertTests-Volume-1536424582', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T15:57:47.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-9a399a62-6a8f-4b6d-8ecf-8dcdf57e5fbb', 'name_id': '9a399a62-6a8f-4b6d-8ecf-8dcdf57e5fbb', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumeRevertTests-Volume-1536424582', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '9c755d3b5d8a4d9c9918b6c4744f510e', 'qos_specs': None, 'reservations': ['50c841a2-120d-4b16-8e0c-160f7b70632b', '75012498-38d8-444b-aeb0-ece3429535fa', '3472950b-da65-4163-bf5e-5efec626310b', 'b7635850-6316-4d2c-8d29-9d8061c89d14'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'e362cfe4a31c444b9ab44a5d60f37ad9', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumeRevertTests-Volume-1536424582', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '9c755d3b5d8a4d9c9918b6c4744f510e', 'qos_specs': None, 'reservations': ['50c841a2-120d-4b16-8e0c-160f7b70632b', '75012498-38d8-444b-aeb0-ece3429535fa', '3472950b-da65-4163-bf5e-5efec626310b', 'b7635850-6316-4d2c-8d29-9d8061c89d14'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'e362cfe4a31c444b9ab44a5d60f37ad9', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': 'e362cfe4a31c444b9ab44a5d60f37ad9', 'metadata': {}, 'qos_specs': None}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(9a399a62-6a8f-4b6d-8ecf-8dcdf57e5fbb),volume_id=9a399a62-6a8f-4b6d-8ecf-8dcdf57e5fbb,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:57:47Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumeRevertTests-Volume-1536424582',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=9a399a62-6a8f-4b6d-8ecf-8dcdf57e5fbb,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='9c755d3b5d8a4d9c9918b6c4744f510e',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:57:47Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:57:47Z,use_quota=True,user_id='e362cfe4a31c444b9ab44a5d60f37ad9',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 16:00:04 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:57:47Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumeRevertTests-Volume-1536424582',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=9a399a62-6a8f-4b6d-8ecf-8dcdf57e5fbb,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='9c755d3b5d8a4d9c9918b6c4744f510e',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:57:47Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:57:47Z,use_quota=True,user_id='e362cfe4a31c444b9ab44a5d60f37ad9',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:57:47Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumeRevertTests-Volume-1536424582',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=9a399a62-6a8f-4b6d-8ecf-8dcdf57e5fbb,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='9c755d3b5d8a4d9c9918b6c4744f510e',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:57:47Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:57:47Z,use_quota=True,user_id='e362cfe4a31c444b9ab44a5d60f37ad9',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 16:00:04 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 16:00:04 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 16:00:04 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:00:04 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 16:00:04 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:00:04 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 16:00:04 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:00:04 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 16:00:04 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:00:04 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 16:00:04 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:00:04 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 16:00:04 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:00:04 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 16:00:04 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:00:04 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 16:00:04 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:00:04 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:00:04 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 16:00:04 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 16:00:04 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (fa6c819a-c3c4-4f92-97d3-d0278e26aca5) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:00:04 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (fa6c819a-c3c4-4f92-97d3-d0278e26aca5) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:00:04 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (4459bc91-9990-4b6d-bd17-93ea53768161) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:00:04 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (4459bc91-9990-4b6d-bd17-93ea53768161) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:00:04 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (d5e95f2f-b2ca-4d16-89a5-b33816fc5ac3) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:00:04 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (d5e95f2f-b2ca-4d16-89a5-b33816fc5ac3) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:00:04 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (ddaa5868-0c7e-4fad-ac69-846477b8c783) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:00:04 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Updating volume 9a399a62-6a8f-4b6d-8ecf-8dcdf57e5fbb with {'scheduled_at': datetime.datetime(2023, 4, 17, 16, 0, 4, 247647), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 16:00:04 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Volume 9a399a62-6a8f-4b6d-8ecf-8dcdf57e5fbb: re-scheduling SchedulerAPI.create_volume attempt 1 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 16:00:04 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Volume 9a399a62-6a8f-4b6d-8ecf-8dcdf57e5fbb: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 16:00:04 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] deleating volume v_9a399a62-6a8f-4b6d-8ecf-8dcdf57e5fbb {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 16:00:04 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] get snapshots for volume v_9a399a62-6a8f-4b6d-8ecf-8dcdf57e5fbb {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 16:00:04 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_9a399a62-6a8f-4b6d-8ecf-8dcdf57e5fbb/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:00:04 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_9a399a62-6a8f-4b6d-8ecf-8dcdf57e5fbb/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:00:04 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Volume reschedule parameters: True retry: {'num_attempts': 2, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 16:00:04 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Flow 'volume_create_manager' (77678606-9fe2-4e14-9e75-217e8d1018c1) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 16:00:04 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (fde380d8-9526-4ceb-88c5-668baf7bc4d8) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:00:04 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (fde380d8-9526-4ceb-88c5-668baf7bc4d8) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:57:47Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumeRevertTests-Volume-1536424582',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=9a399a62-6a8f-4b6d-8ecf-8dcdf57e5fbb,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='9c755d3b5d8a4d9c9918b6c4744f510e',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:00:04Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:00:04Z,use_quota=True,user_id='e362cfe4a31c444b9ab44a5d60f37ad9',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:00:04 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (9b7ada94-459d-4587-8d7b-e2197a372198) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:00:04 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (9b7ada94-459d-4587-8d7b-e2197a372198) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:00:04 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (d467f31f-5732-407b-9042-43a0aeb61a36) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:00:04 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (d467f31f-5732-407b-9042-43a0aeb61a36) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': '9a399a62-6a8f-4b6d-8ecf-8dcdf57e5fbb', 'volume_name': 'volume-9a399a62-6a8f-4b6d-8ecf-8dcdf57e5fbb', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:00:04 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (9b878174-1ccb-4a96-a848-c22d15f4aab9) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:00:04 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (9b878174-1ccb-4a96-a848-c22d15f4aab9) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:00:04 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (ee2be40e-352e-4b95-8bf2-4d4659185323) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:00:04 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Volume 9a399a62-6a8f-4b6d-8ecf-8dcdf57e5fbb: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-9a399a62-6a8f-4b6d-8ecf-8dcdf57e5fbb', 'volume_size': 1} Apr 17 16:00:04 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] creating volume v_9a399a62-6a8f-4b6d-8ecf-8dcdf57e5fbb. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 16:00:04 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] create volume {'name': 'v_9a399a62-6a8f-4b6d-8ecf-8dcdf57e5fbb', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 16:00:04 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:00:04 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:00:07 user cinder-volume[84696]: DEBUG cinder.utils [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.992(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:00:07 user cinder-volume[84696]: DEBUG cinder.utils [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_9a399a62-6a8f-4b6d-8ecf-8dcdf57e5fbb/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:00:07 user cinder-volume[84696]: DEBUG cinder.utils [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.022(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:00:07 user cinder-volume[84696]: DEBUG cinder.utils [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_45e5cc09-6c97-438a-a6a2-c89e7d725ce5/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:00:07 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 74.334(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:00:07 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:00:07 user cinder-volume[84696]: DEBUG cinder.utils [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.457(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:00:07 user cinder-volume[84696]: DEBUG cinder.utils [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:00:07 user cinder-volume[84696]: DEBUG cinder.utils [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 73.252(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:00:07 user cinder-volume[84696]: DEBUG cinder.utils [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:00:07 user cinder-volume[84696]: DEBUG cinder.utils [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.012(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:00:07 user cinder-volume[84696]: DEBUG cinder.utils [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:00:07 user cinder-volume[84696]: DEBUG cinder.utils [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.182(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:00:07 user cinder-volume[84696]: DEBUG cinder.utils [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_87514055-5a0f-459f-b10b-438d7a13d39d/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:00:07 user cinder-volume[84696]: DEBUG cinder.utils [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.812(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:00:07 user cinder-volume[84696]: DEBUG cinder.utils [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:00:10 user cinder-volume[84696]: DEBUG cinder.utils [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.175(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:00:10 user cinder-volume[84696]: DEBUG cinder.utils [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_14aa38ea-7cbb-48f7-a9aa-ffcc7b66faa0/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:00:10 user cinder-volume[84696]: DEBUG cinder.utils [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 11.836(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:00:10 user cinder-volume[84696]: DEBUG cinder.utils [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:00:10 user cinder-volume[84696]: DEBUG cinder.utils [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.876(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:00:10 user cinder-volume[84696]: DEBUG cinder.utils [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:00:10 user cinder-volume[84696]: DEBUG cinder.utils [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.065(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:00:10 user cinder-volume[84696]: DEBUG cinder.utils [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_9a399a62-6a8f-4b6d-8ecf-8dcdf57e5fbb/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:00:13 user cinder-volume[84696]: DEBUG cinder.utils [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.161(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:00:13 user cinder-volume[84696]: DEBUG cinder.utils [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_45e5cc09-6c97-438a-a6a2-c89e7d725ce5/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:00:16 user cinder-volume[84696]: DEBUG cinder.utils [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.017(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:00:16 user cinder-volume[84696]: DEBUG cinder.utils [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:00:16 user cinder-volume[84696]: DEBUG cinder.utils [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.391(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:00:16 user cinder-volume[84696]: DEBUG cinder.utils [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_87514055-5a0f-459f-b10b-438d7a13d39d/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:00:16 user cinder-volume[84696]: DEBUG cinder.utils [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.223(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:00:16 user cinder-volume[84696]: DEBUG cinder.utils [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:00:16 user cinder-volume[84696]: DEBUG cinder.utils [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.211(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:00:16 user cinder-volume[84696]: DEBUG cinder.utils [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_9a399a62-6a8f-4b6d-8ecf-8dcdf57e5fbb/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:00:19 user cinder-volume[84696]: DEBUG cinder.utils [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.050(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:00:19 user cinder-volume[84696]: DEBUG cinder.utils [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:00:19 user cinder-volume[84696]: DEBUG cinder.utils [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.393(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:00:19 user cinder-volume[84696]: DEBUG cinder.utils [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_14aa38ea-7cbb-48f7-a9aa-ffcc7b66faa0/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:00:22 user cinder-volume[84696]: DEBUG cinder.utils [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 73.849(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:00:22 user cinder-volume[84696]: DEBUG cinder.utils [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:00:22 user cinder-volume[84696]: DEBUG cinder.utils [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 74.222(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:00:22 user cinder-volume[84696]: DEBUG cinder.utils [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_b29a7592-128a-41ee-a8e2-57780dee33e7/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:00:22 user cinder-volume[84696]: DEBUG cinder.utils [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.384(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:00:22 user cinder-volume[84696]: DEBUG cinder.utils [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_45e5cc09-6c97-438a-a6a2-c89e7d725ce5/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:00:24 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=23,cinder:UPDATE=6,cinder:INSERT=2 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:00:24 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=26,cinder:UPDATE=11,cinder:INSERT=5 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:00:25 user cinder-volume[84696]: DEBUG cinder.utils [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.420(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:00:25 user cinder-volume[84696]: DEBUG cinder.utils [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_9a399a62-6a8f-4b6d-8ecf-8dcdf57e5fbb/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:00:25 user cinder-volume[84696]: DEBUG cinder.utils [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.885(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:00:25 user cinder-volume[84696]: DEBUG cinder.utils [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:00:25 user cinder-volume[84696]: DEBUG cinder.utils [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.237(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:00:25 user cinder-volume[84696]: DEBUG cinder.utils [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:00:28 user cinder-volume[84696]: DEBUG cinder.utils [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 74.164(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:00:28 user cinder-volume[84696]: DEBUG cinder.utils [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_60979eba-96b1-4cb3-bae1-8874a86cb411/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:00:28 user cinder-volume[84696]: DEBUG cinder.utils [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 73.833(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:00:28 user cinder-volume[84696]: DEBUG cinder.utils [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:00:28 user cinder-volume[84696]: DEBUG cinder.utils [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 74.175(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:00:28 user cinder-volume[84696]: DEBUG cinder.utils [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_75163b78-31b2-4676-b93b-542a8ea784c1/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:00:35 user cinder-volume[84696]: DEBUG cinder.utils [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 40.290(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:00:35 user cinder-volume[84696]: DEBUG cinder.utils [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:00:35 user cinder-volume[84696]: DEBUG cinder.utils [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 40.461(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:00:35 user cinder-volume[84696]: DEBUG cinder.utils [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_87514055-5a0f-459f-b10b-438d7a13d39d/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:00:37 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Volume reschedule parameters: True retry: {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 16:00:37 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Flow 'volume_create_manager' (68f959d6-a03d-42c1-82a7-a5abd9ee1327) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 16:00:37 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (d8f39276-9dea-4eaa-b4ed-8e8595c1fc9d) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:00:37 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (d8f39276-9dea-4eaa-b4ed-8e8595c1fc9d) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:00:38Z,deleted=False,deleted_at=None,display_description=None,display_name='volume-for-test-6e9266ff-a917-4dd5-aa4a-c36e59e7a2a6',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=40efe2ca-3a61-44c9-b9e0-bb29b333f0f9,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='192c91b1109345ccb52f57f4db9d36a0',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:00:38Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:00:38Z,use_quota=True,user_id='d36a4ed743744fc8a1281a42a15e5b2f',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(27e07afb-6233-4285-b990-341afc23321e),volume_type_id=27e07afb-6233-4285-b990-341afc23321e)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:00:37 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (e16a4ead-003c-4857-aa60-d98b521c0783) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:00:37 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (e16a4ead-003c-4857-aa60-d98b521c0783) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:00:37 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (222c2ff9-c07f-4ed0-87a8-76e038bb1d4e) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:00:38 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (222c2ff9-c07f-4ed0-87a8-76e038bb1d4e) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'image', 'volume_id': '40efe2ca-3a61-44c9-b9e0-bb29b333f0f9', 'volume_name': 'volume-40efe2ca-3a61-44c9-b9e0-bb29b333f0f9', 'volume_size': 1, 'image_id': 'c0c829d5-daba-4365-b51f-aabe77d8aa62', 'image_location': (None, None), 'image_meta': {'name': 'image-for-test-6e9266ff-a917-4dd5-aa4a-c36e59e7a2a6', 'disk_format': 'raw', 'container_format': 'bare', 'visibility': 'private', 'size': 1024, 'virtual_size': 1024, 'status': 'active', 'checksum': '9d24cb2b43eab1d7d24f46a1c59d4405', 'protected': False, 'min_ram': 0, 'min_disk': 1, 'owner': '192c91b1109345ccb52f57f4db9d36a0', 'os_hidden': False, 'os_hash_algo': 'sha512', 'os_hash_value': '11384674df78c62ef87566997f126283d8379bbba0f2865f3f37e41378078ebe9ad290c06b163f2a99198703c2d61c2145df7d24bfec81fd5a51872b7b5836c1', 'id': 'c0c829d5-daba-4365-b51f-aabe77d8aa62', 'created_at': datetime.datetime(2023, 4, 17, 16, 0, 36, tzinfo=datetime.timezone.utc), 'updated_at': datetime.datetime(2023, 4, 17, 16, 0, 37, tzinfo=datetime.timezone.utc), 'tags': [], 'file': '/v2/images/c0c829d5-daba-4365-b51f-aabe77d8aa62/file', 'properties': {'cinder_img_volume_type': 'vol-type-for-6e9266ff-a917-4dd5-aa4a-c36e59e7a2a6'}}, 'image_service': }' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:00:38 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (5f77b79f-cf8e-449c-b020-0a7b712013da) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:00:38 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (5f77b79f-cf8e-449c-b020-0a7b712013da) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:00:38 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (7cf8c16e-4d59-4a27-9a28-b3832935c6b1) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:00:38 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Volume 40efe2ca-3a61-44c9-b9e0-bb29b333f0f9: being created as image with specification: {'status': 'creating', 'volume_name': 'volume-40efe2ca-3a61-44c9-b9e0-bb29b333f0f9', 'volume_size': 1, 'image_id': 'c0c829d5-daba-4365-b51f-aabe77d8aa62', 'image_location': (None, None), 'image_meta': {'name': 'image-for-test-6e9266ff-a917-4dd5-aa4a-c36e59e7a2a6', 'disk_format': 'raw', 'container_format': 'bare', 'visibility': 'private', 'size': 1024, 'virtual_size': 1024, 'status': 'active', 'checksum': '9d24cb2b43eab1d7d24f46a1c59d4405', 'protected': False, 'min_ram': 0, 'min_disk': 1, 'owner': '192c91b1109345ccb52f57f4db9d36a0', 'os_hidden': False, 'os_hash_algo': 'sha512', 'os_hash_value': '11384674df78c62ef87566997f126283d8379bbba0f2865f3f37e41378078ebe9ad290c06b163f2a99198703c2d61c2145df7d24bfec81fd5a51872b7b5836c1', 'id': 'c0c829d5-daba-4365-b51f-aabe77d8aa62', 'created_at': datetime.datetime(2023, 4, 17, 16, 0, 36, tzinfo=datetime.timezone.utc), 'updated_at': datetime.datetime(2023, 4, 17, 16, 0, 37, tzinfo=datetime.timezone.utc), 'tags': [], 'file': '/v2/images/c0c829d5-daba-4365-b51f-aabe77d8aa62/file', 'properties': {'cinder_img_volume_type': 'vol-type-for-6e9266ff-a917-4dd5-aa4a-c36e59e7a2a6'}}, 'image_service': } Apr 17 16:00:38 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Cloning 40efe2ca-3a61-44c9-b9e0-bb29b333f0f9 from image c0c829d5-daba-4365-b51f-aabe77d8aa62 at location (None, None). {{(pid=84696) _create_from_image /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1071}} Apr 17 16:00:38 user cinder-volume[84696]: DEBUG cinder.coordination [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Acquiring lock "cinder-c0c829d5-daba-4365-b51f-aabe77d8aa62" by "_prepare_image_cache_entry" {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:192}} Apr 17 16:00:38 user cinder-volume[84696]: DEBUG cinder.coordination [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Lock "cinder-c0c829d5-daba-4365-b51f-aabe77d8aa62" acquired by "_prepare_image_cache_entry" :: waited 0.008s {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:196}} Apr 17 16:00:38 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Preparing cache entry for image = c0c829d5-daba-4365-b51f-aabe77d8aa62 on host user@joviandss-2#joviandss-2. {{(pid=84696) _prepare_image_cache_entry /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:891}} Apr 17 16:00:38 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Attempting to retrieve cache entry for image = c0c829d5-daba-4365-b51f-aabe77d8aa62 on host user@joviandss-2#joviandss-2. {{(pid=84696) _create_from_image_cache /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:827}} Apr 17 16:00:38 user cinder-volume[84696]: DEBUG cinder.image.image_utils [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Image fetch details: dest /opt/stack/data/cinder/conversion/image_fetch_c0c829d5-daba-4365-b51f-aabe77d8aa62_08a_cxh9user@joviandss-2, size 0.00 MB, duration 1.00 sec {{(pid=84696) fetch /opt/stack/cinder/cinder/image/image_utils.py:628}} Apr 17 16:00:38 user cinder-volume[84696]: INFO cinder.image.image_utils [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Image download 0.00 MB at 0.00 MB/s Apr 17 16:00:38 user cinder-volume[84696]: DEBUG oslo_concurrency.processutils [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Running cmd (subprocess): /usr/bin/python3.10 -m oslo_concurrency.prlimit --as=1073741824 --cpu=60 -- sudo cinder-rootwrap /etc/cinder/rootwrap.conf env LC_ALL=C qemu-img info --output=json /opt/stack/data/cinder/conversion/image_fetch_c0c829d5-daba-4365-b51f-aabe77d8aa62_08a_cxh9user@joviandss-2 {{(pid=84696) execute /usr/local/lib/python3.10/dist-packages/oslo_concurrency/processutils.py:384}} Apr 17 16:00:38 user sudo[104005]: stack : PWD=/ ; USER=root ; COMMAND=/usr/local/bin/cinder-rootwrap /etc/cinder/rootwrap.conf env LC_ALL=C qemu-img info --output=json /opt/stack/data/cinder/conversion/image_fetch_c0c829d5-daba-4365-b51f-aabe77d8aa62_08a_cxh9user@joviandss-2 Apr 17 16:00:38 user sudo[104005]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=1001) Apr 17 16:00:38 user cinder-volume[84696]: DEBUG cinder.utils [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 40.464(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:00:38 user cinder-volume[84696]: DEBUG cinder.utils [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_14aa38ea-7cbb-48f7-a9aa-ffcc7b66faa0/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:00:38 user cinder-volume[84696]: DEBUG cinder.utils [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 40.127(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:00:38 user cinder-volume[84696]: DEBUG cinder.utils [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:00:39 user sudo[104005]: pam_unix(sudo:session): session closed for user root Apr 17 16:00:39 user cinder-volume[84696]: DEBUG oslo_concurrency.processutils [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] CMD "/usr/bin/python3.10 -m oslo_concurrency.prlimit --as=1073741824 --cpu=60 -- sudo cinder-rootwrap /etc/cinder/rootwrap.conf env LC_ALL=C qemu-img info --output=json /opt/stack/data/cinder/conversion/image_fetch_c0c829d5-daba-4365-b51f-aabe77d8aa62_08a_cxh9user@joviandss-2" returned: 0 in 0.941s {{(pid=84696) execute /usr/local/lib/python3.10/dist-packages/oslo_concurrency/processutils.py:422}} Apr 17 16:00:39 user cinder-volume[84696]: DEBUG cinder.image.image_utils [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Temporary image c0c829d5-daba-4365-b51f-aabe77d8aa62 is fetched for user d36a4ed743744fc8a1281a42a15e5b2f. {{(pid=84696) fetch /opt/stack/cinder/cinder/image/image_utils.py:1279}} Apr 17 16:00:39 user cinder-volume[84696]: DEBUG oslo_concurrency.processutils [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Running cmd (subprocess): /usr/bin/python3.10 -m oslo_concurrency.prlimit --as=1073741824 --cpu=60 -- sudo cinder-rootwrap /etc/cinder/rootwrap.conf env LC_ALL=C qemu-img info --output=json /opt/stack/data/cinder/conversion/image_fetch_c0c829d5-daba-4365-b51f-aabe77d8aa62_08a_cxh9user@joviandss-2 {{(pid=84696) execute /usr/local/lib/python3.10/dist-packages/oslo_concurrency/processutils.py:384}} Apr 17 16:00:39 user sudo[104012]: stack : PWD=/ ; USER=root ; COMMAND=/usr/local/bin/cinder-rootwrap /etc/cinder/rootwrap.conf env LC_ALL=C qemu-img info --output=json /opt/stack/data/cinder/conversion/image_fetch_c0c829d5-daba-4365-b51f-aabe77d8aa62_08a_cxh9user@joviandss-2 Apr 17 16:00:39 user sudo[104012]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=1001) Apr 17 16:00:39 user sudo[104012]: pam_unix(sudo:session): session closed for user root Apr 17 16:00:39 user cinder-volume[84696]: DEBUG oslo_concurrency.processutils [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] CMD "/usr/bin/python3.10 -m oslo_concurrency.prlimit --as=1073741824 --cpu=60 -- sudo cinder-rootwrap /etc/cinder/rootwrap.conf env LC_ALL=C qemu-img info --output=json /opt/stack/data/cinder/conversion/image_fetch_c0c829d5-daba-4365-b51f-aabe77d8aa62_08a_cxh9user@joviandss-2" returned: 0 in 0.767s {{(pid=84696) execute /usr/local/lib/python3.10/dist-packages/oslo_concurrency/processutils.py:422}} Apr 17 16:00:39 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] creating volume v_40efe2ca-3a61-44c9-b9e0-bb29b333f0f9. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 16:00:39 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] create volume {'name': 'v_40efe2ca-3a61-44c9-b9e0-bb29b333f0f9', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 16:00:39 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:00:39 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:00:41 user cinder-volume[84696]: DEBUG cinder.utils [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 40.449(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:00:41 user cinder-volume[84696]: DEBUG cinder.utils [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_45e5cc09-6c97-438a-a6a2-c89e7d725ce5/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:00:41 user cinder-volume[84696]: DEBUG cinder.utils [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 1.776(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:00:41 user cinder-volume[84696]: DEBUG cinder.utils [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:00:44 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Volume reschedule parameters: True retry: {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 16:00:44 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Flow 'volume_create_manager' (434001cf-d61c-49f1-b8da-ca2f66c1fb8f) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 16:00:44 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (e0124657-9f65-43d2-bc18-f1f7f16a47ea) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:00:44 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (e0124657-9f65-43d2-bc18-f1f7f16a47ea) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:00:44Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-AttachVolumeShelveTestJSON-volume-7555594',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=b4281d17-c1d8-48d3-8a0b-8fd7d2f193cc,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='214a0f233f114d7aa22e64048a837c0b',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:00:44Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:00:44Z,use_quota=True,user_id='83b9d77de46143d8b9277a55f5ad4d72',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:00:44 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (d1d2659c-e868-4ea2-a545-fe10a196244c) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:00:44 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (d1d2659c-e868-4ea2-a545-fe10a196244c) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:00:44 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (62588e89-7ca3-4319-8090-260c64894cac) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:00:44 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (62588e89-7ca3-4319-8090-260c64894cac) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': 'b4281d17-c1d8-48d3-8a0b-8fd7d2f193cc', 'volume_name': 'volume-b4281d17-c1d8-48d3-8a0b-8fd7d2f193cc', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:00:44 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (e954b9ad-7c98-4606-9049-528002a22b12) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:00:44 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (e954b9ad-7c98-4606-9049-528002a22b12) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:00:44 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (3607be78-7861-4ce5-9705-ae5cc87302c3) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:00:44 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Volume b4281d17-c1d8-48d3-8a0b-8fd7d2f193cc: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-b4281d17-c1d8-48d3-8a0b-8fd7d2f193cc', 'volume_size': 1} Apr 17 16:00:44 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] creating volume v_b4281d17-c1d8-48d3-8a0b-8fd7d2f193cc. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 16:00:44 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] create volume {'name': 'v_b4281d17-c1d8-48d3-8a0b-8fd7d2f193cc', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 16:00:44 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:00:44 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:00:44 user cinder-volume[84696]: DEBUG cinder.utils [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 40.492(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:00:44 user cinder-volume[84696]: DEBUG cinder.utils [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_9a399a62-6a8f-4b6d-8ecf-8dcdf57e5fbb/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:00:44 user cinder-volume[84696]: DEBUG cinder.utils [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 4.848(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:00:44 user cinder-volume[84696]: DEBUG cinder.utils [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:00:44 user cinder-volume[84696]: DEBUG cinder.utils [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 0.277(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:00:44 user cinder-volume[84696]: DEBUG cinder.utils [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:00:44 user cinder-volume[84696]: DEBUG cinder.utils [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 40.308(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:00:44 user cinder-volume[84696]: DEBUG cinder.utils [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:00:47 user cinder-volume[84696]: DEBUG cinder.utils [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 3.350(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:00:47 user cinder-volume[84696]: DEBUG cinder.utils [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:00:50 user cinder-volume[84696]: DEBUG cinder.utils [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 10.990(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:00:50 user cinder-volume[84696]: DEBUG cinder.utils [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:00:53 user cinder-volume[84696]: DEBUG cinder.utils [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 9.490(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:00:53 user cinder-volume[84696]: DEBUG cinder.utils [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:01:00 user cinder-volume[84696]: DEBUG cinder.utils [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 20.206(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:01:00 user cinder-volume[84696]: DEBUG cinder.utils [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:01:00 user cinder-volume[84696]: DEBUG cinder.utils [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 74.315(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:01:00 user cinder-volume[84696]: DEBUG cinder.utils [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:01:03 user cinder-volume[84696]: DEBUG cinder.utils [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 18.707(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:01:03 user cinder-volume[84696]: DEBUG cinder.utils [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:01:04 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=17,cinder:UPDATE=1 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:01:04 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=14,cinder:UPDATE=5,cinder:INSERT=1 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:01:06 user cinder-volume[84696]: DEBUG cinder.utils [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 139.384(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:01:06 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_bff6fdbb-a1fb-4b48-bca8-2f7efbc3f0ff/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:01:06 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (c8a9a018-c594-4f2a-b9b3-d1b09245a3fb) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 16:01:06 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (d4d9be2e-5bd2-49db-9e90-cc6fbb3cef06) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:01:06 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (d4d9be2e-5bd2-49db-9e90-cc6fbb3cef06) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:01:06 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Flow 'volume_create_manager' (21b8dbdb-80d0-4b1f-9c54-7ac019e3531c) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 16:01:06 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 16:01:06 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 16:01:06 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 16:01:06 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 16:01:06 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 16:01:06 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 16:01:06 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 16:01:06 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 16:01:06 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 16:01:06 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 16:01:06 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 16:01:06 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 16:01:06 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 16:01:06 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 16:01:06 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 16:01:06 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 16:01:06 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 16:01:06 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 16:01:06 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 16:01:06 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 16:01:06 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 16:01:06 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 16:01:06 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 16:01:06 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 16:01:06 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:01:06 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 16:01:06 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:01:06 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 16:01:06 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:01:06 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 16:01:06 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:01:06 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 16:01:06 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:01:06 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 16:01:06 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:01:06 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 16:01:06 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:01:06 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 16:01:06 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:01:06 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:01:06 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 16:01:06 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 16:01:06 user cinder-volume[84696]: DEBUG cinder.utils [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 139.231(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:01:06 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:01:06 user cinder-volume[84696]: INFO cinder.message.api [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Creating message record for request_id = req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 Apr 17 16:01:06 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 16:01:06 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (beed5bb1-af79-4e0a-88f6-18309da5d010) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 16:01:06 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 16:01:06 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:56:26Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumeRetypeWithoutMigrationTest-Volume-636058295',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=bff6fdbb-a1fb-4b48-bca8-2f7efbc3f0ff,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='46c9cac22c32484181c934b27e67f512',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:58:47Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:58:47Z,use_quota=True,user_id='a0751e2174804d7ab462edae5f88d581',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(c9c52f37-7390-466f-a6ab-56ae9a61e5ed),volume_type_id=c9c52f37-7390-466f-a6ab-56ae9a61e5ed)}, 'provides': None} Apr 17 16:01:06 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:56:26Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumeRetypeWithoutMigrationTest-Volume-636058295',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=bff6fdbb-a1fb-4b48-bca8-2f7efbc3f0ff,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='46c9cac22c32484181c934b27e67f512',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:58:47Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:58:47Z,use_quota=True,user_id='a0751e2174804d7ab462edae5f88d581',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(c9c52f37-7390-466f-a6ab-56ae9a61e5ed),volume_type_id=c9c52f37-7390-466f-a6ab-56ae9a61e5ed), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(bff6fdbb-a1fb-4b48-bca8-2f7efbc3f0ff),volume_id=bff6fdbb-a1fb-4b48-bca8-2f7efbc3f0ff,volume_properties=VolumeProperties,volume_type=VolumeType(c9c52f37-7390-466f-a6ab-56ae9a61e5ed))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': 'bff6fdbb-a1fb-4b48-bca8-2f7efbc3f0ff', 'volume_name': 'volume-bff6fdbb-a1fb-4b48-bca8-2f7efbc3f0ff', 'volume_size': 1}} Apr 17 16:01:06 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': 'bff6fdbb-a1fb-4b48-bca8-2f7efbc3f0ff', 'volume': {'id': 'bff6fdbb-a1fb-4b48-bca8-2f7efbc3f0ff', '_name_id': None, 'ec2_id': None, 'user_id': 'a0751e2174804d7ab462edae5f88d581', 'project_id': '46c9cac22c32484181c934b27e67f512', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-VolumeRetypeWithoutMigrationTest-Volume-636058295', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': 'c9c52f37-7390-466f-a6ab-56ae9a61e5ed', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'admin_metadata': {}, 'glance_metadata': {}, 'volume_type': {'id': 'c9c52f37-7390-466f-a6ab-56ae9a61e5ed', 'name': 'volume-type-1', 'description': None, 'is_public': True, 'extra_specs': {}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:56:25.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T15:56:25.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-bff6fdbb-a1fb-4b48-bca8-2f7efbc3f0ff', 'name_id': 'bff6fdbb-a1fb-4b48-bca8-2f7efbc3f0ff', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': 'c9c52f37-7390-466f-a6ab-56ae9a61e5ed', 'name': 'volume-type-1', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:56:25.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumeRetypeWithoutMigrationTest-Volume-636058295', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '46c9cac22c32484181c934b27e67f512', 'qos_specs': None, 'reservations': ['622ac28a-652f-4d3c-b14a-4e0cf9309855', '3f27124e-4ba9-448f-a619-0c453485c180', '93656d14-fa21-40b5-acf9-87e241ec0ef5', 'a859cf24-fdaa-4c02-a26d-31ecb6289b76'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'a0751e2174804d7ab462edae5f88d581', 'volume_type_id': 'c9c52f37-7390-466f-a6ab-56ae9a61e5ed'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumeRetypeWithoutMigrationTest-Volume-636058295', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '46c9cac22c32484181c934b27e67f512', 'qos_specs': None, 'reservations': ['622ac28a-652f-4d3c-b14a-4e0cf9309855', '3f27124e-4ba9-448f-a619-0c453485c180', '93656d14-fa21-40b5-acf9-87e241ec0ef5', 'a859cf24-fdaa-4c02-a26d-31ecb6289b76'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'a0751e2174804d7ab462edae5f88d581', 'volume_type_id': 'c9c52f37-7390-466f-a6ab-56ae9a61e5ed'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:56:25Z,deleted=False,deleted_at=None,description=None,extra_specs={},id=c9c52f37-7390-466f-a6ab-56ae9a61e5ed,is_public=True,name='volume-type-1',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:56:25Z,deleted=False,deleted_at=None,description=None,extra_specs={},id=c9c52f37-7390-466f-a6ab-56ae9a61e5ed,is_public=True,name='volume-type-1',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': 'a0751e2174804d7ab462edae5f88d581', 'metadata': {}, 'qos_specs': None, 'retry': {'num_attempts': 2, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']}}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(bff6fdbb-a1fb-4b48-bca8-2f7efbc3f0ff),volume_id=bff6fdbb-a1fb-4b48-bca8-2f7efbc3f0ff,volume_properties=VolumeProperties,volume_type=VolumeType(c9c52f37-7390-466f-a6ab-56ae9a61e5ed)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:56:26Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumeRetypeWithoutMigrationTest-Volume-636058295',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=bff6fdbb-a1fb-4b48-bca8-2f7efbc3f0ff,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='46c9cac22c32484181c934b27e67f512',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:58:47Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:58:47Z,use_quota=True,user_id='a0751e2174804d7ab462edae5f88d581',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(c9c52f37-7390-466f-a6ab-56ae9a61e5ed),volume_type_id=c9c52f37-7390-466f-a6ab-56ae9a61e5ed), 'context': }, 'provides': None} Apr 17 16:01:06 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:56:26Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumeRetypeWithoutMigrationTest-Volume-636058295',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=bff6fdbb-a1fb-4b48-bca8-2f7efbc3f0ff,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='46c9cac22c32484181c934b27e67f512',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:58:47Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:58:47Z,use_quota=True,user_id='a0751e2174804d7ab462edae5f88d581',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(c9c52f37-7390-466f-a6ab-56ae9a61e5ed),volume_type_id=c9c52f37-7390-466f-a6ab-56ae9a61e5ed)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:56:26Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumeRetypeWithoutMigrationTest-Volume-636058295',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=bff6fdbb-a1fb-4b48-bca8-2f7efbc3f0ff,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='46c9cac22c32484181c934b27e67f512',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:58:47Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:58:47Z,use_quota=True,user_id='a0751e2174804d7ab462edae5f88d581',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(c9c52f37-7390-466f-a6ab-56ae9a61e5ed),volume_type_id=c9c52f37-7390-466f-a6ab-56ae9a61e5ed)} Apr 17 16:01:06 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 16:01:06 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 16:01:06 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:01:06 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 16:01:06 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:01:06 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 16:01:06 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:01:06 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 16:01:06 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:01:06 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 16:01:06 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:01:06 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 16:01:06 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:01:06 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 16:01:06 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:01:06 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 16:01:06 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:01:06 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:01:06 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 16:01:06 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 16:01:06 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (beed5bb1-af79-4e0a-88f6-18309da5d010) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:01:06 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (beed5bb1-af79-4e0a-88f6-18309da5d010) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:01:06 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (7bd1a7db-0187-4994-8ace-37be25ed3b59) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:01:06 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (7bd1a7db-0187-4994-8ace-37be25ed3b59) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:01:06 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (80f20411-2ada-47f3-af2d-dbec4f4428fa) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:01:06 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (80f20411-2ada-47f3-af2d-dbec4f4428fa) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:01:06 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (51865b1d-35c0-44f9-8c4d-2f75409a79d7) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:01:06 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Updating volume bff6fdbb-a1fb-4b48-bca8-2f7efbc3f0ff with {'scheduled_at': datetime.datetime(2023, 4, 17, 16, 1, 6, 355764), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 16:01:06 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Volume bff6fdbb-a1fb-4b48-bca8-2f7efbc3f0ff: re-scheduling SchedulerAPI.create_volume attempt 2 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 16:01:06 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Volume bff6fdbb-a1fb-4b48-bca8-2f7efbc3f0ff: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 16:01:06 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] deleating volume v_bff6fdbb-a1fb-4b48-bca8-2f7efbc3f0ff {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 16:01:06 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] get snapshots for volume v_bff6fdbb-a1fb-4b48-bca8-2f7efbc3f0ff {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 16:01:06 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_bff6fdbb-a1fb-4b48-bca8-2f7efbc3f0ff/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:01:06 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_bff6fdbb-a1fb-4b48-bca8-2f7efbc3f0ff/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:01:06 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Volume reschedule parameters: True retry: {'num_attempts': 3, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 16:01:06 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Flow 'volume_create_manager' (925f5bf9-fd63-4750-b793-b971d34a6a40) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 16:01:06 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (eca6f0b1-a41d-49c3-b746-77e961f97b80) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:01:06 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (eca6f0b1-a41d-49c3-b746-77e961f97b80) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:56:26Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumeRetypeWithoutMigrationTest-Volume-636058295',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=bff6fdbb-a1fb-4b48-bca8-2f7efbc3f0ff,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='46c9cac22c32484181c934b27e67f512',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:01:06Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:01:06Z,use_quota=True,user_id='a0751e2174804d7ab462edae5f88d581',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(c9c52f37-7390-466f-a6ab-56ae9a61e5ed),volume_type_id=c9c52f37-7390-466f-a6ab-56ae9a61e5ed)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:01:06 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (0fea95b6-4035-4c1d-b02e-1897705e16da) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:01:06 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (0fea95b6-4035-4c1d-b02e-1897705e16da) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:01:06 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (52b4f907-04c4-43dd-8f00-116ae1cebc2e) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:01:06 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (52b4f907-04c4-43dd-8f00-116ae1cebc2e) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': 'bff6fdbb-a1fb-4b48-bca8-2f7efbc3f0ff', 'volume_name': 'volume-bff6fdbb-a1fb-4b48-bca8-2f7efbc3f0ff', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:01:06 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (0c941742-3688-4923-a159-ff84eba2e1e8) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:01:06 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (0c941742-3688-4923-a159-ff84eba2e1e8) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:01:06 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (5d26daf0-e780-4ae9-8b4a-8f83e3f0f239) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:01:06 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Volume bff6fdbb-a1fb-4b48-bca8-2f7efbc3f0ff: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-bff6fdbb-a1fb-4b48-bca8-2f7efbc3f0ff', 'volume_size': 1} Apr 17 16:01:06 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] creating volume v_bff6fdbb-a1fb-4b48-bca8-2f7efbc3f0ff. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 16:01:06 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] create volume {'name': 'v_bff6fdbb-a1fb-4b48-bca8-2f7efbc3f0ff', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 16:01:06 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:01:06 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:01:09 user cinder-volume[84696]: DEBUG cinder.utils [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.780(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:01:09 user cinder-volume[84696]: DEBUG cinder.utils [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:01:09 user cinder-volume[84696]: DEBUG cinder.utils [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.958(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:01:09 user cinder-volume[84696]: DEBUG cinder.utils [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_bff6fdbb-a1fb-4b48-bca8-2f7efbc3f0ff/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:01:09 user cinder-volume[84696]: DEBUG cinder.utils [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 74.255(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:01:09 user cinder-volume[84696]: DEBUG cinder.utils [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_87514055-5a0f-459f-b10b-438d7a13d39d/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:01:09 user cinder-volume[84696]: DEBUG cinder.utils [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 74.087(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:01:09 user cinder-volume[84696]: DEBUG cinder.utils [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:01:12 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 139.480(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:01:12 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:01:12 user cinder-volume[84696]: ERROR oslo_service.periodic_task [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Error during VolumeManager.publish_service_capabilities: TypeError: 'NoneType' object is not subscriptable Apr 17 16:01:12 user cinder-volume[84696]: ERROR oslo_service.periodic_task Traceback (most recent call last): Apr 17 16:01:12 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/usr/local/lib/python3.10/dist-packages/oslo_service/periodic_task.py", line 216, in run_periodic_tasks Apr 17 16:01:12 user cinder-volume[84696]: ERROR oslo_service.periodic_task task(self, context) Apr 17 16:01:12 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/manager.py", line 2876, in publish_service_capabilities Apr 17 16:01:12 user cinder-volume[84696]: ERROR oslo_service.periodic_task self._report_driver_status(context) Apr 17 16:01:12 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/manager.py", line 2755, in _report_driver_status Apr 17 16:01:12 user cinder-volume[84696]: ERROR oslo_service.periodic_task volume_stats = get_stats() Apr 17 16:01:12 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/usr/local/lib/python3.10/dist-packages/oslo_utils/timeutils.py", line 319, in wrapper Apr 17 16:01:12 user cinder-volume[84696]: ERROR oslo_service.periodic_task result = func(*args, **kwargs) Apr 17 16:01:12 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/manager.py", line 2753, in get_stats Apr 17 16:01:12 user cinder-volume[84696]: ERROR oslo_service.periodic_task return self.driver.get_volume_stats(refresh=True) Apr 17 16:01:12 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/driver.py", line 666, in get_volume_stats Apr 17 16:01:12 user cinder-volume[84696]: ERROR oslo_service.periodic_task self._update_volume_stats() Apr 17 16:01:12 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 726, in _update_volume_stats Apr 17 16:01:12 user cinder-volume[84696]: ERROR oslo_service.periodic_task pool_stats = self.ra.get_pool_stats() Apr 17 16:01:12 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 897, in get_pool_stats Apr 17 16:01:12 user cinder-volume[84696]: ERROR oslo_service.periodic_task if not resp["error"] and resp["code"] == 200: Apr 17 16:01:12 user cinder-volume[84696]: ERROR oslo_service.periodic_task TypeError: 'NoneType' object is not subscriptable Apr 17 16:01:12 user cinder-volume[84696]: ERROR oslo_service.periodic_task Apr 17 16:01:12 user cinder-volume[84696]: DEBUG cinder.utils [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 138.399(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:01:12 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:01:12 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (d2b724be-e363-4692-aa27-06c48d6720f2) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 16:01:12 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 16:01:12 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:58:54Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TestVolumeBootPattern-volume-origin-619636601',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=fd313648-5fc1-4e6a-97a0-92ac803b36fd,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='c8a836b47d954cb9a7c1cd7c0f790426',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:58:54Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:58:54Z,use_quota=True,user_id='a80e753322224ec1a897598b5b9ed6ea',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 16:01:12 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:58:54Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TestVolumeBootPattern-volume-origin-619636601',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=fd313648-5fc1-4e6a-97a0-92ac803b36fd,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='c8a836b47d954cb9a7c1cd7c0f790426',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:58:54Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:58:54Z,use_quota=True,user_id='a80e753322224ec1a897598b5b9ed6ea',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=b3f0dec1-7758-44a8-82e0-76ada8837428,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(fd313648-5fc1-4e6a-97a0-92ac803b36fd),volume_id=fd313648-5fc1-4e6a-97a0-92ac803b36fd,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'image', 'volume_id': 'fd313648-5fc1-4e6a-97a0-92ac803b36fd', 'volume_name': 'volume-fd313648-5fc1-4e6a-97a0-92ac803b36fd', 'volume_size': 1, 'image_id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'image_location': (None, None), 'image_meta': {'name': 'cirros-0.5.2-x86_64-disk', 'disk_format': 'qcow2', 'container_format': 'bare', 'visibility': 'public', 'size': 16300544, 'virtual_size': 117440512, 'status': 'active', 'checksum': 'b874c39491a2377b8490f5f1e89761a4', 'protected': False, 'min_ram': 0, 'min_disk': 0, 'owner': 'a43c211b4ed542538799349489b1d884', 'os_hidden': False, 'os_hash_algo': 'sha512', 'os_hash_value': '6b813aa46bb90b4da216a4d19376593fa3f4fc7e617f03a92b7fe11e9a3981cbe8f0959dbebe36225e5f53dc4492341a4863cac4ed1ee0909f3fc78ef9c3e869', 'id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'created_at': datetime.datetime(2023, 4, 17, 15, 19, 53, tzinfo=datetime.timezone.utc), 'updated_at': datetime.datetime(2023, 4, 17, 15, 19, 55, tzinfo=datetime.timezone.utc), 'tags': [], 'file': '/v2/images/b3f0dec1-7758-44a8-82e0-76ada8837428/file', 'properties': {'hw_rng_model': 'virtio', 'owner_specified.openstack.md5': '', 'owner_specified.openstack.object': 'images/cirros-0.5.2-x86_64-disk', 'owner_specified.openstack.sha256': ''}}, 'image_service': }} Apr 17 16:01:12 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'retry': {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']}, 'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'snapshot_id': None, 'source_volid': None, 'volume_id': 'fd313648-5fc1-4e6a-97a0-92ac803b36fd', 'volume': {'id': 'fd313648-5fc1-4e6a-97a0-92ac803b36fd', '_name_id': None, 'ec2_id': None, 'user_id': 'a80e753322224ec1a897598b5b9ed6ea', 'project_id': 'c8a836b47d954cb9a7c1cd7c0f790426', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-TestVolumeBootPattern-volume-origin-619636601', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T15:58:53.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-fd313648-5fc1-4e6a-97a0-92ac803b36fd', 'name_id': 'fd313648-5fc1-4e6a-97a0-92ac803b36fd', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-TestVolumeBootPattern-volume-origin-619636601', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': 'c8a836b47d954cb9a7c1cd7c0f790426', 'qos_specs': None, 'reservations': ['85aec33d-518b-4853-aab2-fb3033224b8f', '56b9d1aa-d720-448b-857d-904a1f5517db', 'acdbd69e-bf21-488b-9caa-1525a1d521b2', 'd0d43e22-774a-4a41-86f9-bcde7e88d4b7'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'a80e753322224ec1a897598b5b9ed6ea', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-TestVolumeBootPattern-volume-origin-619636601', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': 'c8a836b47d954cb9a7c1cd7c0f790426', 'qos_specs': None, 'reservations': ['85aec33d-518b-4853-aab2-fb3033224b8f', '56b9d1aa-d720-448b-857d-904a1f5517db', 'acdbd69e-bf21-488b-9caa-1525a1d521b2', 'd0d43e22-774a-4a41-86f9-bcde7e88d4b7'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'a80e753322224ec1a897598b5b9ed6ea', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': 'a80e753322224ec1a897598b5b9ed6ea', 'metadata': {}, 'qos_specs': None}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=b3f0dec1-7758-44a8-82e0-76ada8837428,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(fd313648-5fc1-4e6a-97a0-92ac803b36fd),volume_id=fd313648-5fc1-4e6a-97a0-92ac803b36fd,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:58:54Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TestVolumeBootPattern-volume-origin-619636601',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=fd313648-5fc1-4e6a-97a0-92ac803b36fd,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='c8a836b47d954cb9a7c1cd7c0f790426',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:58:54Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:58:54Z,use_quota=True,user_id='a80e753322224ec1a897598b5b9ed6ea',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 16:01:12 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:58:54Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TestVolumeBootPattern-volume-origin-619636601',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=fd313648-5fc1-4e6a-97a0-92ac803b36fd,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='c8a836b47d954cb9a7c1cd7c0f790426',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:58:54Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:58:54Z,use_quota=True,user_id='a80e753322224ec1a897598b5b9ed6ea',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:58:54Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TestVolumeBootPattern-volume-origin-619636601',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=fd313648-5fc1-4e6a-97a0-92ac803b36fd,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='c8a836b47d954cb9a7c1cd7c0f790426',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:58:54Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:58:54Z,use_quota=True,user_id='a80e753322224ec1a897598b5b9ed6ea',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 16:01:12 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 16:01:12 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 16:01:12 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:01:12 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 16:01:12 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1237, in execute Apr 17 16:01:12 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_from_image(context, Apr 17 16:01:12 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/utils.py", line 670, in _wrapper Apr 17 16:01:12 user cinder-volume[84696]: ERROR cinder.volume.manager return r(f, *args, **kwargs) Apr 17 16:01:12 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/tenacity/__init__.py", line 423, in __call__ Apr 17 16:01:12 user cinder-volume[84696]: ERROR cinder.volume.manager do = self.iter(retry_state=retry_state) Apr 17 16:01:12 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/tenacity/__init__.py", line 360, in iter Apr 17 16:01:12 user cinder-volume[84696]: ERROR cinder.volume.manager return fut.result() Apr 17 16:01:12 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/lib/python3.10/concurrent/futures/_base.py", line 451, in result Apr 17 16:01:12 user cinder-volume[84696]: ERROR cinder.volume.manager return self.__get_result() Apr 17 16:01:12 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result Apr 17 16:01:12 user cinder-volume[84696]: ERROR cinder.volume.manager raise self._exception Apr 17 16:01:12 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/tenacity/__init__.py", line 426, in __call__ Apr 17 16:01:12 user cinder-volume[84696]: ERROR cinder.volume.manager result = fn(*args, **kwargs) Apr 17 16:01:12 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1125, in _create_from_image Apr 17 16:01:12 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_from_image_cache_or_download( Apr 17 16:01:12 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 944, in _create_from_image_cache_or_download Apr 17 16:01:12 user cinder-volume[84696]: ERROR cinder.volume.manager model_update, cloned = self._create_from_image_cache( Apr 17 16:01:12 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 843, in _create_from_image_cache Apr 17 16:01:12 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_from_source_volume( Apr 17 16:01:12 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 659, in _create_from_source_volume Apr 17 16:01:12 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self.driver.create_cloned_volume(volume, srcvol_ref) Apr 17 16:01:12 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 525, in create_cloned_volume Apr 17 16:01:12 user cinder-volume[84696]: ERROR cinder.volume.manager self._clone_object(vname, cvname) Apr 17 16:01:12 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 461, in _clone_object Apr 17 16:01:12 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_snapshot(oname, coname) Apr 17 16:01:12 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 639, in create_snapshot Apr 17 16:01:12 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201, 204): Apr 17 16:01:12 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 16:01:12 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 16:01:12 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (d2b724be-e363-4692-aa27-06c48d6720f2) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:01:12 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (d2b724be-e363-4692-aa27-06c48d6720f2) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:01:12 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (5fe711b6-06d8-48e1-8ec3-a2ad9ff5330f) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:01:12 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (5fe711b6-06d8-48e1-8ec3-a2ad9ff5330f) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:01:12 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (b2b7a1f1-1964-4350-bbc5-ad6aef647d46) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:01:12 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (b2b7a1f1-1964-4350-bbc5-ad6aef647d46) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:01:12 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (435cbcff-be2f-4bb7-bcc2-abf1daaaf114) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:01:12 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Updating volume fd313648-5fc1-4e6a-97a0-92ac803b36fd with {'scheduled_at': datetime.datetime(2023, 4, 17, 16, 1, 12, 462295), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 16:01:12 user cinder-volume[84696]: DEBUG cinder.utils [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 73.967(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:01:12 user cinder-volume[84696]: DEBUG cinder.utils [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:01:12 user cinder-volume[84696]: DEBUG cinder.utils [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 74.309(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:01:12 user cinder-volume[84696]: DEBUG cinder.utils [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_14aa38ea-7cbb-48f7-a9aa-ffcc7b66faa0/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:01:12 user cinder-volume[84696]: DEBUG cinder.utils [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.907(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:01:12 user cinder-volume[84696]: DEBUG cinder.utils [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:01:12 user cinder-volume[84696]: WARNING oslo.service.loopingcall [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Function 'cinder.service.Service.periodic_tasks' run outlasted interval by 79.54 sec Apr 17 16:01:12 user cinder-volume[84696]: DEBUG cinder.utils [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.085(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:01:12 user cinder-volume[84696]: DEBUG cinder.utils [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_bff6fdbb-a1fb-4b48-bca8-2f7efbc3f0ff/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:01:12 user cinder-volume[84696]: DEBUG oslo_service.periodic_task [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Running periodic task VolumeManager.publish_service_capabilities {{(pid=84696) run_periodic_tasks /usr/local/lib/python3.10/dist-packages/oslo_service/periodic_task.py:210}} Apr 17 16:01:12 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Updating volume stats {{(pid=84696) _update_volume_stats /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:724}} Apr 17 16:01:12 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Get pool Pool-0 fsprops {{(pid=84696) get_pool_stats /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:894}} Apr 17 16:01:12 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:01:12 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:01:12 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Volume fd313648-5fc1-4e6a-97a0-92ac803b36fd: re-scheduling SchedulerAPI.create_volume attempt 1 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 16:01:12 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Volume fd313648-5fc1-4e6a-97a0-92ac803b36fd: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 16:01:12 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] deleating volume v_fd313648-5fc1-4e6a-97a0-92ac803b36fd {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 16:01:12 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] get snapshots for volume v_fd313648-5fc1-4e6a-97a0-92ac803b36fd {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 16:01:12 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_fd313648-5fc1-4e6a-97a0-92ac803b36fd/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:01:12 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_fd313648-5fc1-4e6a-97a0-92ac803b36fd/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:01:12 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Volume reschedule parameters: True retry: {'num_attempts': 2, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 16:01:12 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Flow 'volume_create_manager' (858235c9-243c-43a5-b5d3-c0af8c60c14f) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 16:01:12 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (2c7909b5-a2dd-487d-8995-156c9934f822) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:01:12 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (2c7909b5-a2dd-487d-8995-156c9934f822) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:58:54Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TestVolumeBootPattern-volume-origin-619636601',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=fd313648-5fc1-4e6a-97a0-92ac803b36fd,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='c8a836b47d954cb9a7c1cd7c0f790426',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:01:13Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:01:13Z,use_quota=True,user_id='a80e753322224ec1a897598b5b9ed6ea',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:01:12 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (63d555cc-1ab2-4e6a-b847-e940a6de6d3a) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:01:12 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (63d555cc-1ab2-4e6a-b847-e940a6de6d3a) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:01:12 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (50813848-b19a-4cf8-8eb5-7d5ad7a084f2) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:01:12 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (50813848-b19a-4cf8-8eb5-7d5ad7a084f2) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'image', 'volume_id': 'fd313648-5fc1-4e6a-97a0-92ac803b36fd', 'volume_name': 'volume-fd313648-5fc1-4e6a-97a0-92ac803b36fd', 'volume_size': 1, 'image_id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'image_location': (None, None), 'image_meta': {'name': 'cirros-0.5.2-x86_64-disk', 'disk_format': 'qcow2', 'container_format': 'bare', 'visibility': 'public', 'size': 16300544, 'virtual_size': 117440512, 'status': 'active', 'checksum': 'b874c39491a2377b8490f5f1e89761a4', 'protected': False, 'min_ram': 0, 'min_disk': 0, 'owner': 'a43c211b4ed542538799349489b1d884', 'os_hidden': False, 'os_hash_algo': 'sha512', 'os_hash_value': '6b813aa46bb90b4da216a4d19376593fa3f4fc7e617f03a92b7fe11e9a3981cbe8f0959dbebe36225e5f53dc4492341a4863cac4ed1ee0909f3fc78ef9c3e869', 'id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'created_at': datetime.datetime(2023, 4, 17, 15, 19, 53, tzinfo=datetime.timezone.utc), 'updated_at': datetime.datetime(2023, 4, 17, 15, 19, 55, tzinfo=datetime.timezone.utc), 'tags': [], 'file': '/v2/images/b3f0dec1-7758-44a8-82e0-76ada8837428/file', 'properties': {'hw_rng_model': 'virtio', 'owner_specified.openstack.md5': '', 'owner_specified.openstack.object': 'images/cirros-0.5.2-x86_64-disk', 'owner_specified.openstack.sha256': ''}}, 'image_service': }' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:01:12 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (96b75f01-0cc2-40ce-8186-8bece58c13ec) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:01:12 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (96b75f01-0cc2-40ce-8186-8bece58c13ec) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:01:12 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (657e58ac-4118-460b-a2d0-e5df1d120a72) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:01:12 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Volume fd313648-5fc1-4e6a-97a0-92ac803b36fd: being created as image with specification: {'status': 'creating', 'volume_name': 'volume-fd313648-5fc1-4e6a-97a0-92ac803b36fd', 'volume_size': 1, 'image_id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'image_location': (None, None), 'image_meta': {'name': 'cirros-0.5.2-x86_64-disk', 'disk_format': 'qcow2', 'container_format': 'bare', 'visibility': 'public', 'size': 16300544, 'virtual_size': 117440512, 'status': 'active', 'checksum': 'b874c39491a2377b8490f5f1e89761a4', 'protected': False, 'min_ram': 0, 'min_disk': 0, 'owner': 'a43c211b4ed542538799349489b1d884', 'os_hidden': False, 'os_hash_algo': 'sha512', 'os_hash_value': '6b813aa46bb90b4da216a4d19376593fa3f4fc7e617f03a92b7fe11e9a3981cbe8f0959dbebe36225e5f53dc4492341a4863cac4ed1ee0909f3fc78ef9c3e869', 'id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'created_at': datetime.datetime(2023, 4, 17, 15, 19, 53, tzinfo=datetime.timezone.utc), 'updated_at': datetime.datetime(2023, 4, 17, 15, 19, 55, tzinfo=datetime.timezone.utc), 'tags': [], 'file': '/v2/images/b3f0dec1-7758-44a8-82e0-76ada8837428/file', 'properties': {'hw_rng_model': 'virtio', 'owner_specified.openstack.md5': '', 'owner_specified.openstack.object': 'images/cirros-0.5.2-x86_64-disk', 'owner_specified.openstack.sha256': ''}}, 'image_service': } Apr 17 16:01:12 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Cloning fd313648-5fc1-4e6a-97a0-92ac803b36fd from image b3f0dec1-7758-44a8-82e0-76ada8837428 at location (None, None). {{(pid=84696) _create_from_image /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1071}} Apr 17 16:01:12 user cinder-volume[84696]: DEBUG cinder.coordination [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Acquiring lock "cinder-b3f0dec1-7758-44a8-82e0-76ada8837428" by "_prepare_image_cache_entry" {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:192}} Apr 17 16:01:12 user cinder-volume[84696]: DEBUG cinder.coordination [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Lock "cinder-b3f0dec1-7758-44a8-82e0-76ada8837428" acquired by "_prepare_image_cache_entry" :: waited 0.010s {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:196}} Apr 17 16:01:12 user cinder-volume[84696]: DEBUG cinder.image.cache [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Found image-volume cache entry: {'id': 1, 'image_id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'volume_id': '006f20ae-5306-44fe-afdf-e433f79284e8', 'host': 'user@joviandss-2#joviandss-2', 'size': 1, 'image_updated_at': datetime.datetime(2023, 4, 17, 15, 19, 55), 'last_used': datetime.datetime(2023, 4, 17, 16, 1, 12, 769501)}. {{(pid=84696) get_entry /opt/stack/cinder/cinder/image/cache.py:76}} Apr 17 16:01:12 user cinder-volume[84696]: DEBUG cinder.image.cache [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Image-volume cache entry image_update_at = 2023-04-17 15:19:55+00:00, requested image updated_at = 2023-04-17 15:19:55+00:00. {{(pid=84696) _should_update_entry /opt/stack/cinder/cinder/image/cache.py:259}} Apr 17 16:01:12 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Found cache entry for image = b3f0dec1-7758-44a8-82e0-76ada8837428 on host user@joviandss-2#joviandss-2. {{(pid=84696) _prepare_image_cache_entry /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:886}} Apr 17 16:01:12 user cinder-volume[84696]: DEBUG cinder.coordination [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Lock "cinder-b3f0dec1-7758-44a8-82e0-76ada8837428" released by "_prepare_image_cache_entry" :: held 0.015s {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:207}} Apr 17 16:01:12 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Attempting to retrieve cache entry for image = b3f0dec1-7758-44a8-82e0-76ada8837428 on host user@joviandss-2#joviandss-2. {{(pid=84696) _create_from_image_cache /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:827}} Apr 17 16:01:12 user cinder-volume[84696]: DEBUG cinder.image.cache [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Found image-volume cache entry: {'id': 1, 'image_id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'volume_id': '006f20ae-5306-44fe-afdf-e433f79284e8', 'host': 'user@joviandss-2#joviandss-2', 'size': 1, 'image_updated_at': datetime.datetime(2023, 4, 17, 15, 19, 55), 'last_used': datetime.datetime(2023, 4, 17, 16, 1, 12, 783247)}. {{(pid=84696) get_entry /opt/stack/cinder/cinder/image/cache.py:76}} Apr 17 16:01:12 user cinder-volume[84696]: DEBUG cinder.image.cache [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Image-volume cache entry image_update_at = 2023-04-17 15:19:55+00:00, requested image updated_at = 2023-04-17 15:19:55+00:00. {{(pid=84696) _should_update_entry /opt/stack/cinder/cinder/image/cache.py:259}} Apr 17 16:01:12 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Creating from source image-volume 006f20ae-5306-44fe-afdf-e433f79284e8 {{(pid=84696) _create_from_image_cache /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:841}} Apr 17 16:01:12 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] cloned volume 006f20ae-5306-44fe-afdf-e433f79284e8 to fd313648-5fc1-4e6a-97a0-92ac803b36fd {{(pid=84696) create_cloned_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:521}} Apr 17 16:01:12 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] cloning v_006f20ae-5306-44fe-afdf-e433f79284e8 to v_fd313648-5fc1-4e6a-97a0-92ac803b36fd {{(pid=84696) _clone_object /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:456}} Apr 17 16:01:12 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] create snapshot v_fd313648-5fc1-4e6a-97a0-92ac803b36fd {{(pid=84696) create_snapshot /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:635}} Apr 17 16:01:12 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:01:12 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:01:15 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 3.013(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:01:15 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:01:15 user cinder-volume[84696]: DEBUG cinder.utils [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.990(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:01:15 user cinder-volume[84696]: DEBUG cinder.utils [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_fd313648-5fc1-4e6a-97a0-92ac803b36fd/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:01:15 user cinder-volume[84696]: DEBUG cinder.utils [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.658(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:01:15 user cinder-volume[84696]: DEBUG cinder.utils [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:01:15 user cinder-volume[84696]: DEBUG cinder.utils [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 74.246(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:01:15 user cinder-volume[84696]: DEBUG cinder.utils [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_45e5cc09-6c97-438a-a6a2-c89e7d725ce5/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:01:18 user cinder-volume[84696]: DEBUG cinder.utils [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.060(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:01:18 user cinder-volume[84696]: DEBUG cinder.utils [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_fd313648-5fc1-4e6a-97a0-92ac803b36fd/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:01:18 user cinder-volume[84696]: DEBUG cinder.utils [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.728(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:01:18 user cinder-volume[84696]: DEBUG cinder.utils [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:01:18 user cinder-volume[84696]: DEBUG cinder.utils [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 74.289(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:01:18 user cinder-volume[84696]: DEBUG cinder.utils [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_9a399a62-6a8f-4b6d-8ecf-8dcdf57e5fbb/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:01:18 user cinder-volume[84696]: DEBUG cinder.utils [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.179(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:01:18 user cinder-volume[84696]: DEBUG cinder.utils [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_bff6fdbb-a1fb-4b48-bca8-2f7efbc3f0ff/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:01:18 user cinder-volume[84696]: DEBUG cinder.utils [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.004(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:01:18 user cinder-volume[84696]: DEBUG cinder.utils [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:01:18 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.094(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:01:18 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:01:18 user cinder-volume[84696]: DEBUG cinder.utils [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 38.649(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:01:18 user cinder-volume[84696]: DEBUG cinder.utils [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:01:18 user cinder-volume[84696]: DEBUG cinder.utils [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 74.109(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:01:18 user cinder-volume[84696]: DEBUG cinder.utils [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:01:19 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Volume reschedule parameters: True retry: {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 16:01:19 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Flow 'volume_create_manager' (f23579c3-74ef-405a-935d-f3d1e46c2757) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 16:01:19 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (86e0addb-999b-4b63-9589-3d1a3f018737) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:01:19 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (86e0addb-999b-4b63-9589-3d1a3f018737) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:01:19Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-AttachVolumeNegativeTest-volume-1267367334',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=63b2c7c3-8482-49d4-bb89-c4ccf3f2db82,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='8586cc10142d4743874f43cdb0f37aba',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:01:20Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:01:20Z,use_quota=True,user_id='9fdc7c80b4654692a52c8a4c4149324f',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:01:19 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (a81e2362-7b10-448f-b218-2b651857d634) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:01:19 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (a81e2362-7b10-448f-b218-2b651857d634) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:01:19 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (622f8b43-f785-4734-9717-aa003a91a2a2) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:01:19 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (622f8b43-f785-4734-9717-aa003a91a2a2) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': '63b2c7c3-8482-49d4-bb89-c4ccf3f2db82', 'volume_name': 'volume-63b2c7c3-8482-49d4-bb89-c4ccf3f2db82', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:01:19 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (d1c27e31-d670-48de-ab59-e0f9def6013e) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:01:19 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (d1c27e31-d670-48de-ab59-e0f9def6013e) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:01:19 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (1c7ea1e4-da9a-4973-a73f-4bdfe10bff8a) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:01:19 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Volume 63b2c7c3-8482-49d4-bb89-c4ccf3f2db82: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-63b2c7c3-8482-49d4-bb89-c4ccf3f2db82', 'volume_size': 1} Apr 17 16:01:19 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] creating volume v_63b2c7c3-8482-49d4-bb89-c4ccf3f2db82. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 16:01:19 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] create volume {'name': 'v_63b2c7c3-8482-49d4-bb89-c4ccf3f2db82', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 16:01:19 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:01:19 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:01:21 user cinder-volume[84696]: DEBUG cinder.utils [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 1.841(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:01:21 user cinder-volume[84696]: DEBUG cinder.utils [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:01:21 user cinder-volume[84696]: DEBUG cinder.utils [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 37.140(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:01:21 user cinder-volume[84696]: DEBUG cinder.utils [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:01:24 user cinder-volume[84696]: DEBUG cinder.utils [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 11.870(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:01:24 user cinder-volume[84696]: DEBUG cinder.utils [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:01:24 user cinder-volume[84696]: DEBUG cinder.utils [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.205(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:01:24 user cinder-volume[84696]: DEBUG cinder.utils [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_fd313648-5fc1-4e6a-97a0-92ac803b36fd/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:01:24 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.231(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:01:24 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:01:24 user cinder-volume[84696]: DEBUG cinder.utils [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 4.917(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:01:24 user cinder-volume[84696]: DEBUG cinder.utils [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:01:27 user cinder-volume[84696]: DEBUG cinder.utils [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 138.997(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:01:27 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:01:27 user cinder-volume[84696]: INFO cinder.message.api [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Creating message record for request_id = req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b Apr 17 16:01:27 user cinder-volume[84696]: DEBUG cinder.utils [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.222(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:01:27 user cinder-volume[84696]: DEBUG cinder.utils [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:01:27 user cinder-volume[84696]: DEBUG cinder.utils [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.399(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:01:27 user cinder-volume[84696]: DEBUG cinder.utils [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_bff6fdbb-a1fb-4b48-bca8-2f7efbc3f0ff/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:01:27 user cinder-volume[84696]: DEBUG cinder.utils [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 139.381(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:01:27 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_b29a7592-128a-41ee-a8e2-57780dee33e7/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:01:27 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (915a393a-340d-41bd-8e81-3465e5e593ca) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 16:01:27 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (d17cff04-6a68-4fb8-8aff-871f5be05fd2) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:01:27 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (d17cff04-6a68-4fb8-8aff-871f5be05fd2) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:01:27 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Flow 'volume_create_manager' (440df6e8-ca58-404d-8b07-c6b0bc9724c5) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 16:01:27 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 16:01:27 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 16:01:27 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 16:01:27 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 16:01:27 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 16:01:27 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 16:01:27 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 16:01:27 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 16:01:27 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 16:01:27 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 16:01:27 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 16:01:27 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 16:01:27 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 16:01:27 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 16:01:27 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 16:01:27 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 16:01:27 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 16:01:27 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 16:01:27 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 16:01:27 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 16:01:27 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 16:01:27 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 16:01:27 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 16:01:27 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 16:01:27 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:01:27 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 16:01:27 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:01:27 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 16:01:27 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:01:27 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 16:01:27 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:01:27 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 16:01:27 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:01:27 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 16:01:27 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:01:27 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 16:01:27 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:01:27 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 16:01:27 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:01:27 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:01:27 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 16:01:27 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 16:01:27 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 16:01:27 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (1c9ae51c-3231-4186-bea9-3e2d1b20072f) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 16:01:27 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 16:01:27 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:54:30Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumeRevertTests-Volume-921998137',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=b29a7592-128a-41ee-a8e2-57780dee33e7,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='9c755d3b5d8a4d9c9918b6c4744f510e',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:59:09Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:59:09Z,use_quota=True,user_id='e362cfe4a31c444b9ab44a5d60f37ad9',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 16:01:27 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:54:30Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumeRevertTests-Volume-921998137',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=b29a7592-128a-41ee-a8e2-57780dee33e7,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='9c755d3b5d8a4d9c9918b6c4744f510e',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:59:09Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:59:09Z,use_quota=True,user_id='e362cfe4a31c444b9ab44a5d60f37ad9',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(b29a7592-128a-41ee-a8e2-57780dee33e7),volume_id=b29a7592-128a-41ee-a8e2-57780dee33e7,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': 'b29a7592-128a-41ee-a8e2-57780dee33e7', 'volume_name': 'volume-b29a7592-128a-41ee-a8e2-57780dee33e7', 'volume_size': 1}} Apr 17 16:01:27 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': 'b29a7592-128a-41ee-a8e2-57780dee33e7', 'volume': {'id': 'b29a7592-128a-41ee-a8e2-57780dee33e7', '_name_id': None, 'ec2_id': None, 'user_id': 'e362cfe4a31c444b9ab44a5d60f37ad9', 'project_id': '9c755d3b5d8a4d9c9918b6c4744f510e', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-VolumeRevertTests-Volume-921998137', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'admin_metadata': {}, 'glance_metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T15:54:30.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-b29a7592-128a-41ee-a8e2-57780dee33e7', 'name_id': 'b29a7592-128a-41ee-a8e2-57780dee33e7', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumeRevertTests-Volume-921998137', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '9c755d3b5d8a4d9c9918b6c4744f510e', 'qos_specs': None, 'reservations': ['42b0fe0b-67bc-438d-b5b5-5cd8608f187a', '35c09dbe-7331-435e-a061-4f56338d0c2b', '2a16b96e-8144-4c17-a826-7c22f889d9e1', 'a666665d-cbb8-4dfe-9bf7-027ec0df440c'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'e362cfe4a31c444b9ab44a5d60f37ad9', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumeRevertTests-Volume-921998137', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '9c755d3b5d8a4d9c9918b6c4744f510e', 'qos_specs': None, 'reservations': ['42b0fe0b-67bc-438d-b5b5-5cd8608f187a', '35c09dbe-7331-435e-a061-4f56338d0c2b', '2a16b96e-8144-4c17-a826-7c22f889d9e1', 'a666665d-cbb8-4dfe-9bf7-027ec0df440c'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'e362cfe4a31c444b9ab44a5d60f37ad9', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': 'e362cfe4a31c444b9ab44a5d60f37ad9', 'metadata': {}, 'qos_specs': None, 'retry': {'num_attempts': 3, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']}}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(b29a7592-128a-41ee-a8e2-57780dee33e7),volume_id=b29a7592-128a-41ee-a8e2-57780dee33e7,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:54:30Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumeRevertTests-Volume-921998137',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=b29a7592-128a-41ee-a8e2-57780dee33e7,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='9c755d3b5d8a4d9c9918b6c4744f510e',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:59:09Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:59:09Z,use_quota=True,user_id='e362cfe4a31c444b9ab44a5d60f37ad9',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 16:01:27 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:54:30Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumeRevertTests-Volume-921998137',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=b29a7592-128a-41ee-a8e2-57780dee33e7,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='9c755d3b5d8a4d9c9918b6c4744f510e',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:59:09Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:59:09Z,use_quota=True,user_id='e362cfe4a31c444b9ab44a5d60f37ad9',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:54:30Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumeRevertTests-Volume-921998137',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=b29a7592-128a-41ee-a8e2-57780dee33e7,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='9c755d3b5d8a4d9c9918b6c4744f510e',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:59:09Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:59:09Z,use_quota=True,user_id='e362cfe4a31c444b9ab44a5d60f37ad9',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 16:01:27 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 16:01:27 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 16:01:27 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:01:27 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 16:01:27 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:01:27 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 16:01:27 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:01:27 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 16:01:27 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:01:27 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 16:01:27 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:01:27 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 16:01:27 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:01:27 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 16:01:27 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:01:27 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 16:01:27 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:01:27 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:01:27 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 16:01:27 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 16:01:27 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (1c9ae51c-3231-4186-bea9-3e2d1b20072f) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:01:27 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (1c9ae51c-3231-4186-bea9-3e2d1b20072f) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:01:27 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (7c46364d-98d6-442f-b104-6b39969ad9e9) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:01:27 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (7c46364d-98d6-442f-b104-6b39969ad9e9) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:01:27 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (05bb19e4-e525-42d4-ad48-90f4fc4cbd1c) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:01:27 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (05bb19e4-e525-42d4-ad48-90f4fc4cbd1c) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:01:27 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (9fda5870-733f-436b-90f4-6a13f218e372) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:01:27 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Updating volume b29a7592-128a-41ee-a8e2-57780dee33e7 with {'scheduled_at': datetime.datetime(2023, 4, 17, 16, 1, 27, 862136), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 16:01:27 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Volume b29a7592-128a-41ee-a8e2-57780dee33e7: re-scheduling SchedulerAPI.create_volume attempt 3 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 16:01:27 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Volume b29a7592-128a-41ee-a8e2-57780dee33e7: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 16:01:27 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] deleating volume v_b29a7592-128a-41ee-a8e2-57780dee33e7 {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 16:01:27 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] get snapshots for volume v_b29a7592-128a-41ee-a8e2-57780dee33e7 {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 16:01:27 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_b29a7592-128a-41ee-a8e2-57780dee33e7/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:01:27 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_b29a7592-128a-41ee-a8e2-57780dee33e7/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:01:30 user cinder-volume[84696]: DEBUG cinder.utils [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 11.057(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:01:30 user cinder-volume[84696]: DEBUG cinder.utils [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:01:30 user cinder-volume[84696]: DEBUG cinder.utils [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.955(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:01:30 user cinder-volume[84696]: DEBUG cinder.utils [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_b29a7592-128a-41ee-a8e2-57780dee33e7/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:01:33 user cinder-volume[84696]: DEBUG cinder.utils [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.026(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:01:33 user cinder-volume[84696]: DEBUG cinder.utils [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_b29a7592-128a-41ee-a8e2-57780dee33e7/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:01:33 user cinder-volume[84696]: DEBUG cinder.utils [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 139.321(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:01:33 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_60979eba-96b1-4cb3-bae1-8874a86cb411/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:01:33 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (4ff6d4e8-85fa-467e-acdc-870b80c1f4fc) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 16:01:33 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (217e28fa-02c6-477f-a0e6-6097aad3a3b8) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:01:33 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (217e28fa-02c6-477f-a0e6-6097aad3a3b8) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:01:33 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Flow 'volume_create_manager' (865561c5-292c-4abc-9f91-06e1f05fe0c2) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 16:01:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-c605e44a-ec00-461d-b74d-10bd8a32bea2 tempest-TestVolumeBootPattern-1545678105 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 16:01:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 16:01:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 16:01:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 16:01:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 16:01:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 16:01:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 16:01:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 16:01:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 16:01:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 16:01:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 16:01:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 16:01:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 16:01:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 16:01:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 16:01:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 16:01:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 16:01:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 16:01:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 16:01:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 16:01:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 16:01:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 16:01:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 16:01:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 16:01:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:01:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 16:01:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1237, in execute Apr 17 16:01:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_from_image(context, Apr 17 16:01:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/utils.py", line 670, in _wrapper Apr 17 16:01:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return r(f, *args, **kwargs) Apr 17 16:01:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/tenacity/__init__.py", line 423, in __call__ Apr 17 16:01:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server do = self.iter(retry_state=retry_state) Apr 17 16:01:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/tenacity/__init__.py", line 360, in iter Apr 17 16:01:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return fut.result() Apr 17 16:01:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/lib/python3.10/concurrent/futures/_base.py", line 451, in result Apr 17 16:01:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self.__get_result() Apr 17 16:01:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result Apr 17 16:01:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self._exception Apr 17 16:01:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/tenacity/__init__.py", line 426, in __call__ Apr 17 16:01:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = fn(*args, **kwargs) Apr 17 16:01:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1125, in _create_from_image Apr 17 16:01:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_from_image_cache_or_download( Apr 17 16:01:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 944, in _create_from_image_cache_or_download Apr 17 16:01:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update, cloned = self._create_from_image_cache( Apr 17 16:01:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 843, in _create_from_image_cache Apr 17 16:01:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_from_source_volume( Apr 17 16:01:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 659, in _create_from_source_volume Apr 17 16:01:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self.driver.create_cloned_volume(volume, srcvol_ref) Apr 17 16:01:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 525, in create_cloned_volume Apr 17 16:01:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self._clone_object(vname, cvname) Apr 17 16:01:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 461, in _clone_object Apr 17 16:01:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_snapshot(oname, coname) Apr 17 16:01:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 639, in create_snapshot Apr 17 16:01:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201, 204): Apr 17 16:01:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 16:01:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 16:01:33 user cinder-volume[84696]: DEBUG cinder.utils [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 139.351(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:01:33 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_75163b78-31b2-4676-b93b-542a8ea784c1/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:01:33 user cinder-volume[84696]: WARNING cinder.volume.manager [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (9ebb7d34-65cd-4f48-a5e2-e36a75702ea7) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 16:01:33 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (7a4fa399-4332-4cef-b05b-afeb82e522d5) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:01:33 user cinder-volume[84696]: WARNING cinder.volume.manager [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (7a4fa399-4332-4cef-b05b-afeb82e522d5) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:01:33 user cinder-volume[84696]: WARNING cinder.volume.manager [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Flow 'volume_create_manager' (b3bd1bf6-c485-4297-8950-6953e3362a34) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 16:01:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 16:01:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 16:01:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 16:01:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 16:01:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 16:01:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 16:01:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 16:01:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 16:01:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 16:01:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 16:01:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 16:01:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 16:01:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 16:01:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 16:01:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 16:01:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 16:01:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 16:01:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 16:01:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 16:01:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 16:01:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 16:01:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 16:01:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 16:01:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 16:01:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:01:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 16:01:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1237, in execute Apr 17 16:01:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_from_image(context, Apr 17 16:01:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/utils.py", line 670, in _wrapper Apr 17 16:01:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return r(f, *args, **kwargs) Apr 17 16:01:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/tenacity/__init__.py", line 423, in __call__ Apr 17 16:01:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server do = self.iter(retry_state=retry_state) Apr 17 16:01:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/tenacity/__init__.py", line 360, in iter Apr 17 16:01:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return fut.result() Apr 17 16:01:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/lib/python3.10/concurrent/futures/_base.py", line 451, in result Apr 17 16:01:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self.__get_result() Apr 17 16:01:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result Apr 17 16:01:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self._exception Apr 17 16:01:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/tenacity/__init__.py", line 426, in __call__ Apr 17 16:01:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = fn(*args, **kwargs) Apr 17 16:01:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1125, in _create_from_image Apr 17 16:01:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_from_image_cache_or_download( Apr 17 16:01:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 944, in _create_from_image_cache_or_download Apr 17 16:01:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update, cloned = self._create_from_image_cache( Apr 17 16:01:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 843, in _create_from_image_cache Apr 17 16:01:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_from_source_volume( Apr 17 16:01:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 659, in _create_from_source_volume Apr 17 16:01:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self.driver.create_cloned_volume(volume, srcvol_ref) Apr 17 16:01:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 525, in create_cloned_volume Apr 17 16:01:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self._clone_object(vname, cvname) Apr 17 16:01:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 461, in _clone_object Apr 17 16:01:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_snapshot(oname, coname) Apr 17 16:01:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 639, in create_snapshot Apr 17 16:01:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201, 204): Apr 17 16:01:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 16:01:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 16:01:33 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.491(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:01:33 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:01:33 user cinder-volume[84696]: DEBUG cinder.utils [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 139.033(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:01:33 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:01:33 user cinder-volume[84696]: WARNING cinder.volume.manager [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (91febb26-0692-4dda-bb8a-1ac82a6371e0) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 16:01:33 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 16:01:33 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:54:35Z,deleted=False,deleted_at=None,display_description='',display_name='',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=75163b78-31b2-4676-b93b-542a8ea784c1,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='540826be4baf46dba9617b2fd47821a3',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:59:15Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:59:15Z,use_quota=True,user_id='d819e3f51368413c99691d1b45bbcfb1',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 16:01:33 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:54:35Z,deleted=False,deleted_at=None,display_description='',display_name='',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=75163b78-31b2-4676-b93b-542a8ea784c1,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='540826be4baf46dba9617b2fd47821a3',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:59:15Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:59:15Z,use_quota=True,user_id='d819e3f51368413c99691d1b45bbcfb1',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=b3f0dec1-7758-44a8-82e0-76ada8837428,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(75163b78-31b2-4676-b93b-542a8ea784c1),volume_id=75163b78-31b2-4676-b93b-542a8ea784c1,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'image', 'volume_id': '75163b78-31b2-4676-b93b-542a8ea784c1', 'volume_name': 'volume-75163b78-31b2-4676-b93b-542a8ea784c1', 'volume_size': 1, 'image_id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'image_location': (None, None), 'image_meta': {'name': 'cirros-0.5.2-x86_64-disk', 'disk_format': 'qcow2', 'container_format': 'bare', 'visibility': 'public', 'size': 16300544, 'virtual_size': 117440512, 'status': 'active', 'checksum': 'b874c39491a2377b8490f5f1e89761a4', 'protected': False, 'min_ram': 0, 'min_disk': 0, 'owner': 'a43c211b4ed542538799349489b1d884', 'os_hidden': False, 'os_hash_algo': 'sha512', 'os_hash_value': '6b813aa46bb90b4da216a4d19376593fa3f4fc7e617f03a92b7fe11e9a3981cbe8f0959dbebe36225e5f53dc4492341a4863cac4ed1ee0909f3fc78ef9c3e869', 'id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'created_at': datetime.datetime(2023, 4, 17, 15, 19, 53, tzinfo=datetime.timezone.utc), 'updated_at': datetime.datetime(2023, 4, 17, 15, 19, 55, tzinfo=datetime.timezone.utc), 'tags': [], 'file': '/v2/images/b3f0dec1-7758-44a8-82e0-76ada8837428/file', 'properties': {'hw_rng_model': 'virtio', 'owner_specified.openstack.md5': '', 'owner_specified.openstack.object': 'images/cirros-0.5.2-x86_64-disk', 'owner_specified.openstack.sha256': ''}}, 'image_service': }} Apr 17 16:01:33 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'snapshot_id': None, 'source_volid': None, 'volume_id': '75163b78-31b2-4676-b93b-542a8ea784c1', 'volume': {'id': '75163b78-31b2-4676-b93b-542a8ea784c1', '_name_id': None, 'ec2_id': None, 'user_id': 'd819e3f51368413c99691d1b45bbcfb1', 'project_id': '540826be4baf46dba9617b2fd47821a3', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': '', 'display_description': '', 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'admin_metadata': {}, 'glance_metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T15:54:35.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-75163b78-31b2-4676-b93b-542a8ea784c1', 'name_id': '75163b78-31b2-4676-b93b-542a8ea784c1', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': '', 'display_name': '', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '540826be4baf46dba9617b2fd47821a3', 'qos_specs': None, 'reservations': ['351bb8e5-5094-4129-bd6a-8f086342ae4f', '4dc05382-d603-486b-ab14-e1caee8118dd', 'b04f0e6c-aaea-44ea-b237-e337edc22236', 'e402f3ac-509b-4dac-8852-9c619b150ea3'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'd819e3f51368413c99691d1b45bbcfb1', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': '', 'display_name': '', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '540826be4baf46dba9617b2fd47821a3', 'qos_specs': None, 'reservations': ['351bb8e5-5094-4129-bd6a-8f086342ae4f', '4dc05382-d603-486b-ab14-e1caee8118dd', 'b04f0e6c-aaea-44ea-b237-e337edc22236', 'e402f3ac-509b-4dac-8852-9c619b150ea3'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'd819e3f51368413c99691d1b45bbcfb1', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': 'd819e3f51368413c99691d1b45bbcfb1', 'metadata': {}, 'qos_specs': None, 'retry': {'num_attempts': 3, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']}}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=b3f0dec1-7758-44a8-82e0-76ada8837428,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(75163b78-31b2-4676-b93b-542a8ea784c1),volume_id=75163b78-31b2-4676-b93b-542a8ea784c1,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:54:35Z,deleted=False,deleted_at=None,display_description='',display_name='',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=75163b78-31b2-4676-b93b-542a8ea784c1,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='540826be4baf46dba9617b2fd47821a3',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:59:15Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:59:15Z,use_quota=True,user_id='d819e3f51368413c99691d1b45bbcfb1',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 16:01:33 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:54:35Z,deleted=False,deleted_at=None,display_description='',display_name='',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=75163b78-31b2-4676-b93b-542a8ea784c1,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='540826be4baf46dba9617b2fd47821a3',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:59:15Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:59:15Z,use_quota=True,user_id='d819e3f51368413c99691d1b45bbcfb1',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:54:35Z,deleted=False,deleted_at=None,display_description='',display_name='',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=75163b78-31b2-4676-b93b-542a8ea784c1,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='540826be4baf46dba9617b2fd47821a3',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:59:15Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:59:15Z,use_quota=True,user_id='d819e3f51368413c99691d1b45bbcfb1',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 16:01:33 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 16:01:33 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 16:01:33 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:01:33 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 16:01:33 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1237, in execute Apr 17 16:01:33 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_from_image(context, Apr 17 16:01:33 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/utils.py", line 670, in _wrapper Apr 17 16:01:33 user cinder-volume[84696]: ERROR cinder.volume.manager return r(f, *args, **kwargs) Apr 17 16:01:33 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/tenacity/__init__.py", line 423, in __call__ Apr 17 16:01:33 user cinder-volume[84696]: ERROR cinder.volume.manager do = self.iter(retry_state=retry_state) Apr 17 16:01:33 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/tenacity/__init__.py", line 360, in iter Apr 17 16:01:33 user cinder-volume[84696]: ERROR cinder.volume.manager return fut.result() Apr 17 16:01:33 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/lib/python3.10/concurrent/futures/_base.py", line 451, in result Apr 17 16:01:33 user cinder-volume[84696]: ERROR cinder.volume.manager return self.__get_result() Apr 17 16:01:33 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result Apr 17 16:01:33 user cinder-volume[84696]: ERROR cinder.volume.manager raise self._exception Apr 17 16:01:33 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/tenacity/__init__.py", line 426, in __call__ Apr 17 16:01:33 user cinder-volume[84696]: ERROR cinder.volume.manager result = fn(*args, **kwargs) Apr 17 16:01:33 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1125, in _create_from_image Apr 17 16:01:33 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_from_image_cache_or_download( Apr 17 16:01:33 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 944, in _create_from_image_cache_or_download Apr 17 16:01:33 user cinder-volume[84696]: ERROR cinder.volume.manager model_update, cloned = self._create_from_image_cache( Apr 17 16:01:33 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 843, in _create_from_image_cache Apr 17 16:01:33 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_from_source_volume( Apr 17 16:01:33 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 659, in _create_from_source_volume Apr 17 16:01:33 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self.driver.create_cloned_volume(volume, srcvol_ref) Apr 17 16:01:33 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 525, in create_cloned_volume Apr 17 16:01:33 user cinder-volume[84696]: ERROR cinder.volume.manager self._clone_object(vname, cvname) Apr 17 16:01:33 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 461, in _clone_object Apr 17 16:01:33 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_snapshot(oname, coname) Apr 17 16:01:33 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 639, in create_snapshot Apr 17 16:01:33 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201, 204): Apr 17 16:01:33 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 16:01:33 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 16:01:33 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (91febb26-0692-4dda-bb8a-1ac82a6371e0) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:01:33 user cinder-volume[84696]: WARNING cinder.volume.manager [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (91febb26-0692-4dda-bb8a-1ac82a6371e0) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:01:33 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (0ec43a89-b097-4912-bab1-f269151f787a) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:01:34 user cinder-volume[84696]: WARNING cinder.volume.manager [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (0ec43a89-b097-4912-bab1-f269151f787a) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:01:34 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (619e164b-5a0a-4ce1-8b8b-ebcf68c81389) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:01:34 user cinder-volume[84696]: WARNING cinder.volume.manager [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (619e164b-5a0a-4ce1-8b8b-ebcf68c81389) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:01:34 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (c46062ad-beb8-4f66-9e1e-d87f80e69bbb) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:01:34 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Updating volume 75163b78-31b2-4676-b93b-542a8ea784c1 with {'scheduled_at': datetime.datetime(2023, 4, 17, 16, 1, 34, 14452), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 16:01:34 user cinder-volume[84696]: DEBUG cinder.utils [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.521(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:01:34 user cinder-volume[84696]: DEBUG cinder.utils [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_fd313648-5fc1-4e6a-97a0-92ac803b36fd/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:01:34 user cinder-volume[84696]: DEBUG cinder.utils [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.188(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:01:34 user cinder-volume[84696]: DEBUG cinder.utils [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:01:34 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Volume 75163b78-31b2-4676-b93b-542a8ea784c1: re-scheduling SchedulerAPI.create_volume attempt 3 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 16:01:34 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Volume 75163b78-31b2-4676-b93b-542a8ea784c1: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 16:01:34 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] deleating volume v_75163b78-31b2-4676-b93b-542a8ea784c1 {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 16:01:34 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] get snapshots for volume v_75163b78-31b2-4676-b93b-542a8ea784c1 {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 16:01:34 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_75163b78-31b2-4676-b93b-542a8ea784c1/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:01:34 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_75163b78-31b2-4676-b93b-542a8ea784c1/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:01:34 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=24,cinder:UPDATE=8,cinder:INSERT=2 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:01:34 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=20,cinder:UPDATE=8,cinder:INSERT=4 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:01:36 user cinder-volume[84696]: DEBUG cinder.utils [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.943(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:01:36 user cinder-volume[84696]: DEBUG cinder.utils [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_75163b78-31b2-4676-b93b-542a8ea784c1/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:01:40 user cinder-volume[84696]: DEBUG cinder.utils [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.174(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:01:40 user cinder-volume[84696]: DEBUG cinder.utils [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_b29a7592-128a-41ee-a8e2-57780dee33e7/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:01:40 user cinder-volume[84696]: DEBUG cinder.utils [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 20.279(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:01:40 user cinder-volume[84696]: DEBUG cinder.utils [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:01:40 user cinder-volume[84696]: DEBUG cinder.utils [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.023(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:01:40 user cinder-volume[84696]: DEBUG cinder.utils [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_75163b78-31b2-4676-b93b-542a8ea784c1/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:01:46 user cinder-volume[84696]: DEBUG cinder.utils [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 40.460(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:01:46 user cinder-volume[84696]: DEBUG cinder.utils [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_bff6fdbb-a1fb-4b48-bca8-2f7efbc3f0ff/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:01:46 user cinder-volume[84696]: DEBUG cinder.utils [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 40.285(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:01:46 user cinder-volume[84696]: DEBUG cinder.utils [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:01:46 user cinder-volume[84696]: DEBUG cinder.utils [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.801(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:01:46 user cinder-volume[84696]: DEBUG cinder.utils [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_75163b78-31b2-4676-b93b-542a8ea784c1/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:01:49 user cinder-volume[84696]: DEBUG cinder.utils [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 22.026(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:01:49 user cinder-volume[84696]: DEBUG cinder.utils [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_b29a7592-128a-41ee-a8e2-57780dee33e7/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:01:52 user cinder-volume[84696]: DEBUG cinder.utils [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 40.158(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:01:52 user cinder-volume[84696]: DEBUG cinder.utils [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:01:52 user cinder-volume[84696]: DEBUG cinder.utils [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 40.494(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:01:52 user cinder-volume[84696]: DEBUG cinder.utils [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_fd313648-5fc1-4e6a-97a0-92ac803b36fd/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:01:52 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 40.520(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:01:52 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:01:52 user cinder-volume[84696]: DEBUG cinder.utils [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 73.075(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:01:52 user cinder-volume[84696]: DEBUG cinder.utils [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:01:56 user cinder-volume[84696]: DEBUG cinder.utils [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 71.574(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:01:56 user cinder-volume[84696]: DEBUG cinder.utils [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:01:56 user cinder-volume[84696]: DEBUG cinder.utils [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 22.020(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:01:56 user cinder-volume[84696]: DEBUG cinder.utils [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_75163b78-31b2-4676-b93b-542a8ea784c1/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:01:59 user cinder-volume[84696]: DEBUG cinder.utils [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.345(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:01:59 user cinder-volume[84696]: DEBUG cinder.utils [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:02:04 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=3,cinder:UPDATE=1 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:02:05 user cinder-volume[84696]: DEBUG cinder.utils [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 140.110(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:02:05 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:02:05 user cinder-volume[84696]: INFO cinder.message.api [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Creating message record for request_id = req-8969563c-70db-45ae-8d53-74d80da7efc1 Apr 17 16:02:05 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 16:02:05 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=7,cinder:UPDATE=3,cinder:INSERT=1 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:02:05 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (42fe441f-f4b2-4f07-bfd6-ab8c0d9aebac) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 16:02:05 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 16:02:05 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:59:46Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesListAdminTestJSON-Volume-237491364',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=26014b4e-ec84-4011-a69f-ed42c3424a24,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='73222838c1d4451281d075e1f9c092b3',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:59:46Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:59:46Z,use_quota=True,user_id='6a809fbfe4264147a2716c1cc5914a28',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 16:02:05 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:59:46Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesListAdminTestJSON-Volume-237491364',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=26014b4e-ec84-4011-a69f-ed42c3424a24,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='73222838c1d4451281d075e1f9c092b3',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:59:46Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:59:46Z,use_quota=True,user_id='6a809fbfe4264147a2716c1cc5914a28',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(26014b4e-ec84-4011-a69f-ed42c3424a24),volume_id=26014b4e-ec84-4011-a69f-ed42c3424a24,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': '26014b4e-ec84-4011-a69f-ed42c3424a24', 'volume_name': 'volume-26014b4e-ec84-4011-a69f-ed42c3424a24', 'volume_size': 1}} Apr 17 16:02:05 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'retry': {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']}, 'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': '26014b4e-ec84-4011-a69f-ed42c3424a24', 'volume': {'id': '26014b4e-ec84-4011-a69f-ed42c3424a24', '_name_id': None, 'ec2_id': None, 'user_id': '6a809fbfe4264147a2716c1cc5914a28', 'project_id': '73222838c1d4451281d075e1f9c092b3', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-VolumesListAdminTestJSON-Volume-237491364', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T15:59:45.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-26014b4e-ec84-4011-a69f-ed42c3424a24', 'name_id': '26014b4e-ec84-4011-a69f-ed42c3424a24', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumesListAdminTestJSON-Volume-237491364', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '73222838c1d4451281d075e1f9c092b3', 'qos_specs': None, 'reservations': ['aae7b3c8-9b21-42c2-b7d1-365a54453839', '40630276-bfc3-4923-b711-7d6a45c12c52', '251f733a-f47a-4155-8d77-c0c8abe4451b', '7007e6ad-6925-4dc6-bc99-8f9d477393a9'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': '6a809fbfe4264147a2716c1cc5914a28', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumesListAdminTestJSON-Volume-237491364', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '73222838c1d4451281d075e1f9c092b3', 'qos_specs': None, 'reservations': ['aae7b3c8-9b21-42c2-b7d1-365a54453839', '40630276-bfc3-4923-b711-7d6a45c12c52', '251f733a-f47a-4155-8d77-c0c8abe4451b', '7007e6ad-6925-4dc6-bc99-8f9d477393a9'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': '6a809fbfe4264147a2716c1cc5914a28', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': '6a809fbfe4264147a2716c1cc5914a28', 'metadata': {}, 'qos_specs': None}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(26014b4e-ec84-4011-a69f-ed42c3424a24),volume_id=26014b4e-ec84-4011-a69f-ed42c3424a24,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:59:46Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesListAdminTestJSON-Volume-237491364',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=26014b4e-ec84-4011-a69f-ed42c3424a24,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='73222838c1d4451281d075e1f9c092b3',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:59:46Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:59:46Z,use_quota=True,user_id='6a809fbfe4264147a2716c1cc5914a28',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 16:02:05 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:59:46Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesListAdminTestJSON-Volume-237491364',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=26014b4e-ec84-4011-a69f-ed42c3424a24,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='73222838c1d4451281d075e1f9c092b3',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:59:46Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:59:46Z,use_quota=True,user_id='6a809fbfe4264147a2716c1cc5914a28',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:59:46Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesListAdminTestJSON-Volume-237491364',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=26014b4e-ec84-4011-a69f-ed42c3424a24,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='73222838c1d4451281d075e1f9c092b3',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:59:46Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:59:46Z,use_quota=True,user_id='6a809fbfe4264147a2716c1cc5914a28',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 16:02:05 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 16:02:05 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 16:02:05 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:02:05 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 16:02:05 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:02:05 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 16:02:05 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:02:05 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 16:02:05 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:02:05 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 16:02:05 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:02:05 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 16:02:05 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:02:05 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 16:02:05 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:02:05 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 16:02:05 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:02:05 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:02:05 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 16:02:05 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 16:02:05 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (42fe441f-f4b2-4f07-bfd6-ab8c0d9aebac) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:02:05 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (42fe441f-f4b2-4f07-bfd6-ab8c0d9aebac) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:02:05 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (efdc759c-5d32-4b0c-870a-eb83a7367157) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:02:05 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (efdc759c-5d32-4b0c-870a-eb83a7367157) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:02:05 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (e3ba53fc-7705-49ab-876c-a0a6c41eac57) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:02:05 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (e3ba53fc-7705-49ab-876c-a0a6c41eac57) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:02:05 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (0017c1da-bbee-4554-b557-920615d6357f) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:02:05 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Updating volume 26014b4e-ec84-4011-a69f-ed42c3424a24 with {'scheduled_at': datetime.datetime(2023, 4, 17, 16, 2, 5, 985151), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 16:02:05 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Volume 26014b4e-ec84-4011-a69f-ed42c3424a24: re-scheduling SchedulerAPI.create_volume attempt 1 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 16:02:06 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Volume 26014b4e-ec84-4011-a69f-ed42c3424a24: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 16:02:06 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] deleating volume v_26014b4e-ec84-4011-a69f-ed42c3424a24 {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 16:02:06 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] get snapshots for volume v_26014b4e-ec84-4011-a69f-ed42c3424a24 {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 16:02:06 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_26014b4e-ec84-4011-a69f-ed42c3424a24/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:02:06 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_26014b4e-ec84-4011-a69f-ed42c3424a24/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:02:06 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Volume reschedule parameters: True retry: {'num_attempts': 2, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 16:02:06 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Flow 'volume_create_manager' (e561083c-5024-4447-8401-bbcd3d4cddcc) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 16:02:06 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (ae4d6c1a-cf85-41e0-85a5-b0980a5d748c) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:02:06 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (ae4d6c1a-cf85-41e0-85a5-b0980a5d748c) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:59:46Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesListAdminTestJSON-Volume-237491364',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=26014b4e-ec84-4011-a69f-ed42c3424a24,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='73222838c1d4451281d075e1f9c092b3',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:02:06Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:02:06Z,use_quota=True,user_id='6a809fbfe4264147a2716c1cc5914a28',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:02:06 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (a35054f0-be95-43c7-93ec-8f4d0735bc3d) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:02:06 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (a35054f0-be95-43c7-93ec-8f4d0735bc3d) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:02:06 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (1ed2b502-ec74-45a7-b8a0-67223ec834d7) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:02:06 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (1ed2b502-ec74-45a7-b8a0-67223ec834d7) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': '26014b4e-ec84-4011-a69f-ed42c3424a24', 'volume_name': 'volume-26014b4e-ec84-4011-a69f-ed42c3424a24', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:02:06 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (78a7ba9d-f362-476e-975d-9a59c358177a) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:02:06 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (78a7ba9d-f362-476e-975d-9a59c358177a) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:02:06 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (eccb938b-be3d-4287-89bf-2eb5b91b4e13) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:02:06 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Volume 26014b4e-ec84-4011-a69f-ed42c3424a24: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-26014b4e-ec84-4011-a69f-ed42c3424a24', 'volume_size': 1} Apr 17 16:02:06 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] creating volume v_26014b4e-ec84-4011-a69f-ed42c3424a24. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 16:02:06 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] create volume {'name': 'v_26014b4e-ec84-4011-a69f-ed42c3424a24', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 16:02:06 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:02:06 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:02:08 user cinder-volume[84696]: DEBUG cinder.utils [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 41.098(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:02:08 user cinder-volume[84696]: DEBUG cinder.utils [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_b29a7592-128a-41ee-a8e2-57780dee33e7/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:02:08 user cinder-volume[84696]: DEBUG cinder.utils [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.980(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:02:08 user cinder-volume[84696]: DEBUG cinder.utils [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_26014b4e-ec84-4011-a69f-ed42c3424a24/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:02:08 user cinder-volume[84696]: DEBUG cinder.utils [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.803(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:02:08 user cinder-volume[84696]: DEBUG cinder.utils [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:02:12 user cinder-volume[84696]: DEBUG cinder.utils [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.871(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:02:12 user cinder-volume[84696]: DEBUG cinder.utils [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:02:12 user cinder-volume[84696]: DEBUG cinder.utils [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.052(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:02:12 user cinder-volume[84696]: DEBUG cinder.utils [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_26014b4e-ec84-4011-a69f-ed42c3424a24/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:02:15 user cinder-volume[84696]: DEBUG cinder.utils [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 41.087(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:02:15 user cinder-volume[84696]: DEBUG cinder.utils [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_75163b78-31b2-4676-b93b-542a8ea784c1/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:02:15 user cinder-volume[84696]: DEBUG cinder.utils [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 139.876(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:02:15 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:02:15 user cinder-volume[84696]: INFO cinder.message.api [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Creating message record for request_id = req-5a40883b-32f9-400a-a3f7-129f4d4677bb Apr 17 16:02:15 user cinder-volume[84696]: DEBUG cinder.utils [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 140.049(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:02:15 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_87514055-5a0f-459f-b10b-438d7a13d39d/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:02:15 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (9db71fee-70ac-4ddc-bef2-8b370efd6663) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 16:02:15 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (6c017e80-6a0a-40ce-9c74-78f17c2dc622) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:02:15 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (6c017e80-6a0a-40ce-9c74-78f17c2dc622) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:02:15 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Flow 'volume_create_manager' (7524b499-f0fc-474f-ab3d-77e348250070) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 16:02:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 16:02:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 16:02:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 16:02:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 16:02:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 16:02:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 16:02:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 16:02:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 16:02:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 16:02:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 16:02:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 16:02:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 16:02:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 16:02:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 16:02:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 16:02:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 16:02:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 16:02:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 16:02:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 16:02:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 16:02:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 16:02:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 16:02:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 16:02:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 16:02:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:02:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 16:02:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:02:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 16:02:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:02:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 16:02:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:02:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 16:02:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:02:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 16:02:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:02:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 16:02:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:02:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 16:02:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:02:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:02:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 16:02:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 16:02:15 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 16:02:15 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (ba0a9923-7993-4363-a092-9649d52e75ec) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 16:02:15 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 16:02:15 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:55:17Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-AttachVolumeNegativeTest-volume-1944200504',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=87514055-5a0f-459f-b10b-438d7a13d39d,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='8586cc10142d4743874f43cdb0f37aba',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:59:55Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:59:55Z,use_quota=True,user_id='9fdc7c80b4654692a52c8a4c4149324f',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 16:02:15 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:55:17Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-AttachVolumeNegativeTest-volume-1944200504',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=87514055-5a0f-459f-b10b-438d7a13d39d,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='8586cc10142d4743874f43cdb0f37aba',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:59:55Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:59:55Z,use_quota=True,user_id='9fdc7c80b4654692a52c8a4c4149324f',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(87514055-5a0f-459f-b10b-438d7a13d39d),volume_id=87514055-5a0f-459f-b10b-438d7a13d39d,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': '87514055-5a0f-459f-b10b-438d7a13d39d', 'volume_name': 'volume-87514055-5a0f-459f-b10b-438d7a13d39d', 'volume_size': 1}} Apr 17 16:02:15 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': '87514055-5a0f-459f-b10b-438d7a13d39d', 'volume': {'id': '87514055-5a0f-459f-b10b-438d7a13d39d', '_name_id': None, 'ec2_id': None, 'user_id': '9fdc7c80b4654692a52c8a4c4149324f', 'project_id': '8586cc10142d4743874f43cdb0f37aba', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-AttachVolumeNegativeTest-volume-1944200504', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'admin_metadata': {}, 'glance_metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T15:55:17.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-87514055-5a0f-459f-b10b-438d7a13d39d', 'name_id': '87514055-5a0f-459f-b10b-438d7a13d39d', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-AttachVolumeNegativeTest-volume-1944200504', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '8586cc10142d4743874f43cdb0f37aba', 'qos_specs': None, 'reservations': ['1c53c030-381a-4511-90a6-073111ef116f', '7bbd5e8b-d708-4476-a281-0ce37fa87819', '438a4611-10fd-4bd4-a60e-52564e4a0b82', 'a28101d3-8126-4720-b09f-75c7afd4c0dd'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': '9fdc7c80b4654692a52c8a4c4149324f', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-AttachVolumeNegativeTest-volume-1944200504', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '8586cc10142d4743874f43cdb0f37aba', 'qos_specs': None, 'reservations': ['1c53c030-381a-4511-90a6-073111ef116f', '7bbd5e8b-d708-4476-a281-0ce37fa87819', '438a4611-10fd-4bd4-a60e-52564e4a0b82', 'a28101d3-8126-4720-b09f-75c7afd4c0dd'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': '9fdc7c80b4654692a52c8a4c4149324f', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': '9fdc7c80b4654692a52c8a4c4149324f', 'metadata': {}, 'qos_specs': None, 'retry': {'num_attempts': 3, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']}}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(87514055-5a0f-459f-b10b-438d7a13d39d),volume_id=87514055-5a0f-459f-b10b-438d7a13d39d,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:55:17Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-AttachVolumeNegativeTest-volume-1944200504',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=87514055-5a0f-459f-b10b-438d7a13d39d,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='8586cc10142d4743874f43cdb0f37aba',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:59:55Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:59:55Z,use_quota=True,user_id='9fdc7c80b4654692a52c8a4c4149324f',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 16:02:15 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:55:17Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-AttachVolumeNegativeTest-volume-1944200504',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=87514055-5a0f-459f-b10b-438d7a13d39d,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='8586cc10142d4743874f43cdb0f37aba',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:59:55Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:59:55Z,use_quota=True,user_id='9fdc7c80b4654692a52c8a4c4149324f',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:55:17Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-AttachVolumeNegativeTest-volume-1944200504',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=87514055-5a0f-459f-b10b-438d7a13d39d,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='8586cc10142d4743874f43cdb0f37aba',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:59:55Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:59:55Z,use_quota=True,user_id='9fdc7c80b4654692a52c8a4c4149324f',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 16:02:15 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 16:02:15 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 16:02:15 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:02:15 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 16:02:15 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:02:15 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 16:02:15 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:02:15 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 16:02:15 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:02:15 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 16:02:15 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:02:15 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 16:02:15 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:02:15 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 16:02:15 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:02:15 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 16:02:15 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:02:15 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:02:15 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 16:02:15 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 16:02:15 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (ba0a9923-7993-4363-a092-9649d52e75ec) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:02:15 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (ba0a9923-7993-4363-a092-9649d52e75ec) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:02:15 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (7db4611c-4af8-4a0e-84d0-58c0123e236c) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:02:15 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (7db4611c-4af8-4a0e-84d0-58c0123e236c) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:02:15 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (eccf2cdf-735f-48be-901e-f224f342f7c5) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:02:15 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (eccf2cdf-735f-48be-901e-f224f342f7c5) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:02:15 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (43048501-f203-43eb-8c24-422c165f661a) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:02:15 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Updating volume 87514055-5a0f-459f-b10b-438d7a13d39d with {'scheduled_at': datetime.datetime(2023, 4, 17, 16, 2, 15, 217984), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 16:02:15 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Volume 87514055-5a0f-459f-b10b-438d7a13d39d: re-scheduling SchedulerAPI.create_volume attempt 3 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 16:02:15 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Volume 87514055-5a0f-459f-b10b-438d7a13d39d: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 16:02:15 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] deleating volume v_87514055-5a0f-459f-b10b-438d7a13d39d {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 16:02:15 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] get snapshots for volume v_87514055-5a0f-459f-b10b-438d7a13d39d {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 16:02:15 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_87514055-5a0f-459f-b10b-438d7a13d39d/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:02:15 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_87514055-5a0f-459f-b10b-438d7a13d39d/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:02:18 user cinder-volume[84696]: DEBUG cinder.utils [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.954(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:02:18 user cinder-volume[84696]: DEBUG cinder.utils [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_87514055-5a0f-459f-b10b-438d7a13d39d/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:02:18 user cinder-volume[84696]: DEBUG cinder.utils [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.020(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:02:18 user cinder-volume[84696]: DEBUG cinder.utils [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:02:18 user cinder-volume[84696]: DEBUG cinder.utils [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 139.712(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:02:18 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:02:18 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (ff78fa8d-6ac1-4704-8c0c-3b932fe191d3) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 16:02:18 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 16:02:18 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:55:22Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-tempest.common.compute-volume-826817831',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=14aa38ea-7cbb-48f7-a9aa-ffcc7b66faa0,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='cb5f6b5ed2fc43b69c47e1fc18646545',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:59:58Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:59:58Z,use_quota=True,user_id='638a4f535e9740fc9d810639a5eb24d7',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 16:02:18 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:55:22Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-tempest.common.compute-volume-826817831',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=14aa38ea-7cbb-48f7-a9aa-ffcc7b66faa0,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='cb5f6b5ed2fc43b69c47e1fc18646545',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:59:58Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:59:58Z,use_quota=True,user_id='638a4f535e9740fc9d810639a5eb24d7',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=b3f0dec1-7758-44a8-82e0-76ada8837428,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(14aa38ea-7cbb-48f7-a9aa-ffcc7b66faa0),volume_id=14aa38ea-7cbb-48f7-a9aa-ffcc7b66faa0,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'image', 'volume_id': '14aa38ea-7cbb-48f7-a9aa-ffcc7b66faa0', 'volume_name': 'volume-14aa38ea-7cbb-48f7-a9aa-ffcc7b66faa0', 'volume_size': 1, 'image_id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'image_location': (None, None), 'image_meta': {'name': 'cirros-0.5.2-x86_64-disk', 'disk_format': 'qcow2', 'container_format': 'bare', 'visibility': 'public', 'size': 16300544, 'virtual_size': 117440512, 'status': 'active', 'checksum': 'b874c39491a2377b8490f5f1e89761a4', 'protected': False, 'min_ram': 0, 'min_disk': 0, 'owner': 'a43c211b4ed542538799349489b1d884', 'os_hidden': False, 'os_hash_algo': 'sha512', 'os_hash_value': '6b813aa46bb90b4da216a4d19376593fa3f4fc7e617f03a92b7fe11e9a3981cbe8f0959dbebe36225e5f53dc4492341a4863cac4ed1ee0909f3fc78ef9c3e869', 'id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'created_at': datetime.datetime(2023, 4, 17, 15, 19, 53, tzinfo=datetime.timezone.utc), 'updated_at': datetime.datetime(2023, 4, 17, 15, 19, 55, tzinfo=datetime.timezone.utc), 'tags': [], 'file': '/v2/images/b3f0dec1-7758-44a8-82e0-76ada8837428/file', 'properties': {'hw_rng_model': 'virtio', 'owner_specified.openstack.md5': '', 'owner_specified.openstack.object': 'images/cirros-0.5.2-x86_64-disk', 'owner_specified.openstack.sha256': ''}}, 'image_service': }} Apr 17 16:02:18 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'snapshot_id': None, 'source_volid': None, 'volume_id': '14aa38ea-7cbb-48f7-a9aa-ffcc7b66faa0', 'volume': {'id': '14aa38ea-7cbb-48f7-a9aa-ffcc7b66faa0', '_name_id': None, 'ec2_id': None, 'user_id': '638a4f535e9740fc9d810639a5eb24d7', 'project_id': 'cb5f6b5ed2fc43b69c47e1fc18646545', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-tempest.common.compute-volume-826817831', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'admin_metadata': {}, 'glance_metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T15:55:22.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-14aa38ea-7cbb-48f7-a9aa-ffcc7b66faa0', 'name_id': '14aa38ea-7cbb-48f7-a9aa-ffcc7b66faa0', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-tempest.common.compute-volume-826817831', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': 'cb5f6b5ed2fc43b69c47e1fc18646545', 'qos_specs': None, 'reservations': ['c3b21a3e-ae73-41f9-8d32-500bd32a5696', '477b1752-2fec-4e18-835a-e7ca57617158', '2e94e8e6-37cc-46c3-b87c-ae7717df65c2', 'a5c6c20e-c7ee-491f-b333-35f64711994a'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': '638a4f535e9740fc9d810639a5eb24d7', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-tempest.common.compute-volume-826817831', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': 'cb5f6b5ed2fc43b69c47e1fc18646545', 'qos_specs': None, 'reservations': ['c3b21a3e-ae73-41f9-8d32-500bd32a5696', '477b1752-2fec-4e18-835a-e7ca57617158', '2e94e8e6-37cc-46c3-b87c-ae7717df65c2', 'a5c6c20e-c7ee-491f-b333-35f64711994a'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': '638a4f535e9740fc9d810639a5eb24d7', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': '638a4f535e9740fc9d810639a5eb24d7', 'metadata': {}, 'qos_specs': None, 'retry': {'num_attempts': 3, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']}}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=b3f0dec1-7758-44a8-82e0-76ada8837428,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(14aa38ea-7cbb-48f7-a9aa-ffcc7b66faa0),volume_id=14aa38ea-7cbb-48f7-a9aa-ffcc7b66faa0,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:55:22Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-tempest.common.compute-volume-826817831',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=14aa38ea-7cbb-48f7-a9aa-ffcc7b66faa0,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='cb5f6b5ed2fc43b69c47e1fc18646545',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:59:58Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:59:58Z,use_quota=True,user_id='638a4f535e9740fc9d810639a5eb24d7',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 16:02:18 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:55:22Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-tempest.common.compute-volume-826817831',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=14aa38ea-7cbb-48f7-a9aa-ffcc7b66faa0,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='cb5f6b5ed2fc43b69c47e1fc18646545',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:59:58Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:59:58Z,use_quota=True,user_id='638a4f535e9740fc9d810639a5eb24d7',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:55:22Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-tempest.common.compute-volume-826817831',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=14aa38ea-7cbb-48f7-a9aa-ffcc7b66faa0,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='cb5f6b5ed2fc43b69c47e1fc18646545',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:59:58Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T15:59:58Z,use_quota=True,user_id='638a4f535e9740fc9d810639a5eb24d7',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 16:02:18 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 16:02:18 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 16:02:18 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:02:18 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 16:02:18 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1237, in execute Apr 17 16:02:18 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_from_image(context, Apr 17 16:02:18 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/utils.py", line 670, in _wrapper Apr 17 16:02:18 user cinder-volume[84696]: ERROR cinder.volume.manager return r(f, *args, **kwargs) Apr 17 16:02:18 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/tenacity/__init__.py", line 423, in __call__ Apr 17 16:02:18 user cinder-volume[84696]: ERROR cinder.volume.manager do = self.iter(retry_state=retry_state) Apr 17 16:02:18 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/tenacity/__init__.py", line 360, in iter Apr 17 16:02:18 user cinder-volume[84696]: ERROR cinder.volume.manager return fut.result() Apr 17 16:02:18 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/lib/python3.10/concurrent/futures/_base.py", line 451, in result Apr 17 16:02:18 user cinder-volume[84696]: ERROR cinder.volume.manager return self.__get_result() Apr 17 16:02:18 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result Apr 17 16:02:18 user cinder-volume[84696]: ERROR cinder.volume.manager raise self._exception Apr 17 16:02:18 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/tenacity/__init__.py", line 426, in __call__ Apr 17 16:02:18 user cinder-volume[84696]: ERROR cinder.volume.manager result = fn(*args, **kwargs) Apr 17 16:02:18 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1125, in _create_from_image Apr 17 16:02:18 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_from_image_cache_or_download( Apr 17 16:02:18 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 944, in _create_from_image_cache_or_download Apr 17 16:02:18 user cinder-volume[84696]: ERROR cinder.volume.manager model_update, cloned = self._create_from_image_cache( Apr 17 16:02:18 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 843, in _create_from_image_cache Apr 17 16:02:18 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_from_source_volume( Apr 17 16:02:18 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 659, in _create_from_source_volume Apr 17 16:02:18 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self.driver.create_cloned_volume(volume, srcvol_ref) Apr 17 16:02:18 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 525, in create_cloned_volume Apr 17 16:02:18 user cinder-volume[84696]: ERROR cinder.volume.manager self._clone_object(vname, cvname) Apr 17 16:02:18 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 461, in _clone_object Apr 17 16:02:18 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_snapshot(oname, coname) Apr 17 16:02:18 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 639, in create_snapshot Apr 17 16:02:18 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201, 204): Apr 17 16:02:18 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 16:02:18 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 16:02:18 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (ff78fa8d-6ac1-4704-8c0c-3b932fe191d3) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:02:18 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (ff78fa8d-6ac1-4704-8c0c-3b932fe191d3) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:02:18 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (cd0c89a4-d7ed-438c-99d0-156bd9bd99bd) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:02:18 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (cd0c89a4-d7ed-438c-99d0-156bd9bd99bd) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:02:18 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (1cbc4a98-22d3-4eab-9959-b94cc8182263) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:02:18 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (1cbc4a98-22d3-4eab-9959-b94cc8182263) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:02:18 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (5faf14b4-e033-41b8-84fb-8b4d162dd650) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:02:18 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Updating volume 14aa38ea-7cbb-48f7-a9aa-ffcc7b66faa0 with {'scheduled_at': datetime.datetime(2023, 4, 17, 16, 2, 18, 267336), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 16:02:18 user cinder-volume[84696]: DEBUG cinder.utils [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.265(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:02:18 user cinder-volume[84696]: DEBUG cinder.utils [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_26014b4e-ec84-4011-a69f-ed42c3424a24/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:02:18 user cinder-volume[84696]: DEBUG cinder.utils [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 140.121(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:02:18 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_14aa38ea-7cbb-48f7-a9aa-ffcc7b66faa0/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:02:18 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (7ad071b2-5c5a-4e67-a1e3-1c6ca780c749) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 16:02:18 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (f17af5f5-9351-435c-993f-6146b54ea7ec) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:02:18 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (f17af5f5-9351-435c-993f-6146b54ea7ec) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:02:18 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Flow 'volume_create_manager' (ee047975-88bc-49e6-ad52-dc7b144acb87) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 16:02:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 16:02:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 16:02:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 16:02:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 16:02:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 16:02:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 16:02:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 16:02:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 16:02:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 16:02:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 16:02:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 16:02:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 16:02:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 16:02:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 16:02:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 16:02:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 16:02:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 16:02:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 16:02:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 16:02:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 16:02:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 16:02:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 16:02:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 16:02:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 16:02:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:02:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 16:02:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1237, in execute Apr 17 16:02:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_from_image(context, Apr 17 16:02:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/utils.py", line 670, in _wrapper Apr 17 16:02:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return r(f, *args, **kwargs) Apr 17 16:02:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/tenacity/__init__.py", line 423, in __call__ Apr 17 16:02:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server do = self.iter(retry_state=retry_state) Apr 17 16:02:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/tenacity/__init__.py", line 360, in iter Apr 17 16:02:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return fut.result() Apr 17 16:02:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/lib/python3.10/concurrent/futures/_base.py", line 451, in result Apr 17 16:02:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self.__get_result() Apr 17 16:02:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result Apr 17 16:02:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self._exception Apr 17 16:02:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/tenacity/__init__.py", line 426, in __call__ Apr 17 16:02:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = fn(*args, **kwargs) Apr 17 16:02:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1125, in _create_from_image Apr 17 16:02:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_from_image_cache_or_download( Apr 17 16:02:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 944, in _create_from_image_cache_or_download Apr 17 16:02:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update, cloned = self._create_from_image_cache( Apr 17 16:02:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 843, in _create_from_image_cache Apr 17 16:02:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_from_source_volume( Apr 17 16:02:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 659, in _create_from_source_volume Apr 17 16:02:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self.driver.create_cloned_volume(volume, srcvol_ref) Apr 17 16:02:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 525, in create_cloned_volume Apr 17 16:02:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self._clone_object(vname, cvname) Apr 17 16:02:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 461, in _clone_object Apr 17 16:02:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_snapshot(oname, coname) Apr 17 16:02:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 639, in create_snapshot Apr 17 16:02:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201, 204): Apr 17 16:02:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 16:02:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 16:02:18 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Volume 14aa38ea-7cbb-48f7-a9aa-ffcc7b66faa0: re-scheduling SchedulerAPI.create_volume attempt 3 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 16:02:18 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Volume 14aa38ea-7cbb-48f7-a9aa-ffcc7b66faa0: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 16:02:18 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] deleating volume v_14aa38ea-7cbb-48f7-a9aa-ffcc7b66faa0 {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 16:02:18 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] get snapshots for volume v_14aa38ea-7cbb-48f7-a9aa-ffcc7b66faa0 {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 16:02:18 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_14aa38ea-7cbb-48f7-a9aa-ffcc7b66faa0/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:02:18 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_14aa38ea-7cbb-48f7-a9aa-ffcc7b66faa0/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:02:21 user cinder-volume[84696]: DEBUG cinder.utils [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.947(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:02:21 user cinder-volume[84696]: DEBUG cinder.utils [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_14aa38ea-7cbb-48f7-a9aa-ffcc7b66faa0/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:02:21 user cinder-volume[84696]: DEBUG cinder.utils [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.033(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:02:21 user cinder-volume[84696]: DEBUG cinder.utils [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_87514055-5a0f-459f-b10b-438d7a13d39d/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:02:21 user cinder-volume[84696]: DEBUG cinder.utils [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 74.725(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:02:21 user cinder-volume[84696]: DEBUG cinder.utils [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:02:21 user cinder-volume[84696]: DEBUG cinder.utils [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 74.902(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:02:21 user cinder-volume[84696]: DEBUG cinder.utils [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_bff6fdbb-a1fb-4b48-bca8-2f7efbc3f0ff/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:02:21 user cinder-volume[84696]: DEBUG cinder.utils [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 140.045(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:02:21 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_45e5cc09-6c97-438a-a6a2-c89e7d725ce5/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:02:21 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (458dc8a3-8d60-43b2-a4a5-5d529293df10) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 16:02:21 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (531f262f-7a09-49a6-b5f9-b8c622dd9382) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:02:21 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (531f262f-7a09-49a6-b5f9-b8c622dd9382) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:02:21 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Flow 'volume_create_manager' (748603ab-e9fd-413b-9ecc-babf53ee3f3b) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 16:02:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-99373bb6-c6e6-4fce-8296-10f7a0995ca3 tempest-TestMinimumBasicScenario-1774469422 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 16:02:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 16:02:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 16:02:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 16:02:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 16:02:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 16:02:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 16:02:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 16:02:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 16:02:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 16:02:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 16:02:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 16:02:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 16:02:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 16:02:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 16:02:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 16:02:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 16:02:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 16:02:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 16:02:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 16:02:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 16:02:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 16:02:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 16:02:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 16:02:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:02:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 16:02:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:02:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 16:02:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:02:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 16:02:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:02:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 16:02:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:02:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 16:02:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:02:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 16:02:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:02:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 16:02:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:02:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:02:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 16:02:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 16:02:24 user cinder-volume[84696]: DEBUG cinder.utils [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 139.893(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:02:24 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:02:24 user cinder-volume[84696]: INFO cinder.message.api [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Creating message record for request_id = req-dc8764a6-903f-44ac-86a4-76a8d1914785 Apr 17 16:02:24 user cinder-volume[84696]: DEBUG cinder.utils [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 140.086(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:02:24 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_9a399a62-6a8f-4b6d-8ecf-8dcdf57e5fbb/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:02:24 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (ddaa5868-0c7e-4fad-ac69-846477b8c783) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 16:02:24 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (3c3f1053-b1fc-43ad-88d3-ca0a3dff442a) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:02:24 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (3c3f1053-b1fc-43ad-88d3-ca0a3dff442a) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:02:24 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Flow 'volume_create_manager' (b5b426a6-1f65-4716-a4a7-8298feedc9da) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 16:02:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 16:02:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 16:02:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 16:02:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 16:02:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 16:02:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 16:02:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 16:02:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 16:02:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 16:02:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 16:02:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 16:02:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 16:02:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 16:02:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 16:02:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 16:02:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 16:02:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 16:02:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 16:02:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 16:02:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 16:02:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 16:02:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 16:02:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 16:02:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 16:02:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:02:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 16:02:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:02:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 16:02:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:02:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 16:02:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:02:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 16:02:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:02:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 16:02:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:02:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 16:02:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:02:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 16:02:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:02:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:02:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 16:02:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 16:02:24 user cinder-volume[84696]: DEBUG cinder.utils [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.044(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:02:24 user cinder-volume[84696]: DEBUG cinder.utils [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_14aa38ea-7cbb-48f7-a9aa-ffcc7b66faa0/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:02:24 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 16:02:24 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (ee2be40e-352e-4b95-8bf2-4d4659185323) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 16:02:24 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 16:02:24 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:57:47Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumeRevertTests-Volume-1536424582',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=9a399a62-6a8f-4b6d-8ecf-8dcdf57e5fbb,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='9c755d3b5d8a4d9c9918b6c4744f510e',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:00:04Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:00:04Z,use_quota=True,user_id='e362cfe4a31c444b9ab44a5d60f37ad9',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 16:02:24 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:57:47Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumeRevertTests-Volume-1536424582',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=9a399a62-6a8f-4b6d-8ecf-8dcdf57e5fbb,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='9c755d3b5d8a4d9c9918b6c4744f510e',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:00:04Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:00:04Z,use_quota=True,user_id='e362cfe4a31c444b9ab44a5d60f37ad9',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(9a399a62-6a8f-4b6d-8ecf-8dcdf57e5fbb),volume_id=9a399a62-6a8f-4b6d-8ecf-8dcdf57e5fbb,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': '9a399a62-6a8f-4b6d-8ecf-8dcdf57e5fbb', 'volume_name': 'volume-9a399a62-6a8f-4b6d-8ecf-8dcdf57e5fbb', 'volume_size': 1}} Apr 17 16:02:24 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': '9a399a62-6a8f-4b6d-8ecf-8dcdf57e5fbb', 'volume': {'id': '9a399a62-6a8f-4b6d-8ecf-8dcdf57e5fbb', '_name_id': None, 'ec2_id': None, 'user_id': 'e362cfe4a31c444b9ab44a5d60f37ad9', 'project_id': '9c755d3b5d8a4d9c9918b6c4744f510e', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-VolumeRevertTests-Volume-1536424582', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'admin_metadata': {}, 'glance_metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T15:57:47.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-9a399a62-6a8f-4b6d-8ecf-8dcdf57e5fbb', 'name_id': '9a399a62-6a8f-4b6d-8ecf-8dcdf57e5fbb', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumeRevertTests-Volume-1536424582', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '9c755d3b5d8a4d9c9918b6c4744f510e', 'qos_specs': None, 'reservations': ['50c841a2-120d-4b16-8e0c-160f7b70632b', '75012498-38d8-444b-aeb0-ece3429535fa', '3472950b-da65-4163-bf5e-5efec626310b', 'b7635850-6316-4d2c-8d29-9d8061c89d14'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'e362cfe4a31c444b9ab44a5d60f37ad9', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumeRevertTests-Volume-1536424582', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '9c755d3b5d8a4d9c9918b6c4744f510e', 'qos_specs': None, 'reservations': ['50c841a2-120d-4b16-8e0c-160f7b70632b', '75012498-38d8-444b-aeb0-ece3429535fa', '3472950b-da65-4163-bf5e-5efec626310b', 'b7635850-6316-4d2c-8d29-9d8061c89d14'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'e362cfe4a31c444b9ab44a5d60f37ad9', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': 'e362cfe4a31c444b9ab44a5d60f37ad9', 'metadata': {}, 'qos_specs': None, 'retry': {'num_attempts': 2, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']}}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(9a399a62-6a8f-4b6d-8ecf-8dcdf57e5fbb),volume_id=9a399a62-6a8f-4b6d-8ecf-8dcdf57e5fbb,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:57:47Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumeRevertTests-Volume-1536424582',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=9a399a62-6a8f-4b6d-8ecf-8dcdf57e5fbb,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='9c755d3b5d8a4d9c9918b6c4744f510e',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:00:04Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:00:04Z,use_quota=True,user_id='e362cfe4a31c444b9ab44a5d60f37ad9',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 16:02:24 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:57:47Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumeRevertTests-Volume-1536424582',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=9a399a62-6a8f-4b6d-8ecf-8dcdf57e5fbb,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='9c755d3b5d8a4d9c9918b6c4744f510e',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:00:04Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:00:04Z,use_quota=True,user_id='e362cfe4a31c444b9ab44a5d60f37ad9',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:57:47Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumeRevertTests-Volume-1536424582',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=9a399a62-6a8f-4b6d-8ecf-8dcdf57e5fbb,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='9c755d3b5d8a4d9c9918b6c4744f510e',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:00:04Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:00:04Z,use_quota=True,user_id='e362cfe4a31c444b9ab44a5d60f37ad9',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 16:02:24 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 16:02:24 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 16:02:24 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:02:24 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 16:02:24 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:02:24 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 16:02:24 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:02:24 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 16:02:24 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:02:24 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 16:02:24 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:02:24 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 16:02:24 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:02:24 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 16:02:24 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:02:24 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 16:02:24 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:02:24 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:02:24 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 16:02:24 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 16:02:24 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (ee2be40e-352e-4b95-8bf2-4d4659185323) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:02:24 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (ee2be40e-352e-4b95-8bf2-4d4659185323) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:02:24 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (9b878174-1ccb-4a96-a848-c22d15f4aab9) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:02:24 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (9b878174-1ccb-4a96-a848-c22d15f4aab9) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:02:24 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (d467f31f-5732-407b-9042-43a0aeb61a36) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:02:24 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (d467f31f-5732-407b-9042-43a0aeb61a36) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:02:24 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (9b7ada94-459d-4587-8d7b-e2197a372198) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:02:24 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Updating volume 9a399a62-6a8f-4b6d-8ecf-8dcdf57e5fbb with {'scheduled_at': datetime.datetime(2023, 4, 17, 16, 2, 24, 436621), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 16:02:24 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Volume 9a399a62-6a8f-4b6d-8ecf-8dcdf57e5fbb: re-scheduling SchedulerAPI.create_volume attempt 2 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 16:02:24 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Volume 9a399a62-6a8f-4b6d-8ecf-8dcdf57e5fbb: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 16:02:24 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] deleating volume v_9a399a62-6a8f-4b6d-8ecf-8dcdf57e5fbb {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 16:02:24 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] get snapshots for volume v_9a399a62-6a8f-4b6d-8ecf-8dcdf57e5fbb {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 16:02:24 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_9a399a62-6a8f-4b6d-8ecf-8dcdf57e5fbb/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:02:24 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_9a399a62-6a8f-4b6d-8ecf-8dcdf57e5fbb/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:02:24 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Volume reschedule parameters: True retry: {'num_attempts': 3, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 16:02:24 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Flow 'volume_create_manager' (41be6685-be50-442b-9510-ccce73f8da2b) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 16:02:24 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (5d2f7300-9d37-486b-a275-c5f243b63c15) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:02:24 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (5d2f7300-9d37-486b-a275-c5f243b63c15) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:57:47Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumeRevertTests-Volume-1536424582',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=9a399a62-6a8f-4b6d-8ecf-8dcdf57e5fbb,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='9c755d3b5d8a4d9c9918b6c4744f510e',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:02:25Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:02:25Z,use_quota=True,user_id='e362cfe4a31c444b9ab44a5d60f37ad9',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:02:24 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (06e44fc5-64be-4eb1-a659-1214b2f82192) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:02:24 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (06e44fc5-64be-4eb1-a659-1214b2f82192) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:02:24 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (33dc10a6-0a6e-4f58-b409-53508534083a) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:02:24 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (33dc10a6-0a6e-4f58-b409-53508534083a) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': '9a399a62-6a8f-4b6d-8ecf-8dcdf57e5fbb', 'volume_name': 'volume-9a399a62-6a8f-4b6d-8ecf-8dcdf57e5fbb', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:02:24 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (bb8359bf-01ae-4469-822c-e9efadc5ff9e) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:02:24 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (bb8359bf-01ae-4469-822c-e9efadc5ff9e) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:02:24 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (dbfa0aea-a640-4eac-aa2b-e626059732a5) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:02:24 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Volume 9a399a62-6a8f-4b6d-8ecf-8dcdf57e5fbb: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-9a399a62-6a8f-4b6d-8ecf-8dcdf57e5fbb', 'volume_size': 1} Apr 17 16:02:24 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] creating volume v_9a399a62-6a8f-4b6d-8ecf-8dcdf57e5fbb. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 16:02:24 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] create volume {'name': 'v_9a399a62-6a8f-4b6d-8ecf-8dcdf57e5fbb', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 16:02:24 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:02:24 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:02:27 user cinder-volume[84696]: DEBUG cinder.utils [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 74.594(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:02:27 user cinder-volume[84696]: DEBUG cinder.utils [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:02:27 user cinder-volume[84696]: DEBUG cinder.utils [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.954(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:02:27 user cinder-volume[84696]: DEBUG cinder.utils [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_9a399a62-6a8f-4b6d-8ecf-8dcdf57e5fbb/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:02:27 user cinder-volume[84696]: DEBUG cinder.utils [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.712(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:02:27 user cinder-volume[84696]: DEBUG cinder.utils [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:02:27 user cinder-volume[84696]: DEBUG cinder.utils [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.418(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:02:27 user cinder-volume[84696]: DEBUG cinder.utils [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_26014b4e-ec84-4011-a69f-ed42c3424a24/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:02:27 user cinder-volume[84696]: DEBUG cinder.utils [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.178(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:02:27 user cinder-volume[84696]: DEBUG cinder.utils [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_87514055-5a0f-459f-b10b-438d7a13d39d/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:02:27 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 74.957(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:02:27 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:02:27 user cinder-volume[84696]: DEBUG cinder.utils [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.241(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:02:27 user cinder-volume[84696]: DEBUG cinder.utils [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:02:27 user cinder-volume[84696]: DEBUG cinder.utils [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 74.935(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:02:27 user cinder-volume[84696]: DEBUG cinder.utils [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_fd313648-5fc1-4e6a-97a0-92ac803b36fd/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:02:30 user cinder-volume[84696]: DEBUG cinder.utils [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.021(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:02:30 user cinder-volume[84696]: DEBUG cinder.utils [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_9a399a62-6a8f-4b6d-8ecf-8dcdf57e5fbb/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:02:30 user cinder-volume[84696]: DEBUG cinder.utils [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.160(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:02:30 user cinder-volume[84696]: DEBUG cinder.utils [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_14aa38ea-7cbb-48f7-a9aa-ffcc7b66faa0/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:02:30 user cinder-volume[84696]: DEBUG cinder.utils [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.780(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:02:30 user cinder-volume[84696]: DEBUG cinder.utils [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:02:33 user cinder-volume[84696]: DEBUG cinder.utils [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 73.777(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:02:33 user cinder-volume[84696]: DEBUG cinder.utils [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:02:34 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=18,cinder:UPDATE=7,cinder:INSERT=5 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:02:36 user cinder-volume[84696]: DEBUG cinder.utils [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.389(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:02:36 user cinder-volume[84696]: DEBUG cinder.utils [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_87514055-5a0f-459f-b10b-438d7a13d39d/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:02:36 user cinder-volume[84696]: DEBUG cinder.utils [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.171(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:02:36 user cinder-volume[84696]: DEBUG cinder.utils [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_9a399a62-6a8f-4b6d-8ecf-8dcdf57e5fbb/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:02:36 user cinder-volume[84696]: DEBUG cinder.utils [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 11.928(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:02:36 user cinder-volume[84696]: DEBUG cinder.utils [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:02:39 user cinder-volume[84696]: DEBUG cinder.utils [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.375(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:02:39 user cinder-volume[84696]: DEBUG cinder.utils [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_14aa38ea-7cbb-48f7-a9aa-ffcc7b66faa0/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:02:42 user cinder-volume[84696]: DEBUG cinder.utils [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 74.890(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:02:42 user cinder-volume[84696]: DEBUG cinder.utils [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_b29a7592-128a-41ee-a8e2-57780dee33e7/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:02:44 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=17,cinder:UPDATE=6,cinder:INSERT=1 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:02:45 user cinder-volume[84696]: DEBUG cinder.utils [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.382(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:02:45 user cinder-volume[84696]: DEBUG cinder.utils [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_9a399a62-6a8f-4b6d-8ecf-8dcdf57e5fbb/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:02:45 user cinder-volume[84696]: DEBUG cinder.utils [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.844(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:02:45 user cinder-volume[84696]: DEBUG cinder.utils [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_26014b4e-ec84-4011-a69f-ed42c3424a24/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:02:45 user cinder-volume[84696]: DEBUG cinder.utils [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.141(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:02:45 user cinder-volume[84696]: DEBUG cinder.utils [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:02:45 user cinder-volume[84696]: DEBUG cinder.utils [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.667(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:02:45 user cinder-volume[84696]: DEBUG cinder.utils [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:02:48 user cinder-volume[84696]: DEBUG cinder.utils [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 74.879(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:02:48 user cinder-volume[84696]: DEBUG cinder.utils [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_75163b78-31b2-4676-b93b-542a8ea784c1/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:02:55 user cinder-volume[84696]: DEBUG cinder.utils [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 40.457(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:02:55 user cinder-volume[84696]: DEBUG cinder.utils [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_87514055-5a0f-459f-b10b-438d7a13d39d/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:02:58 user cinder-volume[84696]: DEBUG cinder.utils [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 40.450(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:02:58 user cinder-volume[84696]: DEBUG cinder.utils [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_14aa38ea-7cbb-48f7-a9aa-ffcc7b66faa0/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:02:58 user cinder-volume[84696]: DEBUG cinder.utils [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 138.867(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:02:58 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:02:58 user cinder-volume[84696]: DEBUG cinder.coordination [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Lock "cinder-c0c829d5-daba-4365-b51f-aabe77d8aa62" released by "_prepare_image_cache_entry" :: held 140.770s {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:207}} Apr 17 16:02:58 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (7cf8c16e-4d59-4a27-9a28-b3832935c6b1) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 16:02:58 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 16:02:58 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:00:38Z,deleted=False,deleted_at=None,display_description=None,display_name='volume-for-test-6e9266ff-a917-4dd5-aa4a-c36e59e7a2a6',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=40efe2ca-3a61-44c9-b9e0-bb29b333f0f9,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='192c91b1109345ccb52f57f4db9d36a0',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:00:38Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:00:38Z,use_quota=True,user_id='d36a4ed743744fc8a1281a42a15e5b2f',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(27e07afb-6233-4285-b990-341afc23321e),volume_type_id=27e07afb-6233-4285-b990-341afc23321e)}, 'provides': None} Apr 17 16:02:58 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:00:38Z,deleted=False,deleted_at=None,display_description=None,display_name='volume-for-test-6e9266ff-a917-4dd5-aa4a-c36e59e7a2a6',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=40efe2ca-3a61-44c9-b9e0-bb29b333f0f9,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='192c91b1109345ccb52f57f4db9d36a0',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:00:38Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:00:38Z,use_quota=True,user_id='d36a4ed743744fc8a1281a42a15e5b2f',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(27e07afb-6233-4285-b990-341afc23321e),volume_type_id=27e07afb-6233-4285-b990-341afc23321e), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=c0c829d5-daba-4365-b51f-aabe77d8aa62,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(40efe2ca-3a61-44c9-b9e0-bb29b333f0f9),volume_id=40efe2ca-3a61-44c9-b9e0-bb29b333f0f9,volume_properties=VolumeProperties,volume_type=VolumeType(27e07afb-6233-4285-b990-341afc23321e))}, 'provides': {'status': 'creating', 'type': 'image', 'volume_id': '40efe2ca-3a61-44c9-b9e0-bb29b333f0f9', 'volume_name': 'volume-40efe2ca-3a61-44c9-b9e0-bb29b333f0f9', 'volume_size': 1, 'image_id': 'c0c829d5-daba-4365-b51f-aabe77d8aa62', 'image_location': (None, None), 'image_meta': {'name': 'image-for-test-6e9266ff-a917-4dd5-aa4a-c36e59e7a2a6', 'disk_format': 'raw', 'container_format': 'bare', 'visibility': 'private', 'size': 1024, 'virtual_size': 1024, 'status': 'active', 'checksum': '9d24cb2b43eab1d7d24f46a1c59d4405', 'protected': False, 'min_ram': 0, 'min_disk': 1, 'owner': '192c91b1109345ccb52f57f4db9d36a0', 'os_hidden': False, 'os_hash_algo': 'sha512', 'os_hash_value': '11384674df78c62ef87566997f126283d8379bbba0f2865f3f37e41378078ebe9ad290c06b163f2a99198703c2d61c2145df7d24bfec81fd5a51872b7b5836c1', 'id': 'c0c829d5-daba-4365-b51f-aabe77d8aa62', 'created_at': datetime.datetime(2023, 4, 17, 16, 0, 36, tzinfo=datetime.timezone.utc), 'updated_at': datetime.datetime(2023, 4, 17, 16, 0, 37, tzinfo=datetime.timezone.utc), 'tags': [], 'file': '/v2/images/c0c829d5-daba-4365-b51f-aabe77d8aa62/file', 'properties': {'cinder_img_volume_type': 'vol-type-for-6e9266ff-a917-4dd5-aa4a-c36e59e7a2a6'}}, 'image_service': }} Apr 17 16:02:58 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'retry': {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']}, 'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': 'c0c829d5-daba-4365-b51f-aabe77d8aa62', 'snapshot_id': None, 'source_volid': None, 'volume_id': '40efe2ca-3a61-44c9-b9e0-bb29b333f0f9', 'volume': {'id': '40efe2ca-3a61-44c9-b9e0-bb29b333f0f9', '_name_id': None, 'ec2_id': None, 'user_id': 'd36a4ed743744fc8a1281a42a15e5b2f', 'project_id': '192c91b1109345ccb52f57f4db9d36a0', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'volume-for-test-6e9266ff-a917-4dd5-aa4a-c36e59e7a2a6', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '27e07afb-6233-4285-b990-341afc23321e', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'volume_type': {'id': '27e07afb-6233-4285-b990-341afc23321e', 'name': 'vol-type-for-6e9266ff-a917-4dd5-aa4a-c36e59e7a2a6', 'description': 'Generic volume_type for test 6e9266ff-a917-4dd5-aa4a-c36e59e7a2a6', 'is_public': True, 'extra_specs': {'storage_protocol': 'iSCSI', 'vendor_name': 'Open-E'}, 'qos_specs_id': None, 'created_at': '2023-04-17T16:00:35.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T16:00:37.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-40efe2ca-3a61-44c9-b9e0-bb29b333f0f9', 'name_id': '40efe2ca-3a61-44c9-b9e0-bb29b333f0f9', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '27e07afb-6233-4285-b990-341afc23321e', 'name': 'vol-type-for-6e9266ff-a917-4dd5-aa4a-c36e59e7a2a6', 'description': 'Generic volume_type for test 6e9266ff-a917-4dd5-aa4a-c36e59e7a2a6', 'is_public': True, 'projects': [], 'extra_specs': {'storage_protocol': 'iSCSI', 'vendor_name': 'Open-E'}, 'qos_specs_id': None, 'created_at': '2023-04-17T16:00:35.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'volume-for-test-6e9266ff-a917-4dd5-aa4a-c36e59e7a2a6', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '192c91b1109345ccb52f57f4db9d36a0', 'qos_specs': None, 'reservations': ['736ffa12-8126-4c39-8b19-58e6bb055ec1', '8fe6d3c8-aeb4-4d01-b82a-e41992476e63', '0f8c093e-38f3-4019-ac82-eb6ab1a38098', '170e5f3e-e662-4951-8204-35be46f13690'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'd36a4ed743744fc8a1281a42a15e5b2f', 'volume_type_id': '27e07afb-6233-4285-b990-341afc23321e'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'volume-for-test-6e9266ff-a917-4dd5-aa4a-c36e59e7a2a6', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '192c91b1109345ccb52f57f4db9d36a0', 'qos_specs': None, 'reservations': ['736ffa12-8126-4c39-8b19-58e6bb055ec1', '8fe6d3c8-aeb4-4d01-b82a-e41992476e63', '0f8c093e-38f3-4019-ac82-eb6ab1a38098', '170e5f3e-e662-4951-8204-35be46f13690'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'd36a4ed743744fc8a1281a42a15e5b2f', 'volume_type_id': '27e07afb-6233-4285-b990-341afc23321e'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T16:00:35Z,deleted=False,deleted_at=None,description='Generic volume_type for test 6e9266ff-a917-4dd5-aa4a-c36e59e7a2a6',extra_specs={storage_protocol='iSCSI',vendor_name='Open-E'},id=27e07afb-6233-4285-b990-341afc23321e,is_public=True,name='vol-type-for-6e9266ff-a917-4dd5-aa4a-c36e59e7a2a6',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T16:00:35Z,deleted=False,deleted_at=None,description='Generic volume_type for test 6e9266ff-a917-4dd5-aa4a-c36e59e7a2a6',extra_specs={storage_protocol='iSCSI',vendor_name='Open-E'},id=27e07afb-6233-4285-b990-341afc23321e,is_public=True,name='vol-type-for-6e9266ff-a917-4dd5-aa4a-c36e59e7a2a6',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': 'd36a4ed743744fc8a1281a42a15e5b2f', 'metadata': {}, 'qos_specs': None}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=c0c829d5-daba-4365-b51f-aabe77d8aa62,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(40efe2ca-3a61-44c9-b9e0-bb29b333f0f9),volume_id=40efe2ca-3a61-44c9-b9e0-bb29b333f0f9,volume_properties=VolumeProperties,volume_type=VolumeType(27e07afb-6233-4285-b990-341afc23321e)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:00:38Z,deleted=False,deleted_at=None,display_description=None,display_name='volume-for-test-6e9266ff-a917-4dd5-aa4a-c36e59e7a2a6',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=40efe2ca-3a61-44c9-b9e0-bb29b333f0f9,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='192c91b1109345ccb52f57f4db9d36a0',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:00:38Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:00:38Z,use_quota=True,user_id='d36a4ed743744fc8a1281a42a15e5b2f',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(27e07afb-6233-4285-b990-341afc23321e),volume_type_id=27e07afb-6233-4285-b990-341afc23321e), 'context': }, 'provides': None} Apr 17 16:02:58 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:00:38Z,deleted=False,deleted_at=None,display_description=None,display_name='volume-for-test-6e9266ff-a917-4dd5-aa4a-c36e59e7a2a6',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=40efe2ca-3a61-44c9-b9e0-bb29b333f0f9,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='192c91b1109345ccb52f57f4db9d36a0',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:00:38Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:00:38Z,use_quota=True,user_id='d36a4ed743744fc8a1281a42a15e5b2f',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(27e07afb-6233-4285-b990-341afc23321e),volume_type_id=27e07afb-6233-4285-b990-341afc23321e)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:00:38Z,deleted=False,deleted_at=None,display_description=None,display_name='volume-for-test-6e9266ff-a917-4dd5-aa4a-c36e59e7a2a6',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=40efe2ca-3a61-44c9-b9e0-bb29b333f0f9,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='192c91b1109345ccb52f57f4db9d36a0',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:00:38Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:00:38Z,use_quota=True,user_id='d36a4ed743744fc8a1281a42a15e5b2f',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(27e07afb-6233-4285-b990-341afc23321e),volume_type_id=27e07afb-6233-4285-b990-341afc23321e)} Apr 17 16:02:58 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 16:02:58 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 16:02:58 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:02:58 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 16:02:58 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1237, in execute Apr 17 16:02:58 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_from_image(context, Apr 17 16:02:58 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/utils.py", line 670, in _wrapper Apr 17 16:02:58 user cinder-volume[84696]: ERROR cinder.volume.manager return r(f, *args, **kwargs) Apr 17 16:02:58 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/tenacity/__init__.py", line 423, in __call__ Apr 17 16:02:58 user cinder-volume[84696]: ERROR cinder.volume.manager do = self.iter(retry_state=retry_state) Apr 17 16:02:58 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/tenacity/__init__.py", line 360, in iter Apr 17 16:02:58 user cinder-volume[84696]: ERROR cinder.volume.manager return fut.result() Apr 17 16:02:58 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/lib/python3.10/concurrent/futures/_base.py", line 451, in result Apr 17 16:02:58 user cinder-volume[84696]: ERROR cinder.volume.manager return self.__get_result() Apr 17 16:02:58 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result Apr 17 16:02:58 user cinder-volume[84696]: ERROR cinder.volume.manager raise self._exception Apr 17 16:02:58 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/tenacity/__init__.py", line 426, in __call__ Apr 17 16:02:58 user cinder-volume[84696]: ERROR cinder.volume.manager result = fn(*args, **kwargs) Apr 17 16:02:58 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1115, in _create_from_image Apr 17 16:02:58 user cinder-volume[84696]: ERROR cinder.volume.manager model_update, cloned = self._prepare_image_cache_entry( Apr 17 16:02:58 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 16:02:58 user cinder-volume[84696]: ERROR cinder.volume.manager return caller(func, *(extras + args), **kw) Apr 17 16:02:58 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/coordination.py", line 200, in _synchronized Apr 17 16:02:58 user cinder-volume[84696]: ERROR cinder.volume.manager return f(*a, **k) Apr 17 16:02:58 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 894, in _prepare_image_cache_entry Apr 17 16:02:58 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_from_image_cache_or_download( Apr 17 16:02:58 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1004, in _create_from_image_cache_or_download Apr 17 16:02:58 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_from_image_download( Apr 17 16:02:58 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 791, in _create_from_image_download Apr 17 16:02:58 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self.driver.create_volume(volume) or {} Apr 17 16:02:58 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:02:58 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 16:02:58 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:02:58 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:02:58 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 16:02:58 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 16:02:58 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (7cf8c16e-4d59-4a27-9a28-b3832935c6b1) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:02:58 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (7cf8c16e-4d59-4a27-9a28-b3832935c6b1) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:02:58 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (5f77b79f-cf8e-449c-b020-0a7b712013da) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:02:58 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (5f77b79f-cf8e-449c-b020-0a7b712013da) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:02:58 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (222c2ff9-c07f-4ed0-87a8-76e038bb1d4e) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:02:58 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (222c2ff9-c07f-4ed0-87a8-76e038bb1d4e) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:02:58 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (e16a4ead-003c-4857-aa60-d98b521c0783) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:02:58 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Updating volume 40efe2ca-3a61-44c9-b9e0-bb29b333f0f9 with {'scheduled_at': datetime.datetime(2023, 4, 17, 16, 2, 58, 844363), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 16:02:58 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Volume 40efe2ca-3a61-44c9-b9e0-bb29b333f0f9: re-scheduling SchedulerAPI.create_volume attempt 1 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 16:02:58 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Volume 40efe2ca-3a61-44c9-b9e0-bb29b333f0f9: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 16:02:58 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] deleating volume v_40efe2ca-3a61-44c9-b9e0-bb29b333f0f9 {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 16:02:58 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] get snapshots for volume v_40efe2ca-3a61-44c9-b9e0-bb29b333f0f9 {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 16:02:58 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_40efe2ca-3a61-44c9-b9e0-bb29b333f0f9/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:02:58 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_40efe2ca-3a61-44c9-b9e0-bb29b333f0f9/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:02:59 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Volume reschedule parameters: True retry: {'num_attempts': 2, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 16:02:59 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Flow 'volume_create_manager' (bbb7cec9-458b-4fb5-aebc-5c9074dc3a69) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 16:02:59 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (4ce647bd-9723-4aa3-97da-c8957240bed8) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:02:59 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (4ce647bd-9723-4aa3-97da-c8957240bed8) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:00:38Z,deleted=False,deleted_at=None,display_description=None,display_name='volume-for-test-6e9266ff-a917-4dd5-aa4a-c36e59e7a2a6',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=40efe2ca-3a61-44c9-b9e0-bb29b333f0f9,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='192c91b1109345ccb52f57f4db9d36a0',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:02:59Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:02:59Z,use_quota=True,user_id='d36a4ed743744fc8a1281a42a15e5b2f',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(27e07afb-6233-4285-b990-341afc23321e),volume_type_id=27e07afb-6233-4285-b990-341afc23321e)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:02:59 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (8fa4ebdb-b6c2-4704-966a-9c68b8939008) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:02:59 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (8fa4ebdb-b6c2-4704-966a-9c68b8939008) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:02:59 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (8a151129-bd00-4744-9e50-c45c7fd1aebb) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:02:59 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (8a151129-bd00-4744-9e50-c45c7fd1aebb) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'image', 'volume_id': '40efe2ca-3a61-44c9-b9e0-bb29b333f0f9', 'volume_name': 'volume-40efe2ca-3a61-44c9-b9e0-bb29b333f0f9', 'volume_size': 1, 'image_id': 'c0c829d5-daba-4365-b51f-aabe77d8aa62', 'image_location': (None, None), 'image_meta': {'name': 'image-for-test-6e9266ff-a917-4dd5-aa4a-c36e59e7a2a6', 'disk_format': 'raw', 'container_format': 'bare', 'visibility': 'private', 'size': 1024, 'virtual_size': 1024, 'status': 'active', 'checksum': '9d24cb2b43eab1d7d24f46a1c59d4405', 'protected': False, 'min_ram': 0, 'min_disk': 1, 'owner': '192c91b1109345ccb52f57f4db9d36a0', 'os_hidden': False, 'os_hash_algo': 'sha512', 'os_hash_value': '11384674df78c62ef87566997f126283d8379bbba0f2865f3f37e41378078ebe9ad290c06b163f2a99198703c2d61c2145df7d24bfec81fd5a51872b7b5836c1', 'id': 'c0c829d5-daba-4365-b51f-aabe77d8aa62', 'created_at': datetime.datetime(2023, 4, 17, 16, 0, 36, tzinfo=datetime.timezone.utc), 'updated_at': datetime.datetime(2023, 4, 17, 16, 0, 37, tzinfo=datetime.timezone.utc), 'tags': [], 'file': '/v2/images/c0c829d5-daba-4365-b51f-aabe77d8aa62/file', 'properties': {'cinder_img_volume_type': 'vol-type-for-6e9266ff-a917-4dd5-aa4a-c36e59e7a2a6'}}, 'image_service': }' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:02:59 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (92ac9051-e9c0-462b-9cd0-10021f555b64) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:02:59 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (92ac9051-e9c0-462b-9cd0-10021f555b64) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:02:59 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (86001331-23d2-4fb0-acd9-fbb933af369d) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:02:59 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Volume 40efe2ca-3a61-44c9-b9e0-bb29b333f0f9: being created as image with specification: {'status': 'creating', 'volume_name': 'volume-40efe2ca-3a61-44c9-b9e0-bb29b333f0f9', 'volume_size': 1, 'image_id': 'c0c829d5-daba-4365-b51f-aabe77d8aa62', 'image_location': (None, None), 'image_meta': {'name': 'image-for-test-6e9266ff-a917-4dd5-aa4a-c36e59e7a2a6', 'disk_format': 'raw', 'container_format': 'bare', 'visibility': 'private', 'size': 1024, 'virtual_size': 1024, 'status': 'active', 'checksum': '9d24cb2b43eab1d7d24f46a1c59d4405', 'protected': False, 'min_ram': 0, 'min_disk': 1, 'owner': '192c91b1109345ccb52f57f4db9d36a0', 'os_hidden': False, 'os_hash_algo': 'sha512', 'os_hash_value': '11384674df78c62ef87566997f126283d8379bbba0f2865f3f37e41378078ebe9ad290c06b163f2a99198703c2d61c2145df7d24bfec81fd5a51872b7b5836c1', 'id': 'c0c829d5-daba-4365-b51f-aabe77d8aa62', 'created_at': datetime.datetime(2023, 4, 17, 16, 0, 36, tzinfo=datetime.timezone.utc), 'updated_at': datetime.datetime(2023, 4, 17, 16, 0, 37, tzinfo=datetime.timezone.utc), 'tags': [], 'file': '/v2/images/c0c829d5-daba-4365-b51f-aabe77d8aa62/file', 'properties': {'cinder_img_volume_type': 'vol-type-for-6e9266ff-a917-4dd5-aa4a-c36e59e7a2a6'}}, 'image_service': } Apr 17 16:02:59 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Cloning 40efe2ca-3a61-44c9-b9e0-bb29b333f0f9 from image c0c829d5-daba-4365-b51f-aabe77d8aa62 at location (None, None). {{(pid=84696) _create_from_image /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1071}} Apr 17 16:02:59 user cinder-volume[84696]: DEBUG cinder.coordination [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Acquiring lock "cinder-c0c829d5-daba-4365-b51f-aabe77d8aa62" by "_prepare_image_cache_entry" {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:192}} Apr 17 16:02:59 user cinder-volume[84696]: DEBUG cinder.coordination [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Lock "cinder-c0c829d5-daba-4365-b51f-aabe77d8aa62" acquired by "_prepare_image_cache_entry" :: waited 0.010s {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:196}} Apr 17 16:02:59 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Preparing cache entry for image = c0c829d5-daba-4365-b51f-aabe77d8aa62 on host user@joviandss-2#joviandss-2. {{(pid=84696) _prepare_image_cache_entry /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:891}} Apr 17 16:02:59 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Attempting to retrieve cache entry for image = c0c829d5-daba-4365-b51f-aabe77d8aa62 on host user@joviandss-2#joviandss-2. {{(pid=84696) _create_from_image_cache /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:827}} Apr 17 16:02:59 user cinder-volume[84696]: DEBUG cinder.image.image_utils [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Image fetch details: dest /opt/stack/data/cinder/conversion/image_fetch_c0c829d5-daba-4365-b51f-aabe77d8aa62_ufk58aicuser@joviandss-2, size 0.00 MB, duration 1.00 sec {{(pid=84696) fetch /opt/stack/cinder/cinder/image/image_utils.py:628}} Apr 17 16:02:59 user cinder-volume[84696]: INFO cinder.image.image_utils [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Image download 0.00 MB at 0.00 MB/s Apr 17 16:02:59 user cinder-volume[84696]: DEBUG oslo_concurrency.processutils [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Running cmd (subprocess): /usr/bin/python3.10 -m oslo_concurrency.prlimit --as=1073741824 --cpu=60 -- sudo cinder-rootwrap /etc/cinder/rootwrap.conf env LC_ALL=C qemu-img info --output=json /opt/stack/data/cinder/conversion/image_fetch_c0c829d5-daba-4365-b51f-aabe77d8aa62_ufk58aicuser@joviandss-2 {{(pid=84696) execute /usr/local/lib/python3.10/dist-packages/oslo_concurrency/processutils.py:384}} Apr 17 16:02:59 user sudo[104432]: stack : PWD=/ ; USER=root ; COMMAND=/usr/local/bin/cinder-rootwrap /etc/cinder/rootwrap.conf env LC_ALL=C qemu-img info --output=json /opt/stack/data/cinder/conversion/image_fetch_c0c829d5-daba-4365-b51f-aabe77d8aa62_ufk58aicuser@joviandss-2 Apr 17 16:02:59 user sudo[104432]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=1001) Apr 17 16:03:00 user sudo[104432]: pam_unix(sudo:session): session closed for user root Apr 17 16:03:00 user cinder-volume[84696]: DEBUG oslo_concurrency.processutils [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] CMD "/usr/bin/python3.10 -m oslo_concurrency.prlimit --as=1073741824 --cpu=60 -- sudo cinder-rootwrap /etc/cinder/rootwrap.conf env LC_ALL=C qemu-img info --output=json /opt/stack/data/cinder/conversion/image_fetch_c0c829d5-daba-4365-b51f-aabe77d8aa62_ufk58aicuser@joviandss-2" returned: 0 in 0.800s {{(pid=84696) execute /usr/local/lib/python3.10/dist-packages/oslo_concurrency/processutils.py:422}} Apr 17 16:03:00 user cinder-volume[84696]: DEBUG cinder.image.image_utils [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Temporary image c0c829d5-daba-4365-b51f-aabe77d8aa62 is fetched for user d36a4ed743744fc8a1281a42a15e5b2f. {{(pid=84696) fetch /opt/stack/cinder/cinder/image/image_utils.py:1279}} Apr 17 16:03:00 user cinder-volume[84696]: DEBUG cinder.coordination [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Lock "cinder-c0c829d5-daba-4365-b51f-aabe77d8aa62" released by "_prepare_image_cache_entry" :: held 0.965s {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:207}} Apr 17 16:03:00 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (86001331-23d2-4fb0-acd9-fbb933af369d) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 16:03:00 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 16:03:00 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:00:38Z,deleted=False,deleted_at=None,display_description=None,display_name='volume-for-test-6e9266ff-a917-4dd5-aa4a-c36e59e7a2a6',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=40efe2ca-3a61-44c9-b9e0-bb29b333f0f9,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='192c91b1109345ccb52f57f4db9d36a0',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:02:59Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:02:59Z,use_quota=True,user_id='d36a4ed743744fc8a1281a42a15e5b2f',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(27e07afb-6233-4285-b990-341afc23321e),volume_type_id=27e07afb-6233-4285-b990-341afc23321e)}, 'provides': None} Apr 17 16:03:00 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:00:38Z,deleted=False,deleted_at=None,display_description=None,display_name='volume-for-test-6e9266ff-a917-4dd5-aa4a-c36e59e7a2a6',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=40efe2ca-3a61-44c9-b9e0-bb29b333f0f9,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='192c91b1109345ccb52f57f4db9d36a0',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:02:59Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:02:59Z,use_quota=True,user_id='d36a4ed743744fc8a1281a42a15e5b2f',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(27e07afb-6233-4285-b990-341afc23321e),volume_type_id=27e07afb-6233-4285-b990-341afc23321e), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=c0c829d5-daba-4365-b51f-aabe77d8aa62,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(40efe2ca-3a61-44c9-b9e0-bb29b333f0f9),volume_id=40efe2ca-3a61-44c9-b9e0-bb29b333f0f9,volume_properties=VolumeProperties,volume_type=VolumeType(27e07afb-6233-4285-b990-341afc23321e))}, 'provides': {'status': 'creating', 'type': 'image', 'volume_id': '40efe2ca-3a61-44c9-b9e0-bb29b333f0f9', 'volume_name': 'volume-40efe2ca-3a61-44c9-b9e0-bb29b333f0f9', 'volume_size': 1, 'image_id': 'c0c829d5-daba-4365-b51f-aabe77d8aa62', 'image_location': (None, None), 'image_meta': {'name': 'image-for-test-6e9266ff-a917-4dd5-aa4a-c36e59e7a2a6', 'disk_format': 'raw', 'container_format': 'bare', 'visibility': 'private', 'size': 1024, 'virtual_size': 1024, 'status': 'active', 'checksum': '9d24cb2b43eab1d7d24f46a1c59d4405', 'protected': False, 'min_ram': 0, 'min_disk': 1, 'owner': '192c91b1109345ccb52f57f4db9d36a0', 'os_hidden': False, 'os_hash_algo': 'sha512', 'os_hash_value': '11384674df78c62ef87566997f126283d8379bbba0f2865f3f37e41378078ebe9ad290c06b163f2a99198703c2d61c2145df7d24bfec81fd5a51872b7b5836c1', 'id': 'c0c829d5-daba-4365-b51f-aabe77d8aa62', 'created_at': datetime.datetime(2023, 4, 17, 16, 0, 36, tzinfo=datetime.timezone.utc), 'updated_at': datetime.datetime(2023, 4, 17, 16, 0, 37, tzinfo=datetime.timezone.utc), 'tags': [], 'file': '/v2/images/c0c829d5-daba-4365-b51f-aabe77d8aa62/file', 'properties': {'cinder_img_volume_type': 'vol-type-for-6e9266ff-a917-4dd5-aa4a-c36e59e7a2a6'}}, 'image_service': }} Apr 17 16:03:00 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': 'c0c829d5-daba-4365-b51f-aabe77d8aa62', 'snapshot_id': None, 'source_volid': None, 'volume_id': '40efe2ca-3a61-44c9-b9e0-bb29b333f0f9', 'volume': {'id': '40efe2ca-3a61-44c9-b9e0-bb29b333f0f9', '_name_id': None, 'ec2_id': None, 'user_id': 'd36a4ed743744fc8a1281a42a15e5b2f', 'project_id': '192c91b1109345ccb52f57f4db9d36a0', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'volume-for-test-6e9266ff-a917-4dd5-aa4a-c36e59e7a2a6', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '27e07afb-6233-4285-b990-341afc23321e', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'admin_metadata': {}, 'glance_metadata': {}, 'volume_type': {'id': '27e07afb-6233-4285-b990-341afc23321e', 'name': 'vol-type-for-6e9266ff-a917-4dd5-aa4a-c36e59e7a2a6', 'description': 'Generic volume_type for test 6e9266ff-a917-4dd5-aa4a-c36e59e7a2a6', 'is_public': True, 'extra_specs': {'storage_protocol': 'iSCSI', 'vendor_name': 'Open-E'}, 'qos_specs_id': None, 'created_at': '2023-04-17T16:00:35.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T16:00:37.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-40efe2ca-3a61-44c9-b9e0-bb29b333f0f9', 'name_id': '40efe2ca-3a61-44c9-b9e0-bb29b333f0f9', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '27e07afb-6233-4285-b990-341afc23321e', 'name': 'vol-type-for-6e9266ff-a917-4dd5-aa4a-c36e59e7a2a6', 'description': 'Generic volume_type for test 6e9266ff-a917-4dd5-aa4a-c36e59e7a2a6', 'is_public': True, 'projects': [], 'extra_specs': {'storage_protocol': 'iSCSI', 'vendor_name': 'Open-E'}, 'qos_specs_id': None, 'created_at': '2023-04-17T16:00:35.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'volume-for-test-6e9266ff-a917-4dd5-aa4a-c36e59e7a2a6', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '192c91b1109345ccb52f57f4db9d36a0', 'qos_specs': None, 'reservations': ['736ffa12-8126-4c39-8b19-58e6bb055ec1', '8fe6d3c8-aeb4-4d01-b82a-e41992476e63', '0f8c093e-38f3-4019-ac82-eb6ab1a38098', '170e5f3e-e662-4951-8204-35be46f13690'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'd36a4ed743744fc8a1281a42a15e5b2f', 'volume_type_id': '27e07afb-6233-4285-b990-341afc23321e'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'volume-for-test-6e9266ff-a917-4dd5-aa4a-c36e59e7a2a6', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '192c91b1109345ccb52f57f4db9d36a0', 'qos_specs': None, 'reservations': ['736ffa12-8126-4c39-8b19-58e6bb055ec1', '8fe6d3c8-aeb4-4d01-b82a-e41992476e63', '0f8c093e-38f3-4019-ac82-eb6ab1a38098', '170e5f3e-e662-4951-8204-35be46f13690'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'd36a4ed743744fc8a1281a42a15e5b2f', 'volume_type_id': '27e07afb-6233-4285-b990-341afc23321e'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T16:00:35Z,deleted=False,deleted_at=None,description='Generic volume_type for test 6e9266ff-a917-4dd5-aa4a-c36e59e7a2a6',extra_specs={storage_protocol='iSCSI',vendor_name='Open-E'},id=27e07afb-6233-4285-b990-341afc23321e,is_public=True,name='vol-type-for-6e9266ff-a917-4dd5-aa4a-c36e59e7a2a6',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T16:00:35Z,deleted=False,deleted_at=None,description='Generic volume_type for test 6e9266ff-a917-4dd5-aa4a-c36e59e7a2a6',extra_specs={storage_protocol='iSCSI',vendor_name='Open-E'},id=27e07afb-6233-4285-b990-341afc23321e,is_public=True,name='vol-type-for-6e9266ff-a917-4dd5-aa4a-c36e59e7a2a6',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': 'd36a4ed743744fc8a1281a42a15e5b2f', 'metadata': {}, 'qos_specs': None, 'retry': {'num_attempts': 2, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']}}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=c0c829d5-daba-4365-b51f-aabe77d8aa62,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(40efe2ca-3a61-44c9-b9e0-bb29b333f0f9),volume_id=40efe2ca-3a61-44c9-b9e0-bb29b333f0f9,volume_properties=VolumeProperties,volume_type=VolumeType(27e07afb-6233-4285-b990-341afc23321e)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:00:38Z,deleted=False,deleted_at=None,display_description=None,display_name='volume-for-test-6e9266ff-a917-4dd5-aa4a-c36e59e7a2a6',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=40efe2ca-3a61-44c9-b9e0-bb29b333f0f9,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='192c91b1109345ccb52f57f4db9d36a0',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:02:59Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:02:59Z,use_quota=True,user_id='d36a4ed743744fc8a1281a42a15e5b2f',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(27e07afb-6233-4285-b990-341afc23321e),volume_type_id=27e07afb-6233-4285-b990-341afc23321e), 'context': }, 'provides': None} Apr 17 16:03:00 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:00:38Z,deleted=False,deleted_at=None,display_description=None,display_name='volume-for-test-6e9266ff-a917-4dd5-aa4a-c36e59e7a2a6',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=40efe2ca-3a61-44c9-b9e0-bb29b333f0f9,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='192c91b1109345ccb52f57f4db9d36a0',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:02:59Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:02:59Z,use_quota=True,user_id='d36a4ed743744fc8a1281a42a15e5b2f',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(27e07afb-6233-4285-b990-341afc23321e),volume_type_id=27e07afb-6233-4285-b990-341afc23321e)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:00:38Z,deleted=False,deleted_at=None,display_description=None,display_name='volume-for-test-6e9266ff-a917-4dd5-aa4a-c36e59e7a2a6',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=40efe2ca-3a61-44c9-b9e0-bb29b333f0f9,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='192c91b1109345ccb52f57f4db9d36a0',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:02:59Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:02:59Z,use_quota=True,user_id='d36a4ed743744fc8a1281a42a15e5b2f',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(27e07afb-6233-4285-b990-341afc23321e),volume_type_id=27e07afb-6233-4285-b990-341afc23321e)} Apr 17 16:03:00 user cinder-volume[84696]: |__Flow 'volume_create_manager': cinder.exception.GlanceMetadataExists: Glance metadata cannot be updated, key signature_verified exists for volume id 40efe2ca-3a61-44c9-b9e0-bb29b333f0f9 Apr 17 16:03:00 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 16:03:00 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:03:00 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 16:03:00 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1237, in execute Apr 17 16:03:00 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_from_image(context, Apr 17 16:03:00 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/utils.py", line 670, in _wrapper Apr 17 16:03:00 user cinder-volume[84696]: ERROR cinder.volume.manager return r(f, *args, **kwargs) Apr 17 16:03:00 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/tenacity/__init__.py", line 423, in __call__ Apr 17 16:03:00 user cinder-volume[84696]: ERROR cinder.volume.manager do = self.iter(retry_state=retry_state) Apr 17 16:03:00 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/tenacity/__init__.py", line 360, in iter Apr 17 16:03:00 user cinder-volume[84696]: ERROR cinder.volume.manager return fut.result() Apr 17 16:03:00 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/lib/python3.10/concurrent/futures/_base.py", line 451, in result Apr 17 16:03:00 user cinder-volume[84696]: ERROR cinder.volume.manager return self.__get_result() Apr 17 16:03:00 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result Apr 17 16:03:00 user cinder-volume[84696]: ERROR cinder.volume.manager raise self._exception Apr 17 16:03:00 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/tenacity/__init__.py", line 426, in __call__ Apr 17 16:03:00 user cinder-volume[84696]: ERROR cinder.volume.manager result = fn(*args, **kwargs) Apr 17 16:03:00 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1115, in _create_from_image Apr 17 16:03:00 user cinder-volume[84696]: ERROR cinder.volume.manager model_update, cloned = self._prepare_image_cache_entry( Apr 17 16:03:00 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 16:03:00 user cinder-volume[84696]: ERROR cinder.volume.manager return caller(func, *(extras + args), **kw) Apr 17 16:03:00 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/coordination.py", line 200, in _synchronized Apr 17 16:03:00 user cinder-volume[84696]: ERROR cinder.volume.manager return f(*a, **k) Apr 17 16:03:00 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 894, in _prepare_image_cache_entry Apr 17 16:03:00 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_from_image_cache_or_download( Apr 17 16:03:00 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 989, in _create_from_image_cache_or_download Apr 17 16:03:00 user cinder-volume[84696]: ERROR cinder.volume.manager self.db.volume_glance_metadata_bulk_create( Apr 17 16:03:00 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/db/api.py", line 1138, in volume_glance_metadata_bulk_create Apr 17 16:03:00 user cinder-volume[84696]: ERROR cinder.volume.manager return IMPL.volume_glance_metadata_bulk_create(context, volume_id, Apr 17 16:03:00 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/db/sqlalchemy/api.py", line 181, in wrapper Apr 17 16:03:00 user cinder-volume[84696]: ERROR cinder.volume.manager return f(*args, **kwargs) Apr 17 16:03:00 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/db/sqlalchemy/api.py", line 211, in wrapper Apr 17 16:03:00 user cinder-volume[84696]: ERROR cinder.volume.manager return f(context, volume_id, *args, **kwargs) Apr 17 16:03:00 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1022, in wrapper Apr 17 16:03:00 user cinder-volume[84696]: ERROR cinder.volume.manager return fn(*args, **kwargs) Apr 17 16:03:00 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/db/sqlalchemy/api.py", line 6178, in volume_glance_metadata_bulk_create Apr 17 16:03:00 user cinder-volume[84696]: ERROR cinder.volume.manager raise exception.GlanceMetadataExists(key=key, volume_id=volume_id) Apr 17 16:03:00 user cinder-volume[84696]: ERROR cinder.volume.manager cinder.exception.GlanceMetadataExists: Glance metadata cannot be updated, key signature_verified exists for volume id 40efe2ca-3a61-44c9-b9e0-bb29b333f0f9 Apr 17 16:03:00 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 16:03:00 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (86001331-23d2-4fb0-acd9-fbb933af369d) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:03:00 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (86001331-23d2-4fb0-acd9-fbb933af369d) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:03:00 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (92ac9051-e9c0-462b-9cd0-10021f555b64) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:03:00 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (92ac9051-e9c0-462b-9cd0-10021f555b64) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:03:00 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (8a151129-bd00-4744-9e50-c45c7fd1aebb) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:03:00 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (8a151129-bd00-4744-9e50-c45c7fd1aebb) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:03:00 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (8fa4ebdb-b6c2-4704-966a-9c68b8939008) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:03:00 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Updating volume 40efe2ca-3a61-44c9-b9e0-bb29b333f0f9 with {'scheduled_at': datetime.datetime(2023, 4, 17, 16, 3, 0, 201409), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 16:03:00 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Volume 40efe2ca-3a61-44c9-b9e0-bb29b333f0f9: re-scheduling SchedulerAPI.create_volume attempt 2 due to Glance metadata cannot be updated, key signature_verified exists for volume id 40efe2ca-3a61-44c9-b9e0-bb29b333f0f9 {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 16:03:00 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Volume 40efe2ca-3a61-44c9-b9e0-bb29b333f0f9: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 16:03:00 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] deleating volume v_40efe2ca-3a61-44c9-b9e0-bb29b333f0f9 {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 16:03:00 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] get snapshots for volume v_40efe2ca-3a61-44c9-b9e0-bb29b333f0f9 {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 16:03:00 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_40efe2ca-3a61-44c9-b9e0-bb29b333f0f9/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:03:00 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_40efe2ca-3a61-44c9-b9e0-bb29b333f0f9/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:03:00 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Volume reschedule parameters: True retry: {'num_attempts': 3, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 16:03:00 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Flow 'volume_create_manager' (99451714-a95d-4706-bc6e-e18b8a77895c) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 16:03:00 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (2cf5e095-a726-4784-b144-23c00b772354) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:03:00 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (2cf5e095-a726-4784-b144-23c00b772354) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:00:38Z,deleted=False,deleted_at=None,display_description=None,display_name='volume-for-test-6e9266ff-a917-4dd5-aa4a-c36e59e7a2a6',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=40efe2ca-3a61-44c9-b9e0-bb29b333f0f9,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='192c91b1109345ccb52f57f4db9d36a0',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:03:00Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:03:00Z,use_quota=True,user_id='d36a4ed743744fc8a1281a42a15e5b2f',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(27e07afb-6233-4285-b990-341afc23321e),volume_type_id=27e07afb-6233-4285-b990-341afc23321e)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:03:00 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (4a5898c7-4880-42c6-90a8-cc3f84431bab) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:03:00 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (4a5898c7-4880-42c6-90a8-cc3f84431bab) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:03:00 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (09c610c6-8611-46e5-bff3-f5c28b86f316) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:03:00 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (09c610c6-8611-46e5-bff3-f5c28b86f316) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'image', 'volume_id': '40efe2ca-3a61-44c9-b9e0-bb29b333f0f9', 'volume_name': 'volume-40efe2ca-3a61-44c9-b9e0-bb29b333f0f9', 'volume_size': 1, 'image_id': 'c0c829d5-daba-4365-b51f-aabe77d8aa62', 'image_location': (None, None), 'image_meta': {'name': 'image-for-test-6e9266ff-a917-4dd5-aa4a-c36e59e7a2a6', 'disk_format': 'raw', 'container_format': 'bare', 'visibility': 'private', 'size': 1024, 'virtual_size': 1024, 'status': 'active', 'checksum': '9d24cb2b43eab1d7d24f46a1c59d4405', 'protected': False, 'min_ram': 0, 'min_disk': 1, 'owner': '192c91b1109345ccb52f57f4db9d36a0', 'os_hidden': False, 'os_hash_algo': 'sha512', 'os_hash_value': '11384674df78c62ef87566997f126283d8379bbba0f2865f3f37e41378078ebe9ad290c06b163f2a99198703c2d61c2145df7d24bfec81fd5a51872b7b5836c1', 'id': 'c0c829d5-daba-4365-b51f-aabe77d8aa62', 'created_at': datetime.datetime(2023, 4, 17, 16, 0, 36, tzinfo=datetime.timezone.utc), 'updated_at': datetime.datetime(2023, 4, 17, 16, 0, 37, tzinfo=datetime.timezone.utc), 'tags': [], 'file': '/v2/images/c0c829d5-daba-4365-b51f-aabe77d8aa62/file', 'properties': {'cinder_img_volume_type': 'vol-type-for-6e9266ff-a917-4dd5-aa4a-c36e59e7a2a6'}}, 'image_service': }' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:03:00 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (7586b01c-dbc8-4583-9a55-970b80b53076) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:03:00 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (7586b01c-dbc8-4583-9a55-970b80b53076) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:03:00 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (3b95a785-26ec-4a10-8bfb-59c525361459) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:03:00 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Volume 40efe2ca-3a61-44c9-b9e0-bb29b333f0f9: being created as image with specification: {'status': 'creating', 'volume_name': 'volume-40efe2ca-3a61-44c9-b9e0-bb29b333f0f9', 'volume_size': 1, 'image_id': 'c0c829d5-daba-4365-b51f-aabe77d8aa62', 'image_location': (None, None), 'image_meta': {'name': 'image-for-test-6e9266ff-a917-4dd5-aa4a-c36e59e7a2a6', 'disk_format': 'raw', 'container_format': 'bare', 'visibility': 'private', 'size': 1024, 'virtual_size': 1024, 'status': 'active', 'checksum': '9d24cb2b43eab1d7d24f46a1c59d4405', 'protected': False, 'min_ram': 0, 'min_disk': 1, 'owner': '192c91b1109345ccb52f57f4db9d36a0', 'os_hidden': False, 'os_hash_algo': 'sha512', 'os_hash_value': '11384674df78c62ef87566997f126283d8379bbba0f2865f3f37e41378078ebe9ad290c06b163f2a99198703c2d61c2145df7d24bfec81fd5a51872b7b5836c1', 'id': 'c0c829d5-daba-4365-b51f-aabe77d8aa62', 'created_at': datetime.datetime(2023, 4, 17, 16, 0, 36, tzinfo=datetime.timezone.utc), 'updated_at': datetime.datetime(2023, 4, 17, 16, 0, 37, tzinfo=datetime.timezone.utc), 'tags': [], 'file': '/v2/images/c0c829d5-daba-4365-b51f-aabe77d8aa62/file', 'properties': {'cinder_img_volume_type': 'vol-type-for-6e9266ff-a917-4dd5-aa4a-c36e59e7a2a6'}}, 'image_service': } Apr 17 16:03:00 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Cloning 40efe2ca-3a61-44c9-b9e0-bb29b333f0f9 from image c0c829d5-daba-4365-b51f-aabe77d8aa62 at location (None, None). {{(pid=84696) _create_from_image /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1071}} Apr 17 16:03:00 user cinder-volume[84696]: DEBUG cinder.coordination [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Acquiring lock "cinder-c0c829d5-daba-4365-b51f-aabe77d8aa62" by "_prepare_image_cache_entry" {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:192}} Apr 17 16:03:00 user cinder-volume[84696]: DEBUG cinder.coordination [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Lock "cinder-c0c829d5-daba-4365-b51f-aabe77d8aa62" acquired by "_prepare_image_cache_entry" :: waited 0.009s {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:196}} Apr 17 16:03:00 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Preparing cache entry for image = c0c829d5-daba-4365-b51f-aabe77d8aa62 on host user@joviandss-2#joviandss-2. {{(pid=84696) _prepare_image_cache_entry /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:891}} Apr 17 16:03:00 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Attempting to retrieve cache entry for image = c0c829d5-daba-4365-b51f-aabe77d8aa62 on host user@joviandss-2#joviandss-2. {{(pid=84696) _create_from_image_cache /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:827}} Apr 17 16:03:00 user cinder-volume[84696]: DEBUG cinder.image.image_utils [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Image fetch details: dest /opt/stack/data/cinder/conversion/image_fetch_c0c829d5-daba-4365-b51f-aabe77d8aa62_wk9ji4causer@joviandss-2, size 0.00 MB, duration 1.00 sec {{(pid=84696) fetch /opt/stack/cinder/cinder/image/image_utils.py:628}} Apr 17 16:03:00 user cinder-volume[84696]: INFO cinder.image.image_utils [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Image download 0.00 MB at 0.00 MB/s Apr 17 16:03:00 user cinder-volume[84696]: DEBUG oslo_concurrency.processutils [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Running cmd (subprocess): /usr/bin/python3.10 -m oslo_concurrency.prlimit --as=1073741824 --cpu=60 -- sudo cinder-rootwrap /etc/cinder/rootwrap.conf env LC_ALL=C qemu-img info --output=json /opt/stack/data/cinder/conversion/image_fetch_c0c829d5-daba-4365-b51f-aabe77d8aa62_wk9ji4causer@joviandss-2 {{(pid=84696) execute /usr/local/lib/python3.10/dist-packages/oslo_concurrency/processutils.py:384}} Apr 17 16:03:00 user sudo[104437]: stack : PWD=/ ; USER=root ; COMMAND=/usr/local/bin/cinder-rootwrap /etc/cinder/rootwrap.conf env LC_ALL=C qemu-img info --output=json /opt/stack/data/cinder/conversion/image_fetch_c0c829d5-daba-4365-b51f-aabe77d8aa62_wk9ji4causer@joviandss-2 Apr 17 16:03:00 user sudo[104437]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=1001) Apr 17 16:03:01 user sudo[104437]: pam_unix(sudo:session): session closed for user root Apr 17 16:03:01 user cinder-volume[84696]: DEBUG oslo_concurrency.processutils [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] CMD "/usr/bin/python3.10 -m oslo_concurrency.prlimit --as=1073741824 --cpu=60 -- sudo cinder-rootwrap /etc/cinder/rootwrap.conf env LC_ALL=C qemu-img info --output=json /opt/stack/data/cinder/conversion/image_fetch_c0c829d5-daba-4365-b51f-aabe77d8aa62_wk9ji4causer@joviandss-2" returned: 0 in 0.772s {{(pid=84696) execute /usr/local/lib/python3.10/dist-packages/oslo_concurrency/processutils.py:422}} Apr 17 16:03:01 user cinder-volume[84696]: DEBUG cinder.image.image_utils [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Temporary image c0c829d5-daba-4365-b51f-aabe77d8aa62 is fetched for user d36a4ed743744fc8a1281a42a15e5b2f. {{(pid=84696) fetch /opt/stack/cinder/cinder/image/image_utils.py:1279}} Apr 17 16:03:01 user cinder-volume[84696]: DEBUG cinder.coordination [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Lock "cinder-c0c829d5-daba-4365-b51f-aabe77d8aa62" released by "_prepare_image_cache_entry" :: held 0.935s {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:207}} Apr 17 16:03:01 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (3b95a785-26ec-4a10-8bfb-59c525361459) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 16:03:01 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 16:03:01 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:00:38Z,deleted=False,deleted_at=None,display_description=None,display_name='volume-for-test-6e9266ff-a917-4dd5-aa4a-c36e59e7a2a6',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=40efe2ca-3a61-44c9-b9e0-bb29b333f0f9,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='192c91b1109345ccb52f57f4db9d36a0',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:03:00Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:03:00Z,use_quota=True,user_id='d36a4ed743744fc8a1281a42a15e5b2f',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(27e07afb-6233-4285-b990-341afc23321e),volume_type_id=27e07afb-6233-4285-b990-341afc23321e)}, 'provides': None} Apr 17 16:03:01 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:00:38Z,deleted=False,deleted_at=None,display_description=None,display_name='volume-for-test-6e9266ff-a917-4dd5-aa4a-c36e59e7a2a6',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=40efe2ca-3a61-44c9-b9e0-bb29b333f0f9,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='192c91b1109345ccb52f57f4db9d36a0',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:03:00Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:03:00Z,use_quota=True,user_id='d36a4ed743744fc8a1281a42a15e5b2f',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(27e07afb-6233-4285-b990-341afc23321e),volume_type_id=27e07afb-6233-4285-b990-341afc23321e), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=c0c829d5-daba-4365-b51f-aabe77d8aa62,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(40efe2ca-3a61-44c9-b9e0-bb29b333f0f9),volume_id=40efe2ca-3a61-44c9-b9e0-bb29b333f0f9,volume_properties=VolumeProperties,volume_type=VolumeType(27e07afb-6233-4285-b990-341afc23321e))}, 'provides': {'status': 'creating', 'type': 'image', 'volume_id': '40efe2ca-3a61-44c9-b9e0-bb29b333f0f9', 'volume_name': 'volume-40efe2ca-3a61-44c9-b9e0-bb29b333f0f9', 'volume_size': 1, 'image_id': 'c0c829d5-daba-4365-b51f-aabe77d8aa62', 'image_location': (None, None), 'image_meta': {'name': 'image-for-test-6e9266ff-a917-4dd5-aa4a-c36e59e7a2a6', 'disk_format': 'raw', 'container_format': 'bare', 'visibility': 'private', 'size': 1024, 'virtual_size': 1024, 'status': 'active', 'checksum': '9d24cb2b43eab1d7d24f46a1c59d4405', 'protected': False, 'min_ram': 0, 'min_disk': 1, 'owner': '192c91b1109345ccb52f57f4db9d36a0', 'os_hidden': False, 'os_hash_algo': 'sha512', 'os_hash_value': '11384674df78c62ef87566997f126283d8379bbba0f2865f3f37e41378078ebe9ad290c06b163f2a99198703c2d61c2145df7d24bfec81fd5a51872b7b5836c1', 'id': 'c0c829d5-daba-4365-b51f-aabe77d8aa62', 'created_at': datetime.datetime(2023, 4, 17, 16, 0, 36, tzinfo=datetime.timezone.utc), 'updated_at': datetime.datetime(2023, 4, 17, 16, 0, 37, tzinfo=datetime.timezone.utc), 'tags': [], 'file': '/v2/images/c0c829d5-daba-4365-b51f-aabe77d8aa62/file', 'properties': {'cinder_img_volume_type': 'vol-type-for-6e9266ff-a917-4dd5-aa4a-c36e59e7a2a6'}}, 'image_service': }} Apr 17 16:03:01 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': 'c0c829d5-daba-4365-b51f-aabe77d8aa62', 'snapshot_id': None, 'source_volid': None, 'volume_id': '40efe2ca-3a61-44c9-b9e0-bb29b333f0f9', 'volume': {'id': '40efe2ca-3a61-44c9-b9e0-bb29b333f0f9', '_name_id': None, 'ec2_id': None, 'user_id': 'd36a4ed743744fc8a1281a42a15e5b2f', 'project_id': '192c91b1109345ccb52f57f4db9d36a0', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'volume-for-test-6e9266ff-a917-4dd5-aa4a-c36e59e7a2a6', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '27e07afb-6233-4285-b990-341afc23321e', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'admin_metadata': {}, 'glance_metadata': {}, 'volume_type': {'id': '27e07afb-6233-4285-b990-341afc23321e', 'name': 'vol-type-for-6e9266ff-a917-4dd5-aa4a-c36e59e7a2a6', 'description': 'Generic volume_type for test 6e9266ff-a917-4dd5-aa4a-c36e59e7a2a6', 'is_public': True, 'extra_specs': {'storage_protocol': 'iSCSI', 'vendor_name': 'Open-E'}, 'qos_specs_id': None, 'created_at': '2023-04-17T16:00:35.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T16:00:37.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-40efe2ca-3a61-44c9-b9e0-bb29b333f0f9', 'name_id': '40efe2ca-3a61-44c9-b9e0-bb29b333f0f9', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '27e07afb-6233-4285-b990-341afc23321e', 'name': 'vol-type-for-6e9266ff-a917-4dd5-aa4a-c36e59e7a2a6', 'description': 'Generic volume_type for test 6e9266ff-a917-4dd5-aa4a-c36e59e7a2a6', 'is_public': True, 'projects': [], 'extra_specs': {'storage_protocol': 'iSCSI', 'vendor_name': 'Open-E'}, 'qos_specs_id': None, 'created_at': '2023-04-17T16:00:35.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'volume-for-test-6e9266ff-a917-4dd5-aa4a-c36e59e7a2a6', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '192c91b1109345ccb52f57f4db9d36a0', 'qos_specs': None, 'reservations': ['736ffa12-8126-4c39-8b19-58e6bb055ec1', '8fe6d3c8-aeb4-4d01-b82a-e41992476e63', '0f8c093e-38f3-4019-ac82-eb6ab1a38098', '170e5f3e-e662-4951-8204-35be46f13690'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'd36a4ed743744fc8a1281a42a15e5b2f', 'volume_type_id': '27e07afb-6233-4285-b990-341afc23321e'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'volume-for-test-6e9266ff-a917-4dd5-aa4a-c36e59e7a2a6', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '192c91b1109345ccb52f57f4db9d36a0', 'qos_specs': None, 'reservations': ['736ffa12-8126-4c39-8b19-58e6bb055ec1', '8fe6d3c8-aeb4-4d01-b82a-e41992476e63', '0f8c093e-38f3-4019-ac82-eb6ab1a38098', '170e5f3e-e662-4951-8204-35be46f13690'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'd36a4ed743744fc8a1281a42a15e5b2f', 'volume_type_id': '27e07afb-6233-4285-b990-341afc23321e'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T16:00:35Z,deleted=False,deleted_at=None,description='Generic volume_type for test 6e9266ff-a917-4dd5-aa4a-c36e59e7a2a6',extra_specs={storage_protocol='iSCSI',vendor_name='Open-E'},id=27e07afb-6233-4285-b990-341afc23321e,is_public=True,name='vol-type-for-6e9266ff-a917-4dd5-aa4a-c36e59e7a2a6',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T16:00:35Z,deleted=False,deleted_at=None,description='Generic volume_type for test 6e9266ff-a917-4dd5-aa4a-c36e59e7a2a6',extra_specs={storage_protocol='iSCSI',vendor_name='Open-E'},id=27e07afb-6233-4285-b990-341afc23321e,is_public=True,name='vol-type-for-6e9266ff-a917-4dd5-aa4a-c36e59e7a2a6',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': 'd36a4ed743744fc8a1281a42a15e5b2f', 'metadata': {}, 'qos_specs': None, 'retry': {'num_attempts': 3, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']}}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=c0c829d5-daba-4365-b51f-aabe77d8aa62,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(40efe2ca-3a61-44c9-b9e0-bb29b333f0f9),volume_id=40efe2ca-3a61-44c9-b9e0-bb29b333f0f9,volume_properties=VolumeProperties,volume_type=VolumeType(27e07afb-6233-4285-b990-341afc23321e)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:00:38Z,deleted=False,deleted_at=None,display_description=None,display_name='volume-for-test-6e9266ff-a917-4dd5-aa4a-c36e59e7a2a6',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=40efe2ca-3a61-44c9-b9e0-bb29b333f0f9,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='192c91b1109345ccb52f57f4db9d36a0',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:03:00Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:03:00Z,use_quota=True,user_id='d36a4ed743744fc8a1281a42a15e5b2f',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(27e07afb-6233-4285-b990-341afc23321e),volume_type_id=27e07afb-6233-4285-b990-341afc23321e), 'context': }, 'provides': None} Apr 17 16:03:01 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:00:38Z,deleted=False,deleted_at=None,display_description=None,display_name='volume-for-test-6e9266ff-a917-4dd5-aa4a-c36e59e7a2a6',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=40efe2ca-3a61-44c9-b9e0-bb29b333f0f9,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='192c91b1109345ccb52f57f4db9d36a0',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:03:00Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:03:00Z,use_quota=True,user_id='d36a4ed743744fc8a1281a42a15e5b2f',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(27e07afb-6233-4285-b990-341afc23321e),volume_type_id=27e07afb-6233-4285-b990-341afc23321e)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:00:38Z,deleted=False,deleted_at=None,display_description=None,display_name='volume-for-test-6e9266ff-a917-4dd5-aa4a-c36e59e7a2a6',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=40efe2ca-3a61-44c9-b9e0-bb29b333f0f9,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='192c91b1109345ccb52f57f4db9d36a0',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:03:00Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:03:00Z,use_quota=True,user_id='d36a4ed743744fc8a1281a42a15e5b2f',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(27e07afb-6233-4285-b990-341afc23321e),volume_type_id=27e07afb-6233-4285-b990-341afc23321e)} Apr 17 16:03:01 user cinder-volume[84696]: |__Flow 'volume_create_manager': cinder.exception.GlanceMetadataExists: Glance metadata cannot be updated, key signature_verified exists for volume id 40efe2ca-3a61-44c9-b9e0-bb29b333f0f9 Apr 17 16:03:01 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 16:03:01 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:03:01 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 16:03:01 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1237, in execute Apr 17 16:03:01 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_from_image(context, Apr 17 16:03:01 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/utils.py", line 670, in _wrapper Apr 17 16:03:01 user cinder-volume[84696]: ERROR cinder.volume.manager return r(f, *args, **kwargs) Apr 17 16:03:01 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/tenacity/__init__.py", line 423, in __call__ Apr 17 16:03:01 user cinder-volume[84696]: ERROR cinder.volume.manager do = self.iter(retry_state=retry_state) Apr 17 16:03:01 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/tenacity/__init__.py", line 360, in iter Apr 17 16:03:01 user cinder-volume[84696]: ERROR cinder.volume.manager return fut.result() Apr 17 16:03:01 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/lib/python3.10/concurrent/futures/_base.py", line 451, in result Apr 17 16:03:01 user cinder-volume[84696]: ERROR cinder.volume.manager return self.__get_result() Apr 17 16:03:01 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result Apr 17 16:03:01 user cinder-volume[84696]: ERROR cinder.volume.manager raise self._exception Apr 17 16:03:01 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/tenacity/__init__.py", line 426, in __call__ Apr 17 16:03:01 user cinder-volume[84696]: ERROR cinder.volume.manager result = fn(*args, **kwargs) Apr 17 16:03:01 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1115, in _create_from_image Apr 17 16:03:01 user cinder-volume[84696]: ERROR cinder.volume.manager model_update, cloned = self._prepare_image_cache_entry( Apr 17 16:03:01 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 16:03:01 user cinder-volume[84696]: ERROR cinder.volume.manager return caller(func, *(extras + args), **kw) Apr 17 16:03:01 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/coordination.py", line 200, in _synchronized Apr 17 16:03:01 user cinder-volume[84696]: ERROR cinder.volume.manager return f(*a, **k) Apr 17 16:03:01 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 894, in _prepare_image_cache_entry Apr 17 16:03:01 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_from_image_cache_or_download( Apr 17 16:03:01 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 989, in _create_from_image_cache_or_download Apr 17 16:03:01 user cinder-volume[84696]: ERROR cinder.volume.manager self.db.volume_glance_metadata_bulk_create( Apr 17 16:03:01 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/db/api.py", line 1138, in volume_glance_metadata_bulk_create Apr 17 16:03:01 user cinder-volume[84696]: ERROR cinder.volume.manager return IMPL.volume_glance_metadata_bulk_create(context, volume_id, Apr 17 16:03:01 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/db/sqlalchemy/api.py", line 181, in wrapper Apr 17 16:03:01 user cinder-volume[84696]: ERROR cinder.volume.manager return f(*args, **kwargs) Apr 17 16:03:01 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/db/sqlalchemy/api.py", line 211, in wrapper Apr 17 16:03:01 user cinder-volume[84696]: ERROR cinder.volume.manager return f(context, volume_id, *args, **kwargs) Apr 17 16:03:01 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1022, in wrapper Apr 17 16:03:01 user cinder-volume[84696]: ERROR cinder.volume.manager return fn(*args, **kwargs) Apr 17 16:03:01 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/db/sqlalchemy/api.py", line 6178, in volume_glance_metadata_bulk_create Apr 17 16:03:01 user cinder-volume[84696]: ERROR cinder.volume.manager raise exception.GlanceMetadataExists(key=key, volume_id=volume_id) Apr 17 16:03:01 user cinder-volume[84696]: ERROR cinder.volume.manager cinder.exception.GlanceMetadataExists: Glance metadata cannot be updated, key signature_verified exists for volume id 40efe2ca-3a61-44c9-b9e0-bb29b333f0f9 Apr 17 16:03:01 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 16:03:01 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (3b95a785-26ec-4a10-8bfb-59c525361459) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:03:01 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (3b95a785-26ec-4a10-8bfb-59c525361459) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:03:01 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (7586b01c-dbc8-4583-9a55-970b80b53076) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:03:01 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (7586b01c-dbc8-4583-9a55-970b80b53076) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:03:01 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (09c610c6-8611-46e5-bff3-f5c28b86f316) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:03:01 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (09c610c6-8611-46e5-bff3-f5c28b86f316) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:03:01 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (4a5898c7-4880-42c6-90a8-cc3f84431bab) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:03:01 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Updating volume 40efe2ca-3a61-44c9-b9e0-bb29b333f0f9 with {'scheduled_at': datetime.datetime(2023, 4, 17, 16, 3, 1, 469205), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 16:03:01 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Volume 40efe2ca-3a61-44c9-b9e0-bb29b333f0f9: re-scheduling SchedulerAPI.create_volume attempt 3 due to Glance metadata cannot be updated, key signature_verified exists for volume id 40efe2ca-3a61-44c9-b9e0-bb29b333f0f9 {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 16:03:01 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Volume 40efe2ca-3a61-44c9-b9e0-bb29b333f0f9: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 16:03:01 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] deleating volume v_40efe2ca-3a61-44c9-b9e0-bb29b333f0f9 {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 16:03:01 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] get snapshots for volume v_40efe2ca-3a61-44c9-b9e0-bb29b333f0f9 {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 16:03:01 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_40efe2ca-3a61-44c9-b9e0-bb29b333f0f9/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:03:01 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_40efe2ca-3a61-44c9-b9e0-bb29b333f0f9/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:03:01 user cinder-volume[84696]: DEBUG cinder.utils [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.974(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:03:01 user cinder-volume[84696]: DEBUG cinder.utils [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_40efe2ca-3a61-44c9-b9e0-bb29b333f0f9/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:03:01 user cinder-volume[84696]: DEBUG cinder.utils [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 1.624(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:03:01 user cinder-volume[84696]: DEBUG cinder.utils [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_40efe2ca-3a61-44c9-b9e0-bb29b333f0f9/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:03:01 user cinder-volume[84696]: DEBUG cinder.utils [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 0.358(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:03:01 user cinder-volume[84696]: DEBUG cinder.utils [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_40efe2ca-3a61-44c9-b9e0-bb29b333f0f9/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:03:01 user cinder-volume[84696]: DEBUG cinder.utils [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 137.369(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:03:01 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:03:01 user cinder-volume[84696]: INFO cinder.message.api [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Creating message record for request_id = req-39f8a1bb-4b23-4f63-8942-bcd810255374 Apr 17 16:03:01 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 16:03:01 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (3607be78-7861-4ce5-9705-ae5cc87302c3) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 16:03:01 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 16:03:01 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:00:44Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-AttachVolumeShelveTestJSON-volume-7555594',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=b4281d17-c1d8-48d3-8a0b-8fd7d2f193cc,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='214a0f233f114d7aa22e64048a837c0b',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:00:44Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:00:44Z,use_quota=True,user_id='83b9d77de46143d8b9277a55f5ad4d72',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 16:03:01 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:00:44Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-AttachVolumeShelveTestJSON-volume-7555594',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=b4281d17-c1d8-48d3-8a0b-8fd7d2f193cc,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='214a0f233f114d7aa22e64048a837c0b',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:00:44Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:00:44Z,use_quota=True,user_id='83b9d77de46143d8b9277a55f5ad4d72',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(b4281d17-c1d8-48d3-8a0b-8fd7d2f193cc),volume_id=b4281d17-c1d8-48d3-8a0b-8fd7d2f193cc,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': 'b4281d17-c1d8-48d3-8a0b-8fd7d2f193cc', 'volume_name': 'volume-b4281d17-c1d8-48d3-8a0b-8fd7d2f193cc', 'volume_size': 1}} Apr 17 16:03:01 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'retry': {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']}, 'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': 'b4281d17-c1d8-48d3-8a0b-8fd7d2f193cc', 'volume': {'id': 'b4281d17-c1d8-48d3-8a0b-8fd7d2f193cc', '_name_id': None, 'ec2_id': None, 'user_id': '83b9d77de46143d8b9277a55f5ad4d72', 'project_id': '214a0f233f114d7aa22e64048a837c0b', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-AttachVolumeShelveTestJSON-volume-7555594', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T16:00:44.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-b4281d17-c1d8-48d3-8a0b-8fd7d2f193cc', 'name_id': 'b4281d17-c1d8-48d3-8a0b-8fd7d2f193cc', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-AttachVolumeShelveTestJSON-volume-7555594', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '214a0f233f114d7aa22e64048a837c0b', 'qos_specs': None, 'reservations': ['d9fcea02-28d2-4dba-a68b-5de4e249400a', 'dc207600-1b77-4709-9acd-aacbcf21051f', '891fc9c0-b5b9-44e3-a1c8-d30fcbfd5cc4', '8f4a2d2a-cb05-4ac0-b722-22369386396e'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': '83b9d77de46143d8b9277a55f5ad4d72', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-AttachVolumeShelveTestJSON-volume-7555594', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '214a0f233f114d7aa22e64048a837c0b', 'qos_specs': None, 'reservations': ['d9fcea02-28d2-4dba-a68b-5de4e249400a', 'dc207600-1b77-4709-9acd-aacbcf21051f', '891fc9c0-b5b9-44e3-a1c8-d30fcbfd5cc4', '8f4a2d2a-cb05-4ac0-b722-22369386396e'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': '83b9d77de46143d8b9277a55f5ad4d72', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': '83b9d77de46143d8b9277a55f5ad4d72', 'metadata': {}, 'qos_specs': None}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(b4281d17-c1d8-48d3-8a0b-8fd7d2f193cc),volume_id=b4281d17-c1d8-48d3-8a0b-8fd7d2f193cc,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:00:44Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-AttachVolumeShelveTestJSON-volume-7555594',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=b4281d17-c1d8-48d3-8a0b-8fd7d2f193cc,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='214a0f233f114d7aa22e64048a837c0b',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:00:44Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:00:44Z,use_quota=True,user_id='83b9d77de46143d8b9277a55f5ad4d72',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 16:03:01 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:00:44Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-AttachVolumeShelveTestJSON-volume-7555594',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=b4281d17-c1d8-48d3-8a0b-8fd7d2f193cc,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='214a0f233f114d7aa22e64048a837c0b',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:00:44Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:00:44Z,use_quota=True,user_id='83b9d77de46143d8b9277a55f5ad4d72',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:00:44Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-AttachVolumeShelveTestJSON-volume-7555594',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=b4281d17-c1d8-48d3-8a0b-8fd7d2f193cc,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='214a0f233f114d7aa22e64048a837c0b',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:00:44Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:00:44Z,use_quota=True,user_id='83b9d77de46143d8b9277a55f5ad4d72',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 16:03:01 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 16:03:01 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 16:03:01 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:03:01 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 16:03:01 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:03:01 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 16:03:01 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:03:01 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 16:03:01 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:03:01 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 16:03:01 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:03:01 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 16:03:01 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:03:01 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 16:03:01 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:03:01 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 16:03:01 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:03:01 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:03:01 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 16:03:01 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 16:03:01 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (3607be78-7861-4ce5-9705-ae5cc87302c3) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:03:01 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (3607be78-7861-4ce5-9705-ae5cc87302c3) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:03:01 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (e954b9ad-7c98-4606-9049-528002a22b12) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:03:01 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (e954b9ad-7c98-4606-9049-528002a22b12) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:03:01 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (62588e89-7ca3-4319-8090-260c64894cac) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:03:01 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (62588e89-7ca3-4319-8090-260c64894cac) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:03:01 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (d1d2659c-e868-4ea2-a545-fe10a196244c) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:03:01 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Updating volume b4281d17-c1d8-48d3-8a0b-8fd7d2f193cc with {'scheduled_at': datetime.datetime(2023, 4, 17, 16, 3, 1, 926073), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 16:03:01 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Volume b4281d17-c1d8-48d3-8a0b-8fd7d2f193cc: re-scheduling SchedulerAPI.create_volume attempt 1 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 16:03:01 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Volume b4281d17-c1d8-48d3-8a0b-8fd7d2f193cc: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 16:03:01 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] deleating volume v_b4281d17-c1d8-48d3-8a0b-8fd7d2f193cc {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 16:03:01 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] get snapshots for volume v_b4281d17-c1d8-48d3-8a0b-8fd7d2f193cc {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 16:03:01 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_b4281d17-c1d8-48d3-8a0b-8fd7d2f193cc/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:03:01 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_b4281d17-c1d8-48d3-8a0b-8fd7d2f193cc/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:03:02 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Volume reschedule parameters: True retry: {'num_attempts': 2, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 16:03:02 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Flow 'volume_create_manager' (3263a579-5dd0-479b-a8d4-fb022da8fc88) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 16:03:02 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (ceb3ed76-d61c-43e2-a2f6-5b95b4b62d52) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:03:02 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (ceb3ed76-d61c-43e2-a2f6-5b95b4b62d52) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:00:44Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-AttachVolumeShelveTestJSON-volume-7555594',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=b4281d17-c1d8-48d3-8a0b-8fd7d2f193cc,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='214a0f233f114d7aa22e64048a837c0b',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:03:02Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:03:02Z,use_quota=True,user_id='83b9d77de46143d8b9277a55f5ad4d72',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:03:02 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (1bd6d8b5-327c-4110-970e-7586cb828ec7) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:03:02 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (1bd6d8b5-327c-4110-970e-7586cb828ec7) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:03:02 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (a928dde2-794b-4453-83d3-3360d279ddd4) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:03:02 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (a928dde2-794b-4453-83d3-3360d279ddd4) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': 'b4281d17-c1d8-48d3-8a0b-8fd7d2f193cc', 'volume_name': 'volume-b4281d17-c1d8-48d3-8a0b-8fd7d2f193cc', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:03:02 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (b662187d-b791-4e86-92b5-dde5cb5f9604) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:03:02 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (b662187d-b791-4e86-92b5-dde5cb5f9604) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:03:02 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (528895d8-d3de-4ab5-be8d-6def8d3366a6) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:03:02 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Volume b4281d17-c1d8-48d3-8a0b-8fd7d2f193cc: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-b4281d17-c1d8-48d3-8a0b-8fd7d2f193cc', 'volume_size': 1} Apr 17 16:03:02 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] creating volume v_b4281d17-c1d8-48d3-8a0b-8fd7d2f193cc. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 16:03:02 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] create volume {'name': 'v_b4281d17-c1d8-48d3-8a0b-8fd7d2f193cc', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 16:03:02 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:03:02 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:03:04 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=23,cinder:UPDATE=7,cinder:INSERT=1 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:03:04 user cinder-volume[84696]: DEBUG cinder.utils [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 40.210(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:03:04 user cinder-volume[84696]: DEBUG cinder.utils [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:03:04 user cinder-volume[84696]: DEBUG cinder.utils [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 40.454(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:03:04 user cinder-volume[84696]: DEBUG cinder.utils [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_9a399a62-6a8f-4b6d-8ecf-8dcdf57e5fbb/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:03:04 user cinder-volume[84696]: DEBUG cinder.utils [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.778(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:03:04 user cinder-volume[84696]: DEBUG cinder.utils [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:03:04 user cinder-volume[84696]: DEBUG cinder.utils [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 3.428(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:03:04 user cinder-volume[84696]: DEBUG cinder.utils [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_40efe2ca-3a61-44c9-b9e0-bb29b333f0f9/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:03:04 user cinder-volume[84696]: DEBUG cinder.utils [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 4.696(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:03:04 user cinder-volume[84696]: DEBUG cinder.utils [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_40efe2ca-3a61-44c9-b9e0-bb29b333f0f9/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:03:04 user cinder-volume[84696]: DEBUG cinder.utils [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.977(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:03:04 user cinder-volume[84696]: DEBUG cinder.utils [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_b4281d17-c1d8-48d3-8a0b-8fd7d2f193cc/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:03:04 user cinder-volume[84696]: DEBUG cinder.utils [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.053(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:03:04 user cinder-volume[84696]: DEBUG cinder.utils [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_40efe2ca-3a61-44c9-b9e0-bb29b333f0f9/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:03:05 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Volume reschedule parameters: True retry: {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 16:03:05 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Flow 'volume_create_manager' (d27fdfd8-f28b-4321-a429-36deaa0990f5) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 16:03:05 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (3ccbe5ca-8953-48f7-86fc-86df8aaeb3cf) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:03:05 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (3ccbe5ca-8953-48f7-86fc-86df8aaeb3cf) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:03:06Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesListTestJSON-Volume-1309258736',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=4d960aac-48dd-470e-831f-eaafe64c9fdf,launched_at=None,metadata={Type='work'},migration_status=None,multiattach=False,previous_status=None,project_id='d1740791cf5e4c7087f45af9015c0a8f',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:03:06Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:03:06Z,use_quota=True,user_id='9091747614a945f6a88207d3662fea8e',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:03:05 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (d39f28e2-aa96-4bf8-bb1d-39acdf57545c) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:03:05 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (d39f28e2-aa96-4bf8-bb1d-39acdf57545c) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:03:05 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (171d181d-4cdf-41de-b293-bb2ec52ad422) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:03:05 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (171d181d-4cdf-41de-b293-bb2ec52ad422) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': '4d960aac-48dd-470e-831f-eaafe64c9fdf', 'volume_name': 'volume-4d960aac-48dd-470e-831f-eaafe64c9fdf', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:03:05 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (e45ec5db-21cf-4496-a5d7-1a524ae82306) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:03:05 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (e45ec5db-21cf-4496-a5d7-1a524ae82306) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:03:05 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (f8adf837-7376-42e2-8183-23cda829175a) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:03:05 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Volume 4d960aac-48dd-470e-831f-eaafe64c9fdf: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-4d960aac-48dd-470e-831f-eaafe64c9fdf', 'volume_size': 1} Apr 17 16:03:05 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] creating volume v_4d960aac-48dd-470e-831f-eaafe64c9fdf. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 16:03:05 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] create volume {'name': 'v_4d960aac-48dd-470e-831f-eaafe64c9fdf', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 16:03:05 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:03:05 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:03:07 user cinder-volume[84696]: DEBUG cinder.utils [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.848(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:03:07 user cinder-volume[84696]: DEBUG cinder.utils [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:03:07 user cinder-volume[84696]: DEBUG cinder.utils [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.118(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:03:07 user cinder-volume[84696]: DEBUG cinder.utils [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:03:08 user cinder-volume[84696]: DEBUG cinder.utils [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.047(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:03:08 user cinder-volume[84696]: DEBUG cinder.utils [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_b4281d17-c1d8-48d3-8a0b-8fd7d2f193cc/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:03:11 user cinder-volume[84696]: DEBUG cinder.utils [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 9.568(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:03:11 user cinder-volume[84696]: DEBUG cinder.utils [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_40efe2ca-3a61-44c9-b9e0-bb29b333f0f9/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:03:11 user cinder-volume[84696]: DEBUG cinder.utils [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 10.837(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:03:11 user cinder-volume[84696]: DEBUG cinder.utils [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_40efe2ca-3a61-44c9-b9e0-bb29b333f0f9/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:03:11 user cinder-volume[84696]: DEBUG cinder.utils [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.192(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:03:11 user cinder-volume[84696]: DEBUG cinder.utils [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:03:11 user cinder-volume[84696]: DEBUG cinder.utils [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.195(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:03:11 user cinder-volume[84696]: DEBUG cinder.utils [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_40efe2ca-3a61-44c9-b9e0-bb29b333f0f9/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:03:14 user cinder-volume[84696]: DEBUG cinder.utils [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 11.992(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:03:14 user cinder-volume[84696]: DEBUG cinder.utils [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:03:14 user cinder-volume[84696]: DEBUG cinder.utils [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.188(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:03:14 user cinder-volume[84696]: DEBUG cinder.utils [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_b4281d17-c1d8-48d3-8a0b-8fd7d2f193cc/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:03:17 user cinder-volume[84696]: DEBUG cinder.utils [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 11.333(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:03:17 user cinder-volume[84696]: DEBUG cinder.utils [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:03:20 user cinder-volume[84696]: DEBUG cinder.utils [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 74.274(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:03:20 user cinder-volume[84696]: DEBUG cinder.utils [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_26014b4e-ec84-4011-a69f-ed42c3424a24/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:03:20 user cinder-volume[84696]: DEBUG cinder.utils [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 74.097(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:03:20 user cinder-volume[84696]: DEBUG cinder.utils [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:03:20 user cinder-volume[84696]: DEBUG cinder.utils [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 18.788(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:03:20 user cinder-volume[84696]: DEBUG cinder.utils [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_40efe2ca-3a61-44c9-b9e0-bb29b333f0f9/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:03:20 user cinder-volume[84696]: DEBUG cinder.utils [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.410(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:03:20 user cinder-volume[84696]: DEBUG cinder.utils [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_40efe2ca-3a61-44c9-b9e0-bb29b333f0f9/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:03:20 user cinder-volume[84696]: DEBUG cinder.utils [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 20.057(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:03:20 user cinder-volume[84696]: DEBUG cinder.utils [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_40efe2ca-3a61-44c9-b9e0-bb29b333f0f9/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:03:23 user cinder-volume[84696]: DEBUG cinder.utils [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.403(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:03:23 user cinder-volume[84696]: DEBUG cinder.utils [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_b4281d17-c1d8-48d3-8a0b-8fd7d2f193cc/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:03:23 user cinder-volume[84696]: DEBUG cinder.utils [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.209(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:03:23 user cinder-volume[84696]: DEBUG cinder.utils [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:03:24 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=33,cinder:UPDATE=7,cinder:INSERT=4 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:03:26 user cinder-volume[84696]: DEBUG cinder.utils [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 20.549(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:03:26 user cinder-volume[84696]: DEBUG cinder.utils [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:03:26 user cinder-volume[84696]: DEBUG cinder.utils [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 140.046(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:03:26 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_bff6fdbb-a1fb-4b48-bca8-2f7efbc3f0ff/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:03:26 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (51865b1d-35c0-44f9-8c4d-2f75409a79d7) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 16:03:26 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (b2c35c35-5246-49c2-98d9-d0327dbbde2b) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:03:26 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (b2c35c35-5246-49c2-98d9-d0327dbbde2b) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:03:26 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Flow 'volume_create_manager' (9b1aa7ce-a821-42f3-a57a-8536cbc161b7) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 16:03:26 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 16:03:26 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 16:03:26 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 16:03:26 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 16:03:26 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 16:03:26 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 16:03:26 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 16:03:26 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 16:03:26 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 16:03:26 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 16:03:26 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 16:03:26 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 16:03:26 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 16:03:26 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 16:03:26 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 16:03:26 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 16:03:26 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 16:03:26 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 16:03:26 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 16:03:26 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 16:03:26 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 16:03:26 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 16:03:26 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 16:03:26 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 16:03:26 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:03:26 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 16:03:26 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:03:26 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 16:03:26 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:03:26 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 16:03:26 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:03:26 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 16:03:26 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:03:26 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 16:03:26 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:03:26 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 16:03:26 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:03:26 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 16:03:26 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:03:26 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:03:26 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 16:03:26 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 16:03:26 user cinder-volume[84696]: DEBUG cinder.utils [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 139.889(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:03:26 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:03:26 user cinder-volume[84696]: INFO cinder.message.api [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Creating message record for request_id = req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 Apr 17 16:03:26 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 16:03:26 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (5d26daf0-e780-4ae9-8b4a-8f83e3f0f239) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 16:03:26 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 16:03:26 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:56:26Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumeRetypeWithoutMigrationTest-Volume-636058295',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=bff6fdbb-a1fb-4b48-bca8-2f7efbc3f0ff,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='46c9cac22c32484181c934b27e67f512',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:01:06Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:01:06Z,use_quota=True,user_id='a0751e2174804d7ab462edae5f88d581',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(c9c52f37-7390-466f-a6ab-56ae9a61e5ed),volume_type_id=c9c52f37-7390-466f-a6ab-56ae9a61e5ed)}, 'provides': None} Apr 17 16:03:26 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:56:26Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumeRetypeWithoutMigrationTest-Volume-636058295',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=bff6fdbb-a1fb-4b48-bca8-2f7efbc3f0ff,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='46c9cac22c32484181c934b27e67f512',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:01:06Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:01:06Z,use_quota=True,user_id='a0751e2174804d7ab462edae5f88d581',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(c9c52f37-7390-466f-a6ab-56ae9a61e5ed),volume_type_id=c9c52f37-7390-466f-a6ab-56ae9a61e5ed), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(bff6fdbb-a1fb-4b48-bca8-2f7efbc3f0ff),volume_id=bff6fdbb-a1fb-4b48-bca8-2f7efbc3f0ff,volume_properties=VolumeProperties,volume_type=VolumeType(c9c52f37-7390-466f-a6ab-56ae9a61e5ed))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': 'bff6fdbb-a1fb-4b48-bca8-2f7efbc3f0ff', 'volume_name': 'volume-bff6fdbb-a1fb-4b48-bca8-2f7efbc3f0ff', 'volume_size': 1}} Apr 17 16:03:26 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': 'bff6fdbb-a1fb-4b48-bca8-2f7efbc3f0ff', 'volume': {'id': 'bff6fdbb-a1fb-4b48-bca8-2f7efbc3f0ff', '_name_id': None, 'ec2_id': None, 'user_id': 'a0751e2174804d7ab462edae5f88d581', 'project_id': '46c9cac22c32484181c934b27e67f512', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-VolumeRetypeWithoutMigrationTest-Volume-636058295', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': 'c9c52f37-7390-466f-a6ab-56ae9a61e5ed', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'admin_metadata': {}, 'glance_metadata': {}, 'volume_type': {'id': 'c9c52f37-7390-466f-a6ab-56ae9a61e5ed', 'name': 'volume-type-1', 'description': None, 'is_public': True, 'extra_specs': {}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:56:25.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T15:56:25.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-bff6fdbb-a1fb-4b48-bca8-2f7efbc3f0ff', 'name_id': 'bff6fdbb-a1fb-4b48-bca8-2f7efbc3f0ff', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': 'c9c52f37-7390-466f-a6ab-56ae9a61e5ed', 'name': 'volume-type-1', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:56:25.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumeRetypeWithoutMigrationTest-Volume-636058295', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '46c9cac22c32484181c934b27e67f512', 'qos_specs': None, 'reservations': ['622ac28a-652f-4d3c-b14a-4e0cf9309855', '3f27124e-4ba9-448f-a619-0c453485c180', '93656d14-fa21-40b5-acf9-87e241ec0ef5', 'a859cf24-fdaa-4c02-a26d-31ecb6289b76'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'a0751e2174804d7ab462edae5f88d581', 'volume_type_id': 'c9c52f37-7390-466f-a6ab-56ae9a61e5ed'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumeRetypeWithoutMigrationTest-Volume-636058295', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '46c9cac22c32484181c934b27e67f512', 'qos_specs': None, 'reservations': ['622ac28a-652f-4d3c-b14a-4e0cf9309855', '3f27124e-4ba9-448f-a619-0c453485c180', '93656d14-fa21-40b5-acf9-87e241ec0ef5', 'a859cf24-fdaa-4c02-a26d-31ecb6289b76'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'a0751e2174804d7ab462edae5f88d581', 'volume_type_id': 'c9c52f37-7390-466f-a6ab-56ae9a61e5ed'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:56:25Z,deleted=False,deleted_at=None,description=None,extra_specs={},id=c9c52f37-7390-466f-a6ab-56ae9a61e5ed,is_public=True,name='volume-type-1',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:56:25Z,deleted=False,deleted_at=None,description=None,extra_specs={},id=c9c52f37-7390-466f-a6ab-56ae9a61e5ed,is_public=True,name='volume-type-1',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': 'a0751e2174804d7ab462edae5f88d581', 'metadata': {}, 'qos_specs': None, 'retry': {'num_attempts': 3, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']}}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(bff6fdbb-a1fb-4b48-bca8-2f7efbc3f0ff),volume_id=bff6fdbb-a1fb-4b48-bca8-2f7efbc3f0ff,volume_properties=VolumeProperties,volume_type=VolumeType(c9c52f37-7390-466f-a6ab-56ae9a61e5ed)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:56:26Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumeRetypeWithoutMigrationTest-Volume-636058295',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=bff6fdbb-a1fb-4b48-bca8-2f7efbc3f0ff,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='46c9cac22c32484181c934b27e67f512',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:01:06Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:01:06Z,use_quota=True,user_id='a0751e2174804d7ab462edae5f88d581',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(c9c52f37-7390-466f-a6ab-56ae9a61e5ed),volume_type_id=c9c52f37-7390-466f-a6ab-56ae9a61e5ed), 'context': }, 'provides': None} Apr 17 16:03:26 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:56:26Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumeRetypeWithoutMigrationTest-Volume-636058295',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=bff6fdbb-a1fb-4b48-bca8-2f7efbc3f0ff,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='46c9cac22c32484181c934b27e67f512',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:01:06Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:01:06Z,use_quota=True,user_id='a0751e2174804d7ab462edae5f88d581',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(c9c52f37-7390-466f-a6ab-56ae9a61e5ed),volume_type_id=c9c52f37-7390-466f-a6ab-56ae9a61e5ed)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:56:26Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumeRetypeWithoutMigrationTest-Volume-636058295',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=bff6fdbb-a1fb-4b48-bca8-2f7efbc3f0ff,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='46c9cac22c32484181c934b27e67f512',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:01:06Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:01:06Z,use_quota=True,user_id='a0751e2174804d7ab462edae5f88d581',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(c9c52f37-7390-466f-a6ab-56ae9a61e5ed),volume_type_id=c9c52f37-7390-466f-a6ab-56ae9a61e5ed)} Apr 17 16:03:26 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 16:03:26 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 16:03:26 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:03:26 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 16:03:26 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:03:26 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 16:03:26 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:03:26 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 16:03:26 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:03:26 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 16:03:26 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:03:26 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 16:03:26 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:03:26 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 16:03:26 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:03:26 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 16:03:26 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:03:26 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:03:26 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 16:03:26 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 16:03:26 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (5d26daf0-e780-4ae9-8b4a-8f83e3f0f239) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:03:26 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (5d26daf0-e780-4ae9-8b4a-8f83e3f0f239) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:03:26 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (0c941742-3688-4923-a159-ff84eba2e1e8) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:03:26 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (0c941742-3688-4923-a159-ff84eba2e1e8) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:03:26 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (52b4f907-04c4-43dd-8f00-116ae1cebc2e) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:03:26 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (52b4f907-04c4-43dd-8f00-116ae1cebc2e) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:03:26 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (0fea95b6-4035-4c1d-b02e-1897705e16da) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:03:26 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Updating volume bff6fdbb-a1fb-4b48-bca8-2f7efbc3f0ff with {'scheduled_at': datetime.datetime(2023, 4, 17, 16, 3, 26, 514492), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 16:03:26 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Volume bff6fdbb-a1fb-4b48-bca8-2f7efbc3f0ff: re-scheduling SchedulerAPI.create_volume attempt 3 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 16:03:26 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Volume bff6fdbb-a1fb-4b48-bca8-2f7efbc3f0ff: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 16:03:26 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] deleating volume v_bff6fdbb-a1fb-4b48-bca8-2f7efbc3f0ff {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 16:03:26 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] get snapshots for volume v_bff6fdbb-a1fb-4b48-bca8-2f7efbc3f0ff {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 16:03:26 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_bff6fdbb-a1fb-4b48-bca8-2f7efbc3f0ff/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:03:26 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_bff6fdbb-a1fb-4b48-bca8-2f7efbc3f0ff/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:03:29 user cinder-volume[84696]: DEBUG cinder.utils [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.958(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:03:29 user cinder-volume[84696]: DEBUG cinder.utils [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_bff6fdbb-a1fb-4b48-bca8-2f7efbc3f0ff/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:03:29 user cinder-volume[84696]: DEBUG cinder.utils [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 74.251(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:03:29 user cinder-volume[84696]: DEBUG cinder.utils [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_87514055-5a0f-459f-b10b-438d7a13d39d/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:03:32 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 140.100(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:03:32 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:03:32 user cinder-volume[84696]: ERROR oslo_service.periodic_task [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Error during VolumeManager.publish_service_capabilities: TypeError: 'NoneType' object is not subscriptable Apr 17 16:03:32 user cinder-volume[84696]: ERROR oslo_service.periodic_task Traceback (most recent call last): Apr 17 16:03:32 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/usr/local/lib/python3.10/dist-packages/oslo_service/periodic_task.py", line 216, in run_periodic_tasks Apr 17 16:03:32 user cinder-volume[84696]: ERROR oslo_service.periodic_task task(self, context) Apr 17 16:03:32 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/manager.py", line 2876, in publish_service_capabilities Apr 17 16:03:32 user cinder-volume[84696]: ERROR oslo_service.periodic_task self._report_driver_status(context) Apr 17 16:03:32 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/manager.py", line 2755, in _report_driver_status Apr 17 16:03:32 user cinder-volume[84696]: ERROR oslo_service.periodic_task volume_stats = get_stats() Apr 17 16:03:32 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/usr/local/lib/python3.10/dist-packages/oslo_utils/timeutils.py", line 319, in wrapper Apr 17 16:03:32 user cinder-volume[84696]: ERROR oslo_service.periodic_task result = func(*args, **kwargs) Apr 17 16:03:32 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/manager.py", line 2753, in get_stats Apr 17 16:03:32 user cinder-volume[84696]: ERROR oslo_service.periodic_task return self.driver.get_volume_stats(refresh=True) Apr 17 16:03:32 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/driver.py", line 666, in get_volume_stats Apr 17 16:03:32 user cinder-volume[84696]: ERROR oslo_service.periodic_task self._update_volume_stats() Apr 17 16:03:32 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 726, in _update_volume_stats Apr 17 16:03:32 user cinder-volume[84696]: ERROR oslo_service.periodic_task pool_stats = self.ra.get_pool_stats() Apr 17 16:03:32 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 897, in get_pool_stats Apr 17 16:03:32 user cinder-volume[84696]: ERROR oslo_service.periodic_task if not resp["error"] and resp["code"] == 200: Apr 17 16:03:32 user cinder-volume[84696]: ERROR oslo_service.periodic_task TypeError: 'NoneType' object is not subscriptable Apr 17 16:03:32 user cinder-volume[84696]: ERROR oslo_service.periodic_task Apr 17 16:03:32 user cinder-volume[84696]: DEBUG cinder.utils [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 139.746(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:03:32 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:03:32 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (657e58ac-4118-460b-a2d0-e5df1d120a72) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 16:03:32 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 16:03:32 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:58:54Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TestVolumeBootPattern-volume-origin-619636601',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=fd313648-5fc1-4e6a-97a0-92ac803b36fd,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='c8a836b47d954cb9a7c1cd7c0f790426',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:01:13Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:01:13Z,use_quota=True,user_id='a80e753322224ec1a897598b5b9ed6ea',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 16:03:32 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:58:54Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TestVolumeBootPattern-volume-origin-619636601',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=fd313648-5fc1-4e6a-97a0-92ac803b36fd,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='c8a836b47d954cb9a7c1cd7c0f790426',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:01:13Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:01:13Z,use_quota=True,user_id='a80e753322224ec1a897598b5b9ed6ea',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=b3f0dec1-7758-44a8-82e0-76ada8837428,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(fd313648-5fc1-4e6a-97a0-92ac803b36fd),volume_id=fd313648-5fc1-4e6a-97a0-92ac803b36fd,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'image', 'volume_id': 'fd313648-5fc1-4e6a-97a0-92ac803b36fd', 'volume_name': 'volume-fd313648-5fc1-4e6a-97a0-92ac803b36fd', 'volume_size': 1, 'image_id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'image_location': (None, None), 'image_meta': {'name': 'cirros-0.5.2-x86_64-disk', 'disk_format': 'qcow2', 'container_format': 'bare', 'visibility': 'public', 'size': 16300544, 'virtual_size': 117440512, 'status': 'active', 'checksum': 'b874c39491a2377b8490f5f1e89761a4', 'protected': False, 'min_ram': 0, 'min_disk': 0, 'owner': 'a43c211b4ed542538799349489b1d884', 'os_hidden': False, 'os_hash_algo': 'sha512', 'os_hash_value': '6b813aa46bb90b4da216a4d19376593fa3f4fc7e617f03a92b7fe11e9a3981cbe8f0959dbebe36225e5f53dc4492341a4863cac4ed1ee0909f3fc78ef9c3e869', 'id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'created_at': datetime.datetime(2023, 4, 17, 15, 19, 53, tzinfo=datetime.timezone.utc), 'updated_at': datetime.datetime(2023, 4, 17, 15, 19, 55, tzinfo=datetime.timezone.utc), 'tags': [], 'file': '/v2/images/b3f0dec1-7758-44a8-82e0-76ada8837428/file', 'properties': {'hw_rng_model': 'virtio', 'owner_specified.openstack.md5': '', 'owner_specified.openstack.object': 'images/cirros-0.5.2-x86_64-disk', 'owner_specified.openstack.sha256': ''}}, 'image_service': }} Apr 17 16:03:32 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'snapshot_id': None, 'source_volid': None, 'volume_id': 'fd313648-5fc1-4e6a-97a0-92ac803b36fd', 'volume': {'id': 'fd313648-5fc1-4e6a-97a0-92ac803b36fd', '_name_id': None, 'ec2_id': None, 'user_id': 'a80e753322224ec1a897598b5b9ed6ea', 'project_id': 'c8a836b47d954cb9a7c1cd7c0f790426', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-TestVolumeBootPattern-volume-origin-619636601', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'admin_metadata': {}, 'glance_metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T15:58:53.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-fd313648-5fc1-4e6a-97a0-92ac803b36fd', 'name_id': 'fd313648-5fc1-4e6a-97a0-92ac803b36fd', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-TestVolumeBootPattern-volume-origin-619636601', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': 'c8a836b47d954cb9a7c1cd7c0f790426', 'qos_specs': None, 'reservations': ['85aec33d-518b-4853-aab2-fb3033224b8f', '56b9d1aa-d720-448b-857d-904a1f5517db', 'acdbd69e-bf21-488b-9caa-1525a1d521b2', 'd0d43e22-774a-4a41-86f9-bcde7e88d4b7'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'a80e753322224ec1a897598b5b9ed6ea', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-TestVolumeBootPattern-volume-origin-619636601', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': 'c8a836b47d954cb9a7c1cd7c0f790426', 'qos_specs': None, 'reservations': ['85aec33d-518b-4853-aab2-fb3033224b8f', '56b9d1aa-d720-448b-857d-904a1f5517db', 'acdbd69e-bf21-488b-9caa-1525a1d521b2', 'd0d43e22-774a-4a41-86f9-bcde7e88d4b7'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'a80e753322224ec1a897598b5b9ed6ea', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': 'a80e753322224ec1a897598b5b9ed6ea', 'metadata': {}, 'qos_specs': None, 'retry': {'num_attempts': 2, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']}}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=b3f0dec1-7758-44a8-82e0-76ada8837428,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(fd313648-5fc1-4e6a-97a0-92ac803b36fd),volume_id=fd313648-5fc1-4e6a-97a0-92ac803b36fd,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:58:54Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TestVolumeBootPattern-volume-origin-619636601',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=fd313648-5fc1-4e6a-97a0-92ac803b36fd,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='c8a836b47d954cb9a7c1cd7c0f790426',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:01:13Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:01:13Z,use_quota=True,user_id='a80e753322224ec1a897598b5b9ed6ea',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 16:03:32 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:58:54Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TestVolumeBootPattern-volume-origin-619636601',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=fd313648-5fc1-4e6a-97a0-92ac803b36fd,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='c8a836b47d954cb9a7c1cd7c0f790426',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:01:13Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:01:13Z,use_quota=True,user_id='a80e753322224ec1a897598b5b9ed6ea',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:58:54Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TestVolumeBootPattern-volume-origin-619636601',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=fd313648-5fc1-4e6a-97a0-92ac803b36fd,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='c8a836b47d954cb9a7c1cd7c0f790426',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:01:13Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:01:13Z,use_quota=True,user_id='a80e753322224ec1a897598b5b9ed6ea',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 16:03:32 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 16:03:32 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 16:03:32 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:03:32 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 16:03:32 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1237, in execute Apr 17 16:03:32 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_from_image(context, Apr 17 16:03:32 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/utils.py", line 670, in _wrapper Apr 17 16:03:32 user cinder-volume[84696]: ERROR cinder.volume.manager return r(f, *args, **kwargs) Apr 17 16:03:32 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/tenacity/__init__.py", line 423, in __call__ Apr 17 16:03:32 user cinder-volume[84696]: ERROR cinder.volume.manager do = self.iter(retry_state=retry_state) Apr 17 16:03:32 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/tenacity/__init__.py", line 360, in iter Apr 17 16:03:32 user cinder-volume[84696]: ERROR cinder.volume.manager return fut.result() Apr 17 16:03:32 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/lib/python3.10/concurrent/futures/_base.py", line 451, in result Apr 17 16:03:32 user cinder-volume[84696]: ERROR cinder.volume.manager return self.__get_result() Apr 17 16:03:32 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result Apr 17 16:03:32 user cinder-volume[84696]: ERROR cinder.volume.manager raise self._exception Apr 17 16:03:32 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/tenacity/__init__.py", line 426, in __call__ Apr 17 16:03:32 user cinder-volume[84696]: ERROR cinder.volume.manager result = fn(*args, **kwargs) Apr 17 16:03:32 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1125, in _create_from_image Apr 17 16:03:32 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_from_image_cache_or_download( Apr 17 16:03:32 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 944, in _create_from_image_cache_or_download Apr 17 16:03:32 user cinder-volume[84696]: ERROR cinder.volume.manager model_update, cloned = self._create_from_image_cache( Apr 17 16:03:32 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 843, in _create_from_image_cache Apr 17 16:03:32 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_from_source_volume( Apr 17 16:03:32 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 659, in _create_from_source_volume Apr 17 16:03:32 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self.driver.create_cloned_volume(volume, srcvol_ref) Apr 17 16:03:32 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 525, in create_cloned_volume Apr 17 16:03:32 user cinder-volume[84696]: ERROR cinder.volume.manager self._clone_object(vname, cvname) Apr 17 16:03:32 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 461, in _clone_object Apr 17 16:03:32 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_snapshot(oname, coname) Apr 17 16:03:32 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 639, in create_snapshot Apr 17 16:03:32 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201, 204): Apr 17 16:03:32 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 16:03:32 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 16:03:32 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (657e58ac-4118-460b-a2d0-e5df1d120a72) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:03:32 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (657e58ac-4118-460b-a2d0-e5df1d120a72) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:03:32 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (96b75f01-0cc2-40ce-8186-8bece58c13ec) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:03:32 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (96b75f01-0cc2-40ce-8186-8bece58c13ec) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:03:32 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (50813848-b19a-4cf8-8eb5-7d5ad7a084f2) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:03:32 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (50813848-b19a-4cf8-8eb5-7d5ad7a084f2) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:03:32 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (63d555cc-1ab2-4e6a-b847-e940a6de6d3a) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:03:32 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Updating volume fd313648-5fc1-4e6a-97a0-92ac803b36fd with {'scheduled_at': datetime.datetime(2023, 4, 17, 16, 3, 32, 627269), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 16:03:32 user cinder-volume[84696]: DEBUG cinder.utils [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.087(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:03:32 user cinder-volume[84696]: DEBUG cinder.utils [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_bff6fdbb-a1fb-4b48-bca8-2f7efbc3f0ff/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:03:32 user cinder-volume[84696]: DEBUG cinder.utils [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 140.134(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:03:32 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_fd313648-5fc1-4e6a-97a0-92ac803b36fd/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:03:32 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (435cbcff-be2f-4bb7-bcc2-abf1daaaf114) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 16:03:32 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (f420ec26-2fc9-4a88-a8f8-96c2cebdacbb) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:03:32 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (f420ec26-2fc9-4a88-a8f8-96c2cebdacbb) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:03:32 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Flow 'volume_create_manager' (0c6cf15d-a93f-4047-8d8d-426ad50dd3b5) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 16:03:32 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 16:03:32 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 16:03:32 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 16:03:32 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 16:03:32 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 16:03:32 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 16:03:32 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 16:03:32 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 16:03:32 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 16:03:32 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 16:03:32 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 16:03:32 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 16:03:32 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 16:03:32 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 16:03:32 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 16:03:32 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 16:03:32 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 16:03:32 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 16:03:32 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 16:03:32 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 16:03:32 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 16:03:32 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 16:03:32 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 16:03:32 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 16:03:32 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:03:32 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 16:03:32 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1237, in execute Apr 17 16:03:32 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_from_image(context, Apr 17 16:03:32 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/utils.py", line 670, in _wrapper Apr 17 16:03:32 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return r(f, *args, **kwargs) Apr 17 16:03:32 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/tenacity/__init__.py", line 423, in __call__ Apr 17 16:03:32 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server do = self.iter(retry_state=retry_state) Apr 17 16:03:32 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/tenacity/__init__.py", line 360, in iter Apr 17 16:03:32 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return fut.result() Apr 17 16:03:32 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/lib/python3.10/concurrent/futures/_base.py", line 451, in result Apr 17 16:03:32 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self.__get_result() Apr 17 16:03:32 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result Apr 17 16:03:32 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self._exception Apr 17 16:03:32 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/tenacity/__init__.py", line 426, in __call__ Apr 17 16:03:32 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = fn(*args, **kwargs) Apr 17 16:03:32 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1125, in _create_from_image Apr 17 16:03:32 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_from_image_cache_or_download( Apr 17 16:03:32 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 944, in _create_from_image_cache_or_download Apr 17 16:03:32 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update, cloned = self._create_from_image_cache( Apr 17 16:03:32 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 843, in _create_from_image_cache Apr 17 16:03:32 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_from_source_volume( Apr 17 16:03:32 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 659, in _create_from_source_volume Apr 17 16:03:32 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self.driver.create_cloned_volume(volume, srcvol_ref) Apr 17 16:03:32 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 525, in create_cloned_volume Apr 17 16:03:32 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self._clone_object(vname, cvname) Apr 17 16:03:32 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 461, in _clone_object Apr 17 16:03:32 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_snapshot(oname, coname) Apr 17 16:03:32 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 639, in create_snapshot Apr 17 16:03:32 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201, 204): Apr 17 16:03:32 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 16:03:32 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 16:03:32 user cinder-volume[84696]: DEBUG cinder.utils [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 74.314(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:03:32 user cinder-volume[84696]: DEBUG cinder.utils [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_14aa38ea-7cbb-48f7-a9aa-ffcc7b66faa0/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:03:32 user cinder-volume[84696]: WARNING oslo.service.loopingcall [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Function 'cinder.service.Service.periodic_tasks' run outlasted interval by 80.18 sec Apr 17 16:03:32 user cinder-volume[84696]: DEBUG oslo_service.periodic_task [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Running periodic task VolumeManager.publish_service_capabilities {{(pid=84696) run_periodic_tasks /usr/local/lib/python3.10/dist-packages/oslo_service/periodic_task.py:210}} Apr 17 16:03:32 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Updating volume stats {{(pid=84696) _update_volume_stats /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:724}} Apr 17 16:03:32 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Get pool Pool-0 fsprops {{(pid=84696) get_pool_stats /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:894}} Apr 17 16:03:32 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:03:32 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:03:32 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Volume fd313648-5fc1-4e6a-97a0-92ac803b36fd: re-scheduling SchedulerAPI.create_volume attempt 2 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 16:03:32 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Volume fd313648-5fc1-4e6a-97a0-92ac803b36fd: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 16:03:32 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] deleating volume v_fd313648-5fc1-4e6a-97a0-92ac803b36fd {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 16:03:32 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] get snapshots for volume v_fd313648-5fc1-4e6a-97a0-92ac803b36fd {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 16:03:32 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_fd313648-5fc1-4e6a-97a0-92ac803b36fd/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:03:32 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_fd313648-5fc1-4e6a-97a0-92ac803b36fd/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:03:32 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Volume reschedule parameters: True retry: {'num_attempts': 3, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 16:03:32 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Flow 'volume_create_manager' (d3929d41-9b1a-4572-a977-36878f2f92a3) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 16:03:32 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (6f3d172e-3e4a-41b8-a0b3-11718edd69d2) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:03:32 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (6f3d172e-3e4a-41b8-a0b3-11718edd69d2) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:58:54Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TestVolumeBootPattern-volume-origin-619636601',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=fd313648-5fc1-4e6a-97a0-92ac803b36fd,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='c8a836b47d954cb9a7c1cd7c0f790426',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:03:33Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:03:33Z,use_quota=True,user_id='a80e753322224ec1a897598b5b9ed6ea',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:03:32 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (468f07b1-f1bb-48dd-b6ea-bcae9f98018c) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:03:32 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (468f07b1-f1bb-48dd-b6ea-bcae9f98018c) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:03:32 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (0e04621c-ff7e-420c-9e64-068fc655bc83) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:03:32 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (0e04621c-ff7e-420c-9e64-068fc655bc83) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'image', 'volume_id': 'fd313648-5fc1-4e6a-97a0-92ac803b36fd', 'volume_name': 'volume-fd313648-5fc1-4e6a-97a0-92ac803b36fd', 'volume_size': 1, 'image_id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'image_location': (None, None), 'image_meta': {'name': 'cirros-0.5.2-x86_64-disk', 'disk_format': 'qcow2', 'container_format': 'bare', 'visibility': 'public', 'size': 16300544, 'virtual_size': 117440512, 'status': 'active', 'checksum': 'b874c39491a2377b8490f5f1e89761a4', 'protected': False, 'min_ram': 0, 'min_disk': 0, 'owner': 'a43c211b4ed542538799349489b1d884', 'os_hidden': False, 'os_hash_algo': 'sha512', 'os_hash_value': '6b813aa46bb90b4da216a4d19376593fa3f4fc7e617f03a92b7fe11e9a3981cbe8f0959dbebe36225e5f53dc4492341a4863cac4ed1ee0909f3fc78ef9c3e869', 'id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'created_at': datetime.datetime(2023, 4, 17, 15, 19, 53, tzinfo=datetime.timezone.utc), 'updated_at': datetime.datetime(2023, 4, 17, 15, 19, 55, tzinfo=datetime.timezone.utc), 'tags': [], 'file': '/v2/images/b3f0dec1-7758-44a8-82e0-76ada8837428/file', 'properties': {'hw_rng_model': 'virtio', 'owner_specified.openstack.md5': '', 'owner_specified.openstack.object': 'images/cirros-0.5.2-x86_64-disk', 'owner_specified.openstack.sha256': ''}}, 'image_service': }' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:03:32 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (9ed763ba-f71f-4aab-89eb-ac3e5159ed81) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:03:32 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (9ed763ba-f71f-4aab-89eb-ac3e5159ed81) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:03:32 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (faf27393-8156-4dee-bd89-b77e4119367a) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:03:32 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Volume fd313648-5fc1-4e6a-97a0-92ac803b36fd: being created as image with specification: {'status': 'creating', 'volume_name': 'volume-fd313648-5fc1-4e6a-97a0-92ac803b36fd', 'volume_size': 1, 'image_id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'image_location': (None, None), 'image_meta': {'name': 'cirros-0.5.2-x86_64-disk', 'disk_format': 'qcow2', 'container_format': 'bare', 'visibility': 'public', 'size': 16300544, 'virtual_size': 117440512, 'status': 'active', 'checksum': 'b874c39491a2377b8490f5f1e89761a4', 'protected': False, 'min_ram': 0, 'min_disk': 0, 'owner': 'a43c211b4ed542538799349489b1d884', 'os_hidden': False, 'os_hash_algo': 'sha512', 'os_hash_value': '6b813aa46bb90b4da216a4d19376593fa3f4fc7e617f03a92b7fe11e9a3981cbe8f0959dbebe36225e5f53dc4492341a4863cac4ed1ee0909f3fc78ef9c3e869', 'id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'created_at': datetime.datetime(2023, 4, 17, 15, 19, 53, tzinfo=datetime.timezone.utc), 'updated_at': datetime.datetime(2023, 4, 17, 15, 19, 55, tzinfo=datetime.timezone.utc), 'tags': [], 'file': '/v2/images/b3f0dec1-7758-44a8-82e0-76ada8837428/file', 'properties': {'hw_rng_model': 'virtio', 'owner_specified.openstack.md5': '', 'owner_specified.openstack.object': 'images/cirros-0.5.2-x86_64-disk', 'owner_specified.openstack.sha256': ''}}, 'image_service': } Apr 17 16:03:32 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Cloning fd313648-5fc1-4e6a-97a0-92ac803b36fd from image b3f0dec1-7758-44a8-82e0-76ada8837428 at location (None, None). {{(pid=84696) _create_from_image /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1071}} Apr 17 16:03:32 user cinder-volume[84696]: DEBUG cinder.coordination [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Acquiring lock "cinder-b3f0dec1-7758-44a8-82e0-76ada8837428" by "_prepare_image_cache_entry" {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:192}} Apr 17 16:03:32 user cinder-volume[84696]: DEBUG cinder.coordination [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Lock "cinder-b3f0dec1-7758-44a8-82e0-76ada8837428" acquired by "_prepare_image_cache_entry" :: waited 0.018s {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:196}} Apr 17 16:03:33 user cinder-volume[84696]: DEBUG cinder.image.cache [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Found image-volume cache entry: {'id': 1, 'image_id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'volume_id': '006f20ae-5306-44fe-afdf-e433f79284e8', 'host': 'user@joviandss-2#joviandss-2', 'size': 1, 'image_updated_at': datetime.datetime(2023, 4, 17, 15, 19, 55), 'last_used': datetime.datetime(2023, 4, 17, 16, 3, 32, 994658)}. {{(pid=84696) get_entry /opt/stack/cinder/cinder/image/cache.py:76}} Apr 17 16:03:33 user cinder-volume[84696]: DEBUG cinder.image.cache [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Image-volume cache entry image_update_at = 2023-04-17 15:19:55+00:00, requested image updated_at = 2023-04-17 15:19:55+00:00. {{(pid=84696) _should_update_entry /opt/stack/cinder/cinder/image/cache.py:259}} Apr 17 16:03:33 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Found cache entry for image = b3f0dec1-7758-44a8-82e0-76ada8837428 on host user@joviandss-2#joviandss-2. {{(pid=84696) _prepare_image_cache_entry /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:886}} Apr 17 16:03:33 user cinder-volume[84696]: DEBUG cinder.coordination [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Lock "cinder-b3f0dec1-7758-44a8-82e0-76ada8837428" released by "_prepare_image_cache_entry" :: held 0.025s {{(pid=84696) _synchronized /opt/stack/cinder/cinder/coordination.py:207}} Apr 17 16:03:33 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Attempting to retrieve cache entry for image = b3f0dec1-7758-44a8-82e0-76ada8837428 on host user@joviandss-2#joviandss-2. {{(pid=84696) _create_from_image_cache /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:827}} Apr 17 16:03:33 user cinder-volume[84696]: DEBUG cinder.image.cache [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Found image-volume cache entry: {'id': 1, 'image_id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'volume_id': '006f20ae-5306-44fe-afdf-e433f79284e8', 'host': 'user@joviandss-2#joviandss-2', 'size': 1, 'image_updated_at': datetime.datetime(2023, 4, 17, 15, 19, 55), 'last_used': datetime.datetime(2023, 4, 17, 16, 3, 33, 11452)}. {{(pid=84696) get_entry /opt/stack/cinder/cinder/image/cache.py:76}} Apr 17 16:03:33 user cinder-volume[84696]: DEBUG cinder.image.cache [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Image-volume cache entry image_update_at = 2023-04-17 15:19:55+00:00, requested image updated_at = 2023-04-17 15:19:55+00:00. {{(pid=84696) _should_update_entry /opt/stack/cinder/cinder/image/cache.py:259}} Apr 17 16:03:33 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Creating from source image-volume 006f20ae-5306-44fe-afdf-e433f79284e8 {{(pid=84696) _create_from_image_cache /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:841}} Apr 17 16:03:33 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] cloned volume 006f20ae-5306-44fe-afdf-e433f79284e8 to fd313648-5fc1-4e6a-97a0-92ac803b36fd {{(pid=84696) create_cloned_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:521}} Apr 17 16:03:33 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] cloning v_006f20ae-5306-44fe-afdf-e433f79284e8 to v_fd313648-5fc1-4e6a-97a0-92ac803b36fd {{(pid=84696) _clone_object /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:456}} Apr 17 16:03:33 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] create snapshot v_fd313648-5fc1-4e6a-97a0-92ac803b36fd {{(pid=84696) create_snapshot /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:635}} Apr 17 16:03:33 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:03:33 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:03:35 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.996(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:03:35 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:03:35 user cinder-volume[84696]: DEBUG cinder.utils [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.974(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:03:35 user cinder-volume[84696]: DEBUG cinder.utils [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_fd313648-5fc1-4e6a-97a0-92ac803b36fd/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:03:35 user cinder-volume[84696]: DEBUG cinder.utils [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.621(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:03:35 user cinder-volume[84696]: DEBUG cinder.utils [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:03:38 user cinder-volume[84696]: DEBUG cinder.utils [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 37.216(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:03:38 user cinder-volume[84696]: DEBUG cinder.utils [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_40efe2ca-3a61-44c9-b9e0-bb29b333f0f9/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:03:38 user cinder-volume[84696]: DEBUG cinder.utils [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 38.484(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:03:38 user cinder-volume[84696]: DEBUG cinder.utils [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_40efe2ca-3a61-44c9-b9e0-bb29b333f0f9/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:03:38 user cinder-volume[84696]: DEBUG cinder.utils [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.840(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:03:38 user cinder-volume[84696]: DEBUG cinder.utils [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_40efe2ca-3a61-44c9-b9e0-bb29b333f0f9/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:03:38 user cinder-volume[84696]: DEBUG cinder.utils [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 74.005(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:03:38 user cinder-volume[84696]: DEBUG cinder.utils [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:03:38 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.068(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:03:38 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:03:38 user cinder-volume[84696]: DEBUG cinder.utils [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.045(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:03:38 user cinder-volume[84696]: DEBUG cinder.utils [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_fd313648-5fc1-4e6a-97a0-92ac803b36fd/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:03:38 user cinder-volume[84696]: DEBUG cinder.utils [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 74.251(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:03:38 user cinder-volume[84696]: DEBUG cinder.utils [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_9a399a62-6a8f-4b6d-8ecf-8dcdf57e5fbb/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:03:38 user cinder-volume[84696]: DEBUG cinder.utils [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 138.936(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:03:38 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:03:38 user cinder-volume[84696]: INFO cinder.message.api [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Creating message record for request_id = req-a293a946-8aa4-47bc-8290-fe9922d7ae21 Apr 17 16:03:38 user cinder-volume[84696]: DEBUG cinder.utils [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.695(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:03:38 user cinder-volume[84696]: DEBUG cinder.utils [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:03:38 user cinder-volume[84696]: DEBUG cinder.utils [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.185(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:03:38 user cinder-volume[84696]: DEBUG cinder.utils [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_bff6fdbb-a1fb-4b48-bca8-2f7efbc3f0ff/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:03:38 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 16:03:38 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=20,cinder:UPDATE=4,cinder:INSERT=1 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:03:38 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (1c7ea1e4-da9a-4973-a73f-4bdfe10bff8a) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 16:03:38 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 16:03:38 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:01:19Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-AttachVolumeNegativeTest-volume-1267367334',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=63b2c7c3-8482-49d4-bb89-c4ccf3f2db82,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='8586cc10142d4743874f43cdb0f37aba',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:01:20Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:01:20Z,use_quota=True,user_id='9fdc7c80b4654692a52c8a4c4149324f',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 16:03:38 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:01:19Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-AttachVolumeNegativeTest-volume-1267367334',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=63b2c7c3-8482-49d4-bb89-c4ccf3f2db82,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='8586cc10142d4743874f43cdb0f37aba',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:01:20Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:01:20Z,use_quota=True,user_id='9fdc7c80b4654692a52c8a4c4149324f',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(63b2c7c3-8482-49d4-bb89-c4ccf3f2db82),volume_id=63b2c7c3-8482-49d4-bb89-c4ccf3f2db82,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': '63b2c7c3-8482-49d4-bb89-c4ccf3f2db82', 'volume_name': 'volume-63b2c7c3-8482-49d4-bb89-c4ccf3f2db82', 'volume_size': 1}} Apr 17 16:03:38 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'retry': {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']}, 'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': '63b2c7c3-8482-49d4-bb89-c4ccf3f2db82', 'volume': {'id': '63b2c7c3-8482-49d4-bb89-c4ccf3f2db82', '_name_id': None, 'ec2_id': None, 'user_id': '9fdc7c80b4654692a52c8a4c4149324f', 'project_id': '8586cc10142d4743874f43cdb0f37aba', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-AttachVolumeNegativeTest-volume-1267367334', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T16:01:19.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-63b2c7c3-8482-49d4-bb89-c4ccf3f2db82', 'name_id': '63b2c7c3-8482-49d4-bb89-c4ccf3f2db82', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-AttachVolumeNegativeTest-volume-1267367334', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '8586cc10142d4743874f43cdb0f37aba', 'qos_specs': None, 'reservations': ['60706a3b-0211-488a-91cd-03061d7521a9', '3ccbaa06-3cae-4fa4-b8ac-ab913175fa13', '74bd9b1b-1b22-4085-aa81-b422f52940be', '3a23250a-64d8-4fa2-826b-865608d68d21'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': '9fdc7c80b4654692a52c8a4c4149324f', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-AttachVolumeNegativeTest-volume-1267367334', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '8586cc10142d4743874f43cdb0f37aba', 'qos_specs': None, 'reservations': ['60706a3b-0211-488a-91cd-03061d7521a9', '3ccbaa06-3cae-4fa4-b8ac-ab913175fa13', '74bd9b1b-1b22-4085-aa81-b422f52940be', '3a23250a-64d8-4fa2-826b-865608d68d21'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': '9fdc7c80b4654692a52c8a4c4149324f', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': '9fdc7c80b4654692a52c8a4c4149324f', 'metadata': {}, 'qos_specs': None}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(63b2c7c3-8482-49d4-bb89-c4ccf3f2db82),volume_id=63b2c7c3-8482-49d4-bb89-c4ccf3f2db82,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:01:19Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-AttachVolumeNegativeTest-volume-1267367334',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=63b2c7c3-8482-49d4-bb89-c4ccf3f2db82,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='8586cc10142d4743874f43cdb0f37aba',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:01:20Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:01:20Z,use_quota=True,user_id='9fdc7c80b4654692a52c8a4c4149324f',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 16:03:38 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:01:19Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-AttachVolumeNegativeTest-volume-1267367334',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=63b2c7c3-8482-49d4-bb89-c4ccf3f2db82,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='8586cc10142d4743874f43cdb0f37aba',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:01:20Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:01:20Z,use_quota=True,user_id='9fdc7c80b4654692a52c8a4c4149324f',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:01:19Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-AttachVolumeNegativeTest-volume-1267367334',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=63b2c7c3-8482-49d4-bb89-c4ccf3f2db82,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='8586cc10142d4743874f43cdb0f37aba',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:01:20Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:01:20Z,use_quota=True,user_id='9fdc7c80b4654692a52c8a4c4149324f',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 16:03:38 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 16:03:38 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 16:03:38 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:03:38 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 16:03:38 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:03:38 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 16:03:38 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:03:38 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 16:03:38 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:03:38 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 16:03:38 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:03:38 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 16:03:38 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:03:38 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 16:03:38 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:03:38 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 16:03:38 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:03:38 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:03:38 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 16:03:38 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 16:03:38 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (1c7ea1e4-da9a-4973-a73f-4bdfe10bff8a) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:03:38 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (1c7ea1e4-da9a-4973-a73f-4bdfe10bff8a) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:03:38 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (d1c27e31-d670-48de-ab59-e0f9def6013e) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:03:38 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (d1c27e31-d670-48de-ab59-e0f9def6013e) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:03:38 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (622f8b43-f785-4734-9717-aa003a91a2a2) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:03:38 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (622f8b43-f785-4734-9717-aa003a91a2a2) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:03:38 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (a81e2362-7b10-448f-b218-2b651857d634) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:03:38 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Updating volume 63b2c7c3-8482-49d4-bb89-c4ccf3f2db82 with {'scheduled_at': datetime.datetime(2023, 4, 17, 16, 3, 38, 789302), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 16:03:38 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Volume 63b2c7c3-8482-49d4-bb89-c4ccf3f2db82: re-scheduling SchedulerAPI.create_volume attempt 1 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 16:03:38 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Volume 63b2c7c3-8482-49d4-bb89-c4ccf3f2db82: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 16:03:38 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] deleating volume v_63b2c7c3-8482-49d4-bb89-c4ccf3f2db82 {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 16:03:38 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] get snapshots for volume v_63b2c7c3-8482-49d4-bb89-c4ccf3f2db82 {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 16:03:38 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_63b2c7c3-8482-49d4-bb89-c4ccf3f2db82/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:03:38 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_63b2c7c3-8482-49d4-bb89-c4ccf3f2db82/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:03:38 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Volume reschedule parameters: True retry: {'num_attempts': 2, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 16:03:38 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Flow 'volume_create_manager' (5e9ae34d-c22c-4075-81de-33ec1630d1ff) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 16:03:38 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (e83b71b6-5e69-4568-bb40-d5a1f1bd4d64) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:03:38 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (e83b71b6-5e69-4568-bb40-d5a1f1bd4d64) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:01:19Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-AttachVolumeNegativeTest-volume-1267367334',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=63b2c7c3-8482-49d4-bb89-c4ccf3f2db82,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='8586cc10142d4743874f43cdb0f37aba',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:03:39Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:03:39Z,use_quota=True,user_id='9fdc7c80b4654692a52c8a4c4149324f',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:03:38 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (6903556b-4102-4597-bd53-3965cfeb9219) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:03:38 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (6903556b-4102-4597-bd53-3965cfeb9219) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:03:39 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (592eb3de-507d-4938-87d0-adb3814a7e3b) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:03:39 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (592eb3de-507d-4938-87d0-adb3814a7e3b) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': '63b2c7c3-8482-49d4-bb89-c4ccf3f2db82', 'volume_name': 'volume-63b2c7c3-8482-49d4-bb89-c4ccf3f2db82', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:03:39 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (f547cce5-2b90-460c-ad6e-9add1931f684) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:03:39 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (f547cce5-2b90-460c-ad6e-9add1931f684) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:03:39 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (3d9e112f-3299-4b47-8eba-5907b33f5509) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:03:39 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Volume 63b2c7c3-8482-49d4-bb89-c4ccf3f2db82: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-63b2c7c3-8482-49d4-bb89-c4ccf3f2db82', 'volume_size': 1} Apr 17 16:03:39 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] creating volume v_63b2c7c3-8482-49d4-bb89-c4ccf3f2db82. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 16:03:39 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] create volume {'name': 'v_63b2c7c3-8482-49d4-bb89-c4ccf3f2db82', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 16:03:39 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:03:39 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:03:41 user cinder-volume[84696]: DEBUG cinder.utils [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.970(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:03:41 user cinder-volume[84696]: DEBUG cinder.utils [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_63b2c7c3-8482-49d4-bb89-c4ccf3f2db82/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:03:41 user cinder-volume[84696]: DEBUG cinder.utils [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.837(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:03:41 user cinder-volume[84696]: DEBUG cinder.utils [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_b4281d17-c1d8-48d3-8a0b-8fd7d2f193cc/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:03:41 user cinder-volume[84696]: DEBUG cinder.utils [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.642(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:03:41 user cinder-volume[84696]: DEBUG cinder.utils [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:03:41 user cinder-volume[84696]: DEBUG cinder.utils [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.772(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:03:41 user cinder-volume[84696]: DEBUG cinder.utils [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:03:44 user cinder-volume[84696]: DEBUG cinder.utils [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 38.985(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:03:44 user cinder-volume[84696]: DEBUG cinder.utils [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:03:44 user cinder-volume[84696]: DEBUG cinder.utils [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.189(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:03:44 user cinder-volume[84696]: DEBUG cinder.utils [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_fd313648-5fc1-4e6a-97a0-92ac803b36fd/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:03:44 user cinder-volume[84696]: DEBUG cinder.utils [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 11.835(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:03:44 user cinder-volume[84696]: DEBUG cinder.utils [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:03:44 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.215(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:03:44 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:03:44 user cinder-volume[84696]: DEBUG cinder.utils [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.848(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:03:44 user cinder-volume[84696]: DEBUG cinder.utils [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:03:44 user cinder-volume[84696]: DEBUG cinder.utils [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.051(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:03:44 user cinder-volume[84696]: DEBUG cinder.utils [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_63b2c7c3-8482-49d4-bb89-c4ccf3f2db82/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:03:47 user cinder-volume[84696]: DEBUG cinder.utils [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.390(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:03:47 user cinder-volume[84696]: DEBUG cinder.utils [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_bff6fdbb-a1fb-4b48-bca8-2f7efbc3f0ff/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:03:47 user cinder-volume[84696]: DEBUG cinder.utils [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 140.045(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:03:47 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_b29a7592-128a-41ee-a8e2-57780dee33e7/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:03:47 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (9fda5870-733f-436b-90f4-6a13f218e372) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 16:03:47 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (f38b98c4-bfbe-41d0-bfcc-8ab694d2bcea) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:03:47 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (f38b98c4-bfbe-41d0-bfcc-8ab694d2bcea) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:03:47 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Flow 'volume_create_manager' (78b522d0-6048-4086-8445-73b5cdb6254e) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 16:03:47 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-eb524e6a-d44c-4107-91cc-ffa4d4a5bf1b tempest-VolumeRevertTests-63962236 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 16:03:47 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 16:03:47 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 16:03:47 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 16:03:47 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 16:03:47 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 16:03:47 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 16:03:47 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 16:03:47 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 16:03:47 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 16:03:47 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 16:03:47 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 16:03:47 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 16:03:47 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 16:03:47 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 16:03:47 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 16:03:47 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 16:03:47 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 16:03:47 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 16:03:47 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 16:03:47 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 16:03:47 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 16:03:47 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 16:03:47 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 16:03:47 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:03:47 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 16:03:47 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:03:47 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 16:03:47 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:03:47 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 16:03:47 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:03:47 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 16:03:47 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:03:47 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 16:03:47 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:03:47 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 16:03:47 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:03:47 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 16:03:47 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:03:47 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:03:47 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 16:03:47 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 16:03:51 user cinder-volume[84696]: DEBUG cinder.utils [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.185(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:03:51 user cinder-volume[84696]: DEBUG cinder.utils [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_63b2c7c3-8482-49d4-bb89-c4ccf3f2db82/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:03:51 user cinder-volume[84696]: DEBUG cinder.utils [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 11.985(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:03:51 user cinder-volume[84696]: DEBUG cinder.utils [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:03:54 user cinder-volume[84696]: DEBUG cinder.utils [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.400(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:03:54 user cinder-volume[84696]: DEBUG cinder.utils [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_fd313648-5fc1-4e6a-97a0-92ac803b36fd/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:03:54 user cinder-volume[84696]: DEBUG cinder.utils [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.048(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:03:54 user cinder-volume[84696]: DEBUG cinder.utils [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:03:54 user cinder-volume[84696]: DEBUG cinder.utils [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 140.035(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:03:54 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_75163b78-31b2-4676-b93b-542a8ea784c1/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:03:54 user cinder-volume[84696]: WARNING cinder.volume.manager [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (c46062ad-beb8-4f66-9e1e-d87f80e69bbb) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 16:03:54 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (ac918f7e-a49c-4935-abb8-d89e944e8841) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:03:54 user cinder-volume[84696]: WARNING cinder.volume.manager [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (ac918f7e-a49c-4935-abb8-d89e944e8841) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:03:54 user cinder-volume[84696]: WARNING cinder.volume.manager [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Flow 'volume_create_manager' (b30c9810-66a7-4121-a690-a2f9ca833f5a) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 16:03:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [req-ccaee0c2-d84b-4650-899b-f7362672ce71 req-a5a6e5c2-62b5-4a76-9bf8-dbfd32544a07 tempest-ServerBootFromVolumeStableRescueTest-1271370980 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 16:03:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 16:03:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 16:03:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 16:03:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 16:03:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 16:03:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 16:03:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 16:03:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 16:03:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 16:03:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 16:03:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 16:03:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 16:03:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 16:03:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 16:03:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 16:03:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 16:03:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 16:03:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 16:03:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 16:03:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 16:03:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 16:03:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 16:03:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 16:03:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:03:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 16:03:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1237, in execute Apr 17 16:03:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_from_image(context, Apr 17 16:03:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/utils.py", line 670, in _wrapper Apr 17 16:03:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return r(f, *args, **kwargs) Apr 17 16:03:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/tenacity/__init__.py", line 423, in __call__ Apr 17 16:03:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server do = self.iter(retry_state=retry_state) Apr 17 16:03:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/tenacity/__init__.py", line 360, in iter Apr 17 16:03:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return fut.result() Apr 17 16:03:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/lib/python3.10/concurrent/futures/_base.py", line 451, in result Apr 17 16:03:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self.__get_result() Apr 17 16:03:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result Apr 17 16:03:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self._exception Apr 17 16:03:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/tenacity/__init__.py", line 426, in __call__ Apr 17 16:03:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = fn(*args, **kwargs) Apr 17 16:03:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1125, in _create_from_image Apr 17 16:03:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_from_image_cache_or_download( Apr 17 16:03:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 944, in _create_from_image_cache_or_download Apr 17 16:03:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update, cloned = self._create_from_image_cache( Apr 17 16:03:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 843, in _create_from_image_cache Apr 17 16:03:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_from_source_volume( Apr 17 16:03:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 659, in _create_from_source_volume Apr 17 16:03:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self.driver.create_cloned_volume(volume, srcvol_ref) Apr 17 16:03:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 525, in create_cloned_volume Apr 17 16:03:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self._clone_object(vname, cvname) Apr 17 16:03:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 461, in _clone_object Apr 17 16:03:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_snapshot(oname, coname) Apr 17 16:03:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 639, in create_snapshot Apr 17 16:03:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201, 204): Apr 17 16:03:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 16:03:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 16:03:54 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.449(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:03:54 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:03:56 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:INSERT=4,cinder:UPDATE=8,cinder:SELECT=18 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:03:56 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-008ec7fb-052b-43ae-952b-3c2d087ab270 req-843599cc-4c80-4fe3-81ef-b83c113ecd84 tempest-ServerActionsTestJSON-645547919 None] terminate connection for 1072d05f-2ebb-4398-902c-902df25464e9 {{(pid=84696) terminate_connection /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:1032}} Apr 17 16:03:56 user cinder-volume[84696]: INFO cinder.volume.manager [req-008ec7fb-052b-43ae-952b-3c2d087ab270 req-843599cc-4c80-4fe3-81ef-b83c113ecd84 tempest-ServerActionsTestJSON-645547919 None] Terminate volume connection completed successfully. Apr 17 16:03:56 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-008ec7fb-052b-43ae-952b-3c2d087ab270 req-843599cc-4c80-4fe3-81ef-b83c113ecd84 tempest-ServerActionsTestJSON-645547919 None] Deleting attachment a68f5fc4-dea3-48a4-bdb0-1d89db356c29. {{(pid=84696) attachment_delete /opt/stack/cinder/cinder/volume/manager.py:5006}} Apr 17 16:03:56 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-008ec7fb-052b-43ae-952b-3c2d087ab270 req-843599cc-4c80-4fe3-81ef-b83c113ecd84 tempest-ServerActionsTestJSON-645547919 None] remove_export for volume: 1072d05f-2ebb-4398-902c-902df25464e9. {{(pid=84696) remove_export /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:718}} Apr 17 16:03:56 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-008ec7fb-052b-43ae-952b-3c2d087ab270 req-843599cc-4c80-4fe3-81ef-b83c113ecd84 tempest-ServerActionsTestJSON-645547919 None] remove export {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:912}} Apr 17 16:03:56 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-008ec7fb-052b-43ae-952b-3c2d087ab270 req-843599cc-4c80-4fe3-81ef-b83c113ecd84 tempest-ServerActionsTestJSON-645547919 None] detach volume:Volume(_name_id=None,admin_metadata=,attach_status='attached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:27:39Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-ServerActionsTestJSON-volume-1520907984',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=1072d05f-2ebb-4398-902c-902df25464e9,launched_at=2023-04-17T15:27:40Z,metadata=,migration_status=None,multiattach=False,previous_status=None,project_id='cb5f6b5ed2fc43b69c47e1fc18646545',provider_auth='CHAP NKISaVyG nHFlRwNr8b5KS2',provider_geometry=None,provider_id=None,provider_location='172.16.0.220:3260,1 iqn.2010-10.org.openstack:1072d05f-2ebb-4398-902c-902df25464e9 0',replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:27:39Z,service_uuid='e628352e-c4cf-40eb-8389-cce40663026a',shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='in-use',terminated_at=None,updated_at=2023-04-17T15:32:47Z,use_quota=True,user_id='638a4f535e9740fc9d810639a5eb24d7',volume_attachment=,volume_type=,volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c) from target:iqn.2010-10.org.openstack:1072d05f-2ebb-4398-902c-902df25464e9. {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:913}} Apr 17 16:03:56 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [req-008ec7fb-052b-43ae-952b-3c2d087ab270 req-843599cc-4c80-4fe3-81ef-b83c113ecd84 tempest-ServerActionsTestJSON-645547919 None] detach volume v_1072d05f-2ebb-4398-902c-902df25464e9 from target iqn.2010-10.org.openstack:1072d05f-2ebb-4398-902c-902df25464e9 {{(pid=84696) detach_target_vol /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:606}} Apr 17 16:03:56 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-008ec7fb-052b-43ae-952b-3c2d087ab270 req-843599cc-4c80-4fe3-81ef-b83c113ecd84 tempest-ServerActionsTestJSON-645547919 None] Sending pool request DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:1072d05f-2ebb-4398-902c-902df25464e9/luns/v_1072d05f-2ebb-4398-902c-902df25464e9 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:03:56 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-008ec7fb-052b-43ae-952b-3c2d087ab270 req-843599cc-4c80-4fe3-81ef-b83c113ecd84 tempest-ServerActionsTestJSON-645547919 None] Sending DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:1072d05f-2ebb-4398-902c-902df25464e9/luns/v_1072d05f-2ebb-4398-902c-902df25464e9 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:03:57 user cinder-volume[84696]: DEBUG cinder.utils [req-008ec7fb-052b-43ae-952b-3c2d087ab270 req-843599cc-4c80-4fe3-81ef-b83c113ecd84 tempest-ServerActionsTestJSON-645547919 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 0.823(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:03:57 user cinder-volume[84696]: DEBUG cinder.utils [req-008ec7fb-052b-43ae-952b-3c2d087ab270 req-843599cc-4c80-4fe3-81ef-b83c113ecd84 tempest-ServerActionsTestJSON-645547919 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:1072d05f-2ebb-4398-902c-902df25464e9/luns/v_1072d05f-2ebb-4398-902c-902df25464e9 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:04:00 user cinder-volume[84696]: DEBUG cinder.utils [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.200(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:04:00 user cinder-volume[84696]: DEBUG cinder.utils [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:04:00 user cinder-volume[84696]: DEBUG cinder.utils [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.403(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:04:00 user cinder-volume[84696]: DEBUG cinder.utils [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_63b2c7c3-8482-49d4-bb89-c4ccf3f2db82/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:04:00 user cinder-volume[84696]: DEBUG cinder.utils [req-008ec7fb-052b-43ae-952b-3c2d087ab270 req-843599cc-4c80-4fe3-81ef-b83c113ecd84 tempest-ServerActionsTestJSON-645547919 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 3.898(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:04:00 user cinder-volume[84696]: DEBUG cinder.utils [req-008ec7fb-052b-43ae-952b-3c2d087ab270 req-843599cc-4c80-4fe3-81ef-b83c113ecd84 tempest-ServerActionsTestJSON-645547919 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:1072d05f-2ebb-4398-902c-902df25464e9/luns/v_1072d05f-2ebb-4398-902c-902df25464e9 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:04:06 user cinder-volume[84696]: DEBUG cinder.utils [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.822(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:04:06 user cinder-volume[84696]: DEBUG cinder.utils [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_bff6fdbb-a1fb-4b48-bca8-2f7efbc3f0ff/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:04:06 user cinder-volume[84696]: DEBUG cinder.utils [req-008ec7fb-052b-43ae-952b-3c2d087ab270 req-843599cc-4c80-4fe3-81ef-b83c113ecd84 tempest-ServerActionsTestJSON-645547919 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 10.040(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:04:06 user cinder-volume[84696]: DEBUG cinder.utils [req-008ec7fb-052b-43ae-952b-3c2d087ab270 req-843599cc-4c80-4fe3-81ef-b83c113ecd84 tempest-ServerActionsTestJSON-645547919 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:1072d05f-2ebb-4398-902c-902df25464e9/luns/v_1072d05f-2ebb-4398-902c-902df25464e9 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:04:12 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.856(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:04:12 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:04:12 user cinder-volume[84696]: DEBUG cinder.utils [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.833(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:04:12 user cinder-volume[84696]: DEBUG cinder.utils [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_fd313648-5fc1-4e6a-97a0-92ac803b36fd/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:04:12 user cinder-volume[84696]: DEBUG cinder.utils [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 71.012(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:04:12 user cinder-volume[84696]: DEBUG cinder.utils [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_40efe2ca-3a61-44c9-b9e0-bb29b333f0f9/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:04:12 user cinder-volume[84696]: DEBUG cinder.utils [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.482(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:04:12 user cinder-volume[84696]: DEBUG cinder.utils [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:04:12 user cinder-volume[84696]: DEBUG cinder.utils [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 72.282(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:04:12 user cinder-volume[84696]: DEBUG cinder.utils [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_40efe2ca-3a61-44c9-b9e0-bb29b333f0f9/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:04:12 user cinder-volume[84696]: DEBUG cinder.utils [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 73.638(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:04:12 user cinder-volume[84696]: DEBUG cinder.utils [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_40efe2ca-3a61-44c9-b9e0-bb29b333f0f9/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:04:14 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=11,cinder:UPDATE=4 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:04:15 user cinder-volume[84696]: DEBUG cinder.utils [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 73.432(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:04:15 user cinder-volume[84696]: DEBUG cinder.utils [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:04:15 user cinder-volume[84696]: DEBUG cinder.utils [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 73.629(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:04:15 user cinder-volume[84696]: DEBUG cinder.utils [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_b4281d17-c1d8-48d3-8a0b-8fd7d2f193cc/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:04:15 user cinder-volume[84696]: DEBUG cinder.utils [req-008ec7fb-052b-43ae-952b-3c2d087ab270 req-843599cc-4c80-4fe3-81ef-b83c113ecd84 tempest-ServerActionsTestJSON-645547919 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 19.258(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:04:15 user cinder-volume[84696]: DEBUG cinder.utils [req-008ec7fb-052b-43ae-952b-3c2d087ab270 req-843599cc-4c80-4fe3-81ef-b83c113ecd84 tempest-ServerActionsTestJSON-645547919 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:1072d05f-2ebb-4398-902c-902df25464e9/luns/v_1072d05f-2ebb-4398-902c-902df25464e9 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:04:18 user cinder-volume[84696]: DEBUG cinder.utils [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.632(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:04:18 user cinder-volume[84696]: DEBUG cinder.utils [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:04:18 user cinder-volume[84696]: DEBUG cinder.utils [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.835(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:04:18 user cinder-volume[84696]: DEBUG cinder.utils [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_63b2c7c3-8482-49d4-bb89-c4ccf3f2db82/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:04:18 user cinder-volume[84696]: DEBUG cinder.utils [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 72.775(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:04:18 user cinder-volume[84696]: DEBUG cinder.utils [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:04:24 user cinder-volume[84696]: DEBUG cinder.utils [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 138.607(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:04:24 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:04:24 user cinder-volume[84696]: INFO cinder.message.api [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Creating message record for request_id = req-8969563c-70db-45ae-8d53-74d80da7efc1 Apr 17 16:04:24 user cinder-volume[84696]: DEBUG cinder.utils [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 138.789(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:04:24 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_26014b4e-ec84-4011-a69f-ed42c3424a24/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:04:24 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (0017c1da-bbee-4554-b557-920615d6357f) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 16:04:24 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (bcf67fd7-208b-4833-90ae-38b887220015) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:04:24 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (bcf67fd7-208b-4833-90ae-38b887220015) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:04:24 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Flow 'volume_create_manager' (e5bae52d-cd04-489c-b1af-1788f6c077fe) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 16:04:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 16:04:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 16:04:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 16:04:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 16:04:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 16:04:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 16:04:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 16:04:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 16:04:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 16:04:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 16:04:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 16:04:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 16:04:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 16:04:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 16:04:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 16:04:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 16:04:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 16:04:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 16:04:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 16:04:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 16:04:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 16:04:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 16:04:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 16:04:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 16:04:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:04:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 16:04:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:04:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 16:04:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:04:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 16:04:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:04:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 16:04:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:04:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 16:04:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:04:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 16:04:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:04:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 16:04:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:04:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:04:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 16:04:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 16:04:24 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 16:04:24 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (eccb938b-be3d-4287-89bf-2eb5b91b4e13) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 16:04:24 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 16:04:24 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:59:46Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesListAdminTestJSON-Volume-237491364',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=26014b4e-ec84-4011-a69f-ed42c3424a24,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='73222838c1d4451281d075e1f9c092b3',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:02:06Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:02:06Z,use_quota=True,user_id='6a809fbfe4264147a2716c1cc5914a28',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 16:04:24 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:59:46Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesListAdminTestJSON-Volume-237491364',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=26014b4e-ec84-4011-a69f-ed42c3424a24,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='73222838c1d4451281d075e1f9c092b3',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:02:06Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:02:06Z,use_quota=True,user_id='6a809fbfe4264147a2716c1cc5914a28',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(26014b4e-ec84-4011-a69f-ed42c3424a24),volume_id=26014b4e-ec84-4011-a69f-ed42c3424a24,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': '26014b4e-ec84-4011-a69f-ed42c3424a24', 'volume_name': 'volume-26014b4e-ec84-4011-a69f-ed42c3424a24', 'volume_size': 1}} Apr 17 16:04:24 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': '26014b4e-ec84-4011-a69f-ed42c3424a24', 'volume': {'id': '26014b4e-ec84-4011-a69f-ed42c3424a24', '_name_id': None, 'ec2_id': None, 'user_id': '6a809fbfe4264147a2716c1cc5914a28', 'project_id': '73222838c1d4451281d075e1f9c092b3', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-VolumesListAdminTestJSON-Volume-237491364', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'admin_metadata': {}, 'glance_metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T15:59:45.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-26014b4e-ec84-4011-a69f-ed42c3424a24', 'name_id': '26014b4e-ec84-4011-a69f-ed42c3424a24', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumesListAdminTestJSON-Volume-237491364', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '73222838c1d4451281d075e1f9c092b3', 'qos_specs': None, 'reservations': ['aae7b3c8-9b21-42c2-b7d1-365a54453839', '40630276-bfc3-4923-b711-7d6a45c12c52', '251f733a-f47a-4155-8d77-c0c8abe4451b', '7007e6ad-6925-4dc6-bc99-8f9d477393a9'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': '6a809fbfe4264147a2716c1cc5914a28', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumesListAdminTestJSON-Volume-237491364', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '73222838c1d4451281d075e1f9c092b3', 'qos_specs': None, 'reservations': ['aae7b3c8-9b21-42c2-b7d1-365a54453839', '40630276-bfc3-4923-b711-7d6a45c12c52', '251f733a-f47a-4155-8d77-c0c8abe4451b', '7007e6ad-6925-4dc6-bc99-8f9d477393a9'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': '6a809fbfe4264147a2716c1cc5914a28', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': '6a809fbfe4264147a2716c1cc5914a28', 'metadata': {}, 'qos_specs': None, 'retry': {'num_attempts': 2, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']}}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(26014b4e-ec84-4011-a69f-ed42c3424a24),volume_id=26014b4e-ec84-4011-a69f-ed42c3424a24,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:59:46Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesListAdminTestJSON-Volume-237491364',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=26014b4e-ec84-4011-a69f-ed42c3424a24,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='73222838c1d4451281d075e1f9c092b3',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:02:06Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:02:06Z,use_quota=True,user_id='6a809fbfe4264147a2716c1cc5914a28',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 16:04:24 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:59:46Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesListAdminTestJSON-Volume-237491364',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=26014b4e-ec84-4011-a69f-ed42c3424a24,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='73222838c1d4451281d075e1f9c092b3',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:02:06Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:02:06Z,use_quota=True,user_id='6a809fbfe4264147a2716c1cc5914a28',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:59:46Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesListAdminTestJSON-Volume-237491364',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=26014b4e-ec84-4011-a69f-ed42c3424a24,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='73222838c1d4451281d075e1f9c092b3',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:02:06Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:02:06Z,use_quota=True,user_id='6a809fbfe4264147a2716c1cc5914a28',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 16:04:24 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 16:04:24 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 16:04:24 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:04:24 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 16:04:24 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:04:24 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 16:04:24 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:04:24 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 16:04:24 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:04:24 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 16:04:24 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:04:24 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 16:04:24 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:04:24 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 16:04:24 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:04:24 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 16:04:24 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:04:24 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:04:24 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 16:04:24 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 16:04:24 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (eccb938b-be3d-4287-89bf-2eb5b91b4e13) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:04:24 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (eccb938b-be3d-4287-89bf-2eb5b91b4e13) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:04:24 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (78a7ba9d-f362-476e-975d-9a59c358177a) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:04:24 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (78a7ba9d-f362-476e-975d-9a59c358177a) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:04:24 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (1ed2b502-ec74-45a7-b8a0-67223ec834d7) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:04:24 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (1ed2b502-ec74-45a7-b8a0-67223ec834d7) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:04:24 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (a35054f0-be95-43c7-93ec-8f4d0735bc3d) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:04:24 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Updating volume 26014b4e-ec84-4011-a69f-ed42c3424a24 with {'scheduled_at': datetime.datetime(2023, 4, 17, 16, 4, 24, 874669), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 16:04:24 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Volume 26014b4e-ec84-4011-a69f-ed42c3424a24: re-scheduling SchedulerAPI.create_volume attempt 2 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 16:04:24 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Volume 26014b4e-ec84-4011-a69f-ed42c3424a24: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 16:04:24 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] deleating volume v_26014b4e-ec84-4011-a69f-ed42c3424a24 {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 16:04:24 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] get snapshots for volume v_26014b4e-ec84-4011-a69f-ed42c3424a24 {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 16:04:24 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_26014b4e-ec84-4011-a69f-ed42c3424a24/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:04:24 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_26014b4e-ec84-4011-a69f-ed42c3424a24/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:04:25 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Volume reschedule parameters: True retry: {'num_attempts': 3, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 16:04:25 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Flow 'volume_create_manager' (0dcce114-7d1d-4690-ad6b-a588f3750e32) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 16:04:25 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (1d3529f0-9a7a-4d7c-9d56-b03414fdc4d1) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:04:25 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (1d3529f0-9a7a-4d7c-9d56-b03414fdc4d1) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:59:46Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesListAdminTestJSON-Volume-237491364',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=26014b4e-ec84-4011-a69f-ed42c3424a24,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='73222838c1d4451281d075e1f9c092b3',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:04:25Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:04:25Z,use_quota=True,user_id='6a809fbfe4264147a2716c1cc5914a28',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:04:25 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (b17bc9b9-8b5d-4670-9061-37ba32d86af2) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:04:25 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (b17bc9b9-8b5d-4670-9061-37ba32d86af2) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:04:25 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (18195d25-eb30-419f-bf9e-82839ffdfac9) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:04:25 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (18195d25-eb30-419f-bf9e-82839ffdfac9) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': '26014b4e-ec84-4011-a69f-ed42c3424a24', 'volume_name': 'volume-26014b4e-ec84-4011-a69f-ed42c3424a24', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:04:25 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (39ffc2ad-3cb4-47af-8cb6-005975eb2062) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:04:25 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (39ffc2ad-3cb4-47af-8cb6-005975eb2062) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:04:25 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (79508278-836c-4b32-9b74-a94b07600a38) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:04:25 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Volume 26014b4e-ec84-4011-a69f-ed42c3424a24: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-26014b4e-ec84-4011-a69f-ed42c3424a24', 'volume_size': 1} Apr 17 16:04:25 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] creating volume v_26014b4e-ec84-4011-a69f-ed42c3424a24. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 16:04:25 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] create volume {'name': 'v_26014b4e-ec84-4011-a69f-ed42c3424a24', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 16:04:25 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:04:25 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:04:27 user cinder-volume[84696]: DEBUG cinder.utils [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.768(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:04:27 user cinder-volume[84696]: DEBUG cinder.utils [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:04:27 user cinder-volume[84696]: DEBUG cinder.utils [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.966(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:04:27 user cinder-volume[84696]: DEBUG cinder.utils [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_26014b4e-ec84-4011-a69f-ed42c3424a24/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:04:30 user cinder-volume[84696]: DEBUG cinder.utils [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.036(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:04:30 user cinder-volume[84696]: DEBUG cinder.utils [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_26014b4e-ec84-4011-a69f-ed42c3424a24/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:04:30 user cinder-volume[84696]: DEBUG cinder.utils [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.841(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:04:30 user cinder-volume[84696]: DEBUG cinder.utils [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:04:34 user cinder-volume[84696]: DEBUG cinder.utils [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 138.761(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:04:34 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_87514055-5a0f-459f-b10b-438d7a13d39d/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:04:34 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (43048501-f203-43eb-8c24-422c165f661a) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 16:04:34 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (afae940a-47b8-4175-a547-e2ad29901639) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:04:34 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (afae940a-47b8-4175-a547-e2ad29901639) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:04:34 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Flow 'volume_create_manager' (69a56206-5eb3-46ac-aec1-c0693642611c) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 16:04:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-5a40883b-32f9-400a-a3f7-129f4d4677bb tempest-AttachVolumeNegativeTest-331389537 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 16:04:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 16:04:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 16:04:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 16:04:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 16:04:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 16:04:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 16:04:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 16:04:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 16:04:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 16:04:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 16:04:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 16:04:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 16:04:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 16:04:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 16:04:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 16:04:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 16:04:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 16:04:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 16:04:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 16:04:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 16:04:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 16:04:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 16:04:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 16:04:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:04:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 16:04:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:04:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 16:04:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:04:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 16:04:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:04:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 16:04:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:04:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 16:04:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:04:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 16:04:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:04:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 16:04:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:04:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:04:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 16:04:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 16:04:34 user cinder-volume[84696]: DEBUG cinder.utils [req-008ec7fb-052b-43ae-952b-3c2d087ab270 req-843599cc-4c80-4fe3-81ef-b83c113ecd84 tempest-ServerActionsTestJSON-645547919 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 37.705(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:04:34 user cinder-volume[84696]: DEBUG cinder.utils [req-008ec7fb-052b-43ae-952b-3c2d087ab270 req-843599cc-4c80-4fe3-81ef-b83c113ecd84 tempest-ServerActionsTestJSON-645547919 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:1072d05f-2ebb-4398-902c-902df25464e9/luns/v_1072d05f-2ebb-4398-902c-902df25464e9 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:04:34 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=10,cinder:UPDATE=5 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:04:37 user cinder-volume[84696]: DEBUG cinder.utils [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.183(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:04:37 user cinder-volume[84696]: DEBUG cinder.utils [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_26014b4e-ec84-4011-a69f-ed42c3424a24/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:04:37 user cinder-volume[84696]: DEBUG cinder.utils [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 11.989(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:04:37 user cinder-volume[84696]: DEBUG cinder.utils [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:04:37 user cinder-volume[84696]: DEBUG cinder.utils [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 138.756(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:04:37 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_14aa38ea-7cbb-48f7-a9aa-ffcc7b66faa0/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:04:37 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (5faf14b4-e033-41b8-84fb-8b4d162dd650) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 16:04:37 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (cb21c6b9-8d0f-4c51-a6ff-25ffcc1a0545) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:04:37 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (cb21c6b9-8d0f-4c51-a6ff-25ffcc1a0545) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:04:37 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Flow 'volume_create_manager' (2f00e9a7-d1a6-4c53-923c-c7a818782e35) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 16:04:37 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-e5f547d7-ac89-4987-be0b-aaa457cd4a79 tempest-ServerActionsTestJSON-645547919 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 16:04:37 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 16:04:37 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 16:04:37 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 16:04:37 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 16:04:37 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 16:04:37 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 16:04:37 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 16:04:37 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 16:04:37 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 16:04:37 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 16:04:37 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 16:04:37 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 16:04:37 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 16:04:37 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 16:04:37 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 16:04:37 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 16:04:37 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 16:04:37 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 16:04:37 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 16:04:37 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 16:04:37 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 16:04:37 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 16:04:37 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 16:04:37 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:04:37 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 16:04:37 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1237, in execute Apr 17 16:04:37 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_from_image(context, Apr 17 16:04:37 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/utils.py", line 670, in _wrapper Apr 17 16:04:37 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return r(f, *args, **kwargs) Apr 17 16:04:37 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/tenacity/__init__.py", line 423, in __call__ Apr 17 16:04:37 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server do = self.iter(retry_state=retry_state) Apr 17 16:04:37 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/tenacity/__init__.py", line 360, in iter Apr 17 16:04:37 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return fut.result() Apr 17 16:04:37 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/lib/python3.10/concurrent/futures/_base.py", line 451, in result Apr 17 16:04:37 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self.__get_result() Apr 17 16:04:37 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result Apr 17 16:04:37 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self._exception Apr 17 16:04:37 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/tenacity/__init__.py", line 426, in __call__ Apr 17 16:04:37 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = fn(*args, **kwargs) Apr 17 16:04:37 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1125, in _create_from_image Apr 17 16:04:37 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_from_image_cache_or_download( Apr 17 16:04:37 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 944, in _create_from_image_cache_or_download Apr 17 16:04:37 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update, cloned = self._create_from_image_cache( Apr 17 16:04:37 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 843, in _create_from_image_cache Apr 17 16:04:37 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_from_source_volume( Apr 17 16:04:37 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 659, in _create_from_source_volume Apr 17 16:04:37 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self.driver.create_cloned_volume(volume, srcvol_ref) Apr 17 16:04:37 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 525, in create_cloned_volume Apr 17 16:04:37 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self._clone_object(vname, cvname) Apr 17 16:04:37 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 461, in _clone_object Apr 17 16:04:37 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_snapshot(oname, coname) Apr 17 16:04:37 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 639, in create_snapshot Apr 17 16:04:37 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201, 204): Apr 17 16:04:37 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 16:04:37 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 16:04:40 user cinder-volume[84696]: DEBUG cinder.utils [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 73.614(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:04:40 user cinder-volume[84696]: DEBUG cinder.utils [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_bff6fdbb-a1fb-4b48-bca8-2f7efbc3f0ff/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:04:43 user cinder-volume[84696]: DEBUG cinder.utils [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 138.514(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:04:43 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:04:43 user cinder-volume[84696]: INFO cinder.message.api [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Creating message record for request_id = req-dc8764a6-903f-44ac-86a4-76a8d1914785 Apr 17 16:04:43 user cinder-volume[84696]: DEBUG cinder.utils [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 138.762(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:04:43 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_9a399a62-6a8f-4b6d-8ecf-8dcdf57e5fbb/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:04:43 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (9b7ada94-459d-4587-8d7b-e2197a372198) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 16:04:43 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (fde380d8-9526-4ceb-88c5-668baf7bc4d8) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:04:43 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (fde380d8-9526-4ceb-88c5-668baf7bc4d8) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:04:43 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Flow 'volume_create_manager' (77678606-9fe2-4e14-9e75-217e8d1018c1) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 16:04:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 16:04:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 16:04:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 16:04:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 16:04:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 16:04:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 16:04:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 16:04:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 16:04:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 16:04:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 16:04:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 16:04:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 16:04:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 16:04:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 16:04:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 16:04:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 16:04:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 16:04:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 16:04:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 16:04:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 16:04:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 16:04:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 16:04:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 16:04:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 16:04:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:04:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 16:04:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:04:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 16:04:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:04:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 16:04:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:04:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 16:04:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:04:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 16:04:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:04:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 16:04:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:04:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 16:04:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:04:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:04:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 16:04:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 16:04:43 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 16:04:43 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (dbfa0aea-a640-4eac-aa2b-e626059732a5) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 16:04:43 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 16:04:43 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:57:47Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumeRevertTests-Volume-1536424582',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=9a399a62-6a8f-4b6d-8ecf-8dcdf57e5fbb,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='9c755d3b5d8a4d9c9918b6c4744f510e',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:02:25Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:02:25Z,use_quota=True,user_id='e362cfe4a31c444b9ab44a5d60f37ad9',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 16:04:43 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:57:47Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumeRevertTests-Volume-1536424582',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=9a399a62-6a8f-4b6d-8ecf-8dcdf57e5fbb,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='9c755d3b5d8a4d9c9918b6c4744f510e',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:02:25Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:02:25Z,use_quota=True,user_id='e362cfe4a31c444b9ab44a5d60f37ad9',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(9a399a62-6a8f-4b6d-8ecf-8dcdf57e5fbb),volume_id=9a399a62-6a8f-4b6d-8ecf-8dcdf57e5fbb,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': '9a399a62-6a8f-4b6d-8ecf-8dcdf57e5fbb', 'volume_name': 'volume-9a399a62-6a8f-4b6d-8ecf-8dcdf57e5fbb', 'volume_size': 1}} Apr 17 16:04:43 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': '9a399a62-6a8f-4b6d-8ecf-8dcdf57e5fbb', 'volume': {'id': '9a399a62-6a8f-4b6d-8ecf-8dcdf57e5fbb', '_name_id': None, 'ec2_id': None, 'user_id': 'e362cfe4a31c444b9ab44a5d60f37ad9', 'project_id': '9c755d3b5d8a4d9c9918b6c4744f510e', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-VolumeRevertTests-Volume-1536424582', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'admin_metadata': {}, 'glance_metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T15:57:47.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-9a399a62-6a8f-4b6d-8ecf-8dcdf57e5fbb', 'name_id': '9a399a62-6a8f-4b6d-8ecf-8dcdf57e5fbb', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumeRevertTests-Volume-1536424582', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '9c755d3b5d8a4d9c9918b6c4744f510e', 'qos_specs': None, 'reservations': ['50c841a2-120d-4b16-8e0c-160f7b70632b', '75012498-38d8-444b-aeb0-ece3429535fa', '3472950b-da65-4163-bf5e-5efec626310b', 'b7635850-6316-4d2c-8d29-9d8061c89d14'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'e362cfe4a31c444b9ab44a5d60f37ad9', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumeRevertTests-Volume-1536424582', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '9c755d3b5d8a4d9c9918b6c4744f510e', 'qos_specs': None, 'reservations': ['50c841a2-120d-4b16-8e0c-160f7b70632b', '75012498-38d8-444b-aeb0-ece3429535fa', '3472950b-da65-4163-bf5e-5efec626310b', 'b7635850-6316-4d2c-8d29-9d8061c89d14'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'e362cfe4a31c444b9ab44a5d60f37ad9', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': 'e362cfe4a31c444b9ab44a5d60f37ad9', 'metadata': {}, 'qos_specs': None, 'retry': {'num_attempts': 3, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']}}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(9a399a62-6a8f-4b6d-8ecf-8dcdf57e5fbb),volume_id=9a399a62-6a8f-4b6d-8ecf-8dcdf57e5fbb,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:57:47Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumeRevertTests-Volume-1536424582',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=9a399a62-6a8f-4b6d-8ecf-8dcdf57e5fbb,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='9c755d3b5d8a4d9c9918b6c4744f510e',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:02:25Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:02:25Z,use_quota=True,user_id='e362cfe4a31c444b9ab44a5d60f37ad9',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 16:04:43 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:57:47Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumeRevertTests-Volume-1536424582',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=9a399a62-6a8f-4b6d-8ecf-8dcdf57e5fbb,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='9c755d3b5d8a4d9c9918b6c4744f510e',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:02:25Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:02:25Z,use_quota=True,user_id='e362cfe4a31c444b9ab44a5d60f37ad9',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:57:47Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumeRevertTests-Volume-1536424582',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=9a399a62-6a8f-4b6d-8ecf-8dcdf57e5fbb,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='9c755d3b5d8a4d9c9918b6c4744f510e',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:02:25Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:02:25Z,use_quota=True,user_id='e362cfe4a31c444b9ab44a5d60f37ad9',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 16:04:43 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 16:04:43 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 16:04:43 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:04:43 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 16:04:43 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:04:43 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 16:04:43 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:04:43 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 16:04:43 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:04:43 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 16:04:43 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:04:43 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 16:04:43 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:04:43 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 16:04:43 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:04:43 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 16:04:43 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:04:43 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:04:43 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 16:04:43 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 16:04:43 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (dbfa0aea-a640-4eac-aa2b-e626059732a5) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:04:43 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (dbfa0aea-a640-4eac-aa2b-e626059732a5) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:04:43 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (bb8359bf-01ae-4469-822c-e9efadc5ff9e) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:04:43 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (bb8359bf-01ae-4469-822c-e9efadc5ff9e) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:04:43 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (33dc10a6-0a6e-4f58-b409-53508534083a) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:04:43 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (33dc10a6-0a6e-4f58-b409-53508534083a) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:04:43 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (06e44fc5-64be-4eb1-a659-1214b2f82192) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:04:43 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Updating volume 9a399a62-6a8f-4b6d-8ecf-8dcdf57e5fbb with {'scheduled_at': datetime.datetime(2023, 4, 17, 16, 4, 43, 313454), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 16:04:43 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Volume 9a399a62-6a8f-4b6d-8ecf-8dcdf57e5fbb: re-scheduling SchedulerAPI.create_volume attempt 3 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 16:04:43 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Volume 9a399a62-6a8f-4b6d-8ecf-8dcdf57e5fbb: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 16:04:43 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] deleating volume v_9a399a62-6a8f-4b6d-8ecf-8dcdf57e5fbb {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 16:04:43 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] get snapshots for volume v_9a399a62-6a8f-4b6d-8ecf-8dcdf57e5fbb {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 16:04:43 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_9a399a62-6a8f-4b6d-8ecf-8dcdf57e5fbb/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:04:43 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_9a399a62-6a8f-4b6d-8ecf-8dcdf57e5fbb/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:04:44 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=11,cinder:INSERT=2,cinder:UPDATE=3 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:04:46 user cinder-volume[84696]: DEBUG cinder.utils [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 3.057(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:04:46 user cinder-volume[84696]: DEBUG cinder.utils [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_9a399a62-6a8f-4b6d-8ecf-8dcdf57e5fbb/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:04:46 user cinder-volume[84696]: DEBUG cinder.utils [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.493(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:04:46 user cinder-volume[84696]: DEBUG cinder.utils [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_26014b4e-ec84-4011-a69f-ed42c3424a24/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:04:46 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 73.746(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:04:46 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:04:46 user cinder-volume[84696]: DEBUG cinder.utils [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 73.368(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:04:46 user cinder-volume[84696]: DEBUG cinder.utils [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:04:46 user cinder-volume[84696]: DEBUG cinder.utils [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.300(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:04:46 user cinder-volume[84696]: DEBUG cinder.utils [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:04:46 user cinder-volume[84696]: DEBUG cinder.utils [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 73.725(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:04:46 user cinder-volume[84696]: DEBUG cinder.utils [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_fd313648-5fc1-4e6a-97a0-92ac803b36fd/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:04:49 user cinder-volume[84696]: DEBUG cinder.utils [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.129(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:04:49 user cinder-volume[84696]: DEBUG cinder.utils [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_9a399a62-6a8f-4b6d-8ecf-8dcdf57e5fbb/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:04:52 user cinder-volume[84696]: DEBUG cinder.utils [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 73.524(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:04:52 user cinder-volume[84696]: DEBUG cinder.utils [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:04:52 user cinder-volume[84696]: DEBUG cinder.utils [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 73.727(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:04:52 user cinder-volume[84696]: DEBUG cinder.utils [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_63b2c7c3-8482-49d4-bb89-c4ccf3f2db82/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:04:55 user cinder-volume[84696]: DEBUG cinder.utils [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.273(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:04:55 user cinder-volume[84696]: DEBUG cinder.utils [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_9a399a62-6a8f-4b6d-8ecf-8dcdf57e5fbb/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:05:05 user cinder-volume[84696]: DEBUG cinder.utils [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 40.148(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:05:05 user cinder-volume[84696]: DEBUG cinder.utils [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_26014b4e-ec84-4011-a69f-ed42c3424a24/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:05:05 user cinder-volume[84696]: DEBUG cinder.utils [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.715(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:05:05 user cinder-volume[84696]: DEBUG cinder.utils [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_9a399a62-6a8f-4b6d-8ecf-8dcdf57e5fbb/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:05:05 user cinder-volume[84696]: DEBUG cinder.utils [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.955(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:05:05 user cinder-volume[84696]: DEBUG cinder.utils [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:05:08 user cinder-volume[84696]: DEBUG cinder.utils [req-008ec7fb-052b-43ae-952b-3c2d087ab270 req-843599cc-4c80-4fe3-81ef-b83c113ecd84 tempest-ServerActionsTestJSON-645547919 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 71.799(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:05:08 user cinder-volume[84696]: DEBUG cinder.utils [req-008ec7fb-052b-43ae-952b-3c2d087ab270 req-843599cc-4c80-4fe3-81ef-b83c113ecd84 tempest-ServerActionsTestJSON-645547919 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:1072d05f-2ebb-4398-902c-902df25464e9/luns/v_1072d05f-2ebb-4398-902c-902df25464e9 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:05:14 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=9,cinder:INSERT=2,cinder:UPDATE=3 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:05:18 user cinder-volume[84696]: DEBUG cinder.utils [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 136.576(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:05:18 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_40efe2ca-3a61-44c9-b9e0-bb29b333f0f9/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:05:18 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (4a5898c7-4880-42c6-90a8-cc3f84431bab) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 16:05:18 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (2cf5e095-a726-4784-b144-23c00b772354) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:05:18 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (2cf5e095-a726-4784-b144-23c00b772354) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:05:18 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Flow 'volume_create_manager' (99451714-a95d-4706-bc6e-e18b8a77895c) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Exception during message handling: cinder.exception.GlanceMetadataExists: Glance metadata cannot be updated, key signature_verified exists for volume id 40efe2ca-3a61-44c9-b9e0-bb29b333f0f9 Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1237, in execute Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_from_image(context, Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/utils.py", line 670, in _wrapper Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return r(f, *args, **kwargs) Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/tenacity/__init__.py", line 423, in __call__ Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server do = self.iter(retry_state=retry_state) Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/tenacity/__init__.py", line 360, in iter Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return fut.result() Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/lib/python3.10/concurrent/futures/_base.py", line 451, in result Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self.__get_result() Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self._exception Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/tenacity/__init__.py", line 426, in __call__ Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = fn(*args, **kwargs) Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1115, in _create_from_image Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update, cloned = self._prepare_image_cache_entry( Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/coordination.py", line 200, in _synchronized Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return f(*a, **k) Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 894, in _prepare_image_cache_entry Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_from_image_cache_or_download( Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 989, in _create_from_image_cache_or_download Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.db.volume_glance_metadata_bulk_create( Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/db/api.py", line 1138, in volume_glance_metadata_bulk_create Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return IMPL.volume_glance_metadata_bulk_create(context, volume_id, Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/db/sqlalchemy/api.py", line 181, in wrapper Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return f(*args, **kwargs) Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/db/sqlalchemy/api.py", line 211, in wrapper Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return f(context, volume_id, *args, **kwargs) Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1022, in wrapper Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return fn(*args, **kwargs) Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/db/sqlalchemy/api.py", line 6178, in volume_glance_metadata_bulk_create Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise exception.GlanceMetadataExists(key=key, volume_id=volume_id) Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server cinder.exception.GlanceMetadataExists: Glance metadata cannot be updated, key signature_verified exists for volume id 40efe2ca-3a61-44c9-b9e0-bb29b333f0f9 Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 16:05:18 user cinder-volume[84696]: DEBUG cinder.utils [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 137.863(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:05:18 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_40efe2ca-3a61-44c9-b9e0-bb29b333f0f9/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:05:18 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (8fa4ebdb-b6c2-4704-966a-9c68b8939008) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 16:05:18 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (4ce647bd-9723-4aa3-97da-c8957240bed8) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:05:18 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (4ce647bd-9723-4aa3-97da-c8957240bed8) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:05:18 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Flow 'volume_create_manager' (bbb7cec9-458b-4fb5-aebc-5c9074dc3a69) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Exception during message handling: cinder.exception.GlanceMetadataExists: Glance metadata cannot be updated, key signature_verified exists for volume id 40efe2ca-3a61-44c9-b9e0-bb29b333f0f9 Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1237, in execute Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_from_image(context, Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/utils.py", line 670, in _wrapper Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return r(f, *args, **kwargs) Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/tenacity/__init__.py", line 423, in __call__ Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server do = self.iter(retry_state=retry_state) Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/tenacity/__init__.py", line 360, in iter Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return fut.result() Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/lib/python3.10/concurrent/futures/_base.py", line 451, in result Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self.__get_result() Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self._exception Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/tenacity/__init__.py", line 426, in __call__ Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = fn(*args, **kwargs) Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1115, in _create_from_image Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update, cloned = self._prepare_image_cache_entry( Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/coordination.py", line 200, in _synchronized Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return f(*a, **k) Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 894, in _prepare_image_cache_entry Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_from_image_cache_or_download( Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 989, in _create_from_image_cache_or_download Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.db.volume_glance_metadata_bulk_create( Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/db/api.py", line 1138, in volume_glance_metadata_bulk_create Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return IMPL.volume_glance_metadata_bulk_create(context, volume_id, Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/db/sqlalchemy/api.py", line 181, in wrapper Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return f(*args, **kwargs) Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/db/sqlalchemy/api.py", line 211, in wrapper Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return f(context, volume_id, *args, **kwargs) Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1022, in wrapper Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return fn(*args, **kwargs) Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/db/sqlalchemy/api.py", line 6178, in volume_glance_metadata_bulk_create Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise exception.GlanceMetadataExists(key=key, volume_id=volume_id) Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server cinder.exception.GlanceMetadataExists: Glance metadata cannot be updated, key signature_verified exists for volume id 40efe2ca-3a61-44c9-b9e0-bb29b333f0f9 Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 16:05:18 user cinder-volume[84696]: DEBUG cinder.utils [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 139.235(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:05:18 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_40efe2ca-3a61-44c9-b9e0-bb29b333f0f9/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:05:18 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (e16a4ead-003c-4857-aa60-d98b521c0783) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 16:05:18 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (d8f39276-9dea-4eaa-b4ed-8e8595c1fc9d) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:05:18 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (d8f39276-9dea-4eaa-b4ed-8e8595c1fc9d) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:05:18 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Flow 'volume_create_manager' (68f959d6-a03d-42c1-82a7-a5abd9ee1327) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-9f7266e8-b4c8-475d-a860-ee39dd076d33 tempest-VolumeAndVolumeTypeFromImageTest-1053354066 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1237, in execute Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_from_image(context, Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/utils.py", line 670, in _wrapper Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return r(f, *args, **kwargs) Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/tenacity/__init__.py", line 423, in __call__ Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server do = self.iter(retry_state=retry_state) Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/tenacity/__init__.py", line 360, in iter Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return fut.result() Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/lib/python3.10/concurrent/futures/_base.py", line 451, in result Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self.__get_result() Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self._exception Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/tenacity/__init__.py", line 426, in __call__ Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = fn(*args, **kwargs) Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1115, in _create_from_image Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update, cloned = self._prepare_image_cache_entry( Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/coordination.py", line 200, in _synchronized Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return f(*a, **k) Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 894, in _prepare_image_cache_entry Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_from_image_cache_or_download( Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1004, in _create_from_image_cache_or_download Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_from_image_download( Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 791, in _create_from_image_download Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self.driver.create_volume(volume) or {} Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 16:05:18 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 16:05:21 user cinder-volume[84696]: DEBUG cinder.utils [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 139.195(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:05:21 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_b4281d17-c1d8-48d3-8a0b-8fd7d2f193cc/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:05:21 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (d1d2659c-e868-4ea2-a545-fe10a196244c) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 16:05:21 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (e0124657-9f65-43d2-bc18-f1f7f16a47ea) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:05:21 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (e0124657-9f65-43d2-bc18-f1f7f16a47ea) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:05:21 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Flow 'volume_create_manager' (434001cf-d61c-49f1-b8da-ca2f66c1fb8f) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 16:05:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 16:05:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 16:05:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 16:05:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 16:05:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 16:05:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 16:05:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 16:05:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 16:05:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 16:05:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 16:05:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 16:05:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 16:05:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 16:05:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 16:05:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 16:05:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 16:05:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 16:05:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 16:05:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 16:05:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 16:05:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 16:05:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 16:05:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 16:05:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 16:05:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:05:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 16:05:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:05:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 16:05:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:05:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 16:05:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:05:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 16:05:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:05:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 16:05:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:05:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 16:05:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:05:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 16:05:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:05:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:05:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 16:05:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 16:05:21 user cinder-volume[84696]: DEBUG cinder.utils [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 37.828(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:05:21 user cinder-volume[84696]: DEBUG cinder.utils [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_9a399a62-6a8f-4b6d-8ecf-8dcdf57e5fbb/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:05:21 user cinder-volume[84696]: DEBUG cinder.utils [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 139.020(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:05:21 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:05:21 user cinder-volume[84696]: INFO cinder.message.api [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Creating message record for request_id = req-39f8a1bb-4b23-4f63-8942-bcd810255374 Apr 17 16:05:21 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 16:05:21 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=6,cinder:UPDATE=2 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:05:21 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (528895d8-d3de-4ab5-be8d-6def8d3366a6) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 16:05:21 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 16:05:21 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:00:44Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-AttachVolumeShelveTestJSON-volume-7555594',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=b4281d17-c1d8-48d3-8a0b-8fd7d2f193cc,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='214a0f233f114d7aa22e64048a837c0b',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:03:02Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:03:02Z,use_quota=True,user_id='83b9d77de46143d8b9277a55f5ad4d72',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 16:05:21 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:00:44Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-AttachVolumeShelveTestJSON-volume-7555594',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=b4281d17-c1d8-48d3-8a0b-8fd7d2f193cc,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='214a0f233f114d7aa22e64048a837c0b',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:03:02Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:03:02Z,use_quota=True,user_id='83b9d77de46143d8b9277a55f5ad4d72',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(b4281d17-c1d8-48d3-8a0b-8fd7d2f193cc),volume_id=b4281d17-c1d8-48d3-8a0b-8fd7d2f193cc,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': 'b4281d17-c1d8-48d3-8a0b-8fd7d2f193cc', 'volume_name': 'volume-b4281d17-c1d8-48d3-8a0b-8fd7d2f193cc', 'volume_size': 1}} Apr 17 16:05:21 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': 'b4281d17-c1d8-48d3-8a0b-8fd7d2f193cc', 'volume': {'id': 'b4281d17-c1d8-48d3-8a0b-8fd7d2f193cc', '_name_id': None, 'ec2_id': None, 'user_id': '83b9d77de46143d8b9277a55f5ad4d72', 'project_id': '214a0f233f114d7aa22e64048a837c0b', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-AttachVolumeShelveTestJSON-volume-7555594', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'admin_metadata': {}, 'glance_metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T16:00:44.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-b4281d17-c1d8-48d3-8a0b-8fd7d2f193cc', 'name_id': 'b4281d17-c1d8-48d3-8a0b-8fd7d2f193cc', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-AttachVolumeShelveTestJSON-volume-7555594', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '214a0f233f114d7aa22e64048a837c0b', 'qos_specs': None, 'reservations': ['d9fcea02-28d2-4dba-a68b-5de4e249400a', 'dc207600-1b77-4709-9acd-aacbcf21051f', '891fc9c0-b5b9-44e3-a1c8-d30fcbfd5cc4', '8f4a2d2a-cb05-4ac0-b722-22369386396e'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': '83b9d77de46143d8b9277a55f5ad4d72', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-AttachVolumeShelveTestJSON-volume-7555594', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '214a0f233f114d7aa22e64048a837c0b', 'qos_specs': None, 'reservations': ['d9fcea02-28d2-4dba-a68b-5de4e249400a', 'dc207600-1b77-4709-9acd-aacbcf21051f', '891fc9c0-b5b9-44e3-a1c8-d30fcbfd5cc4', '8f4a2d2a-cb05-4ac0-b722-22369386396e'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': '83b9d77de46143d8b9277a55f5ad4d72', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': '83b9d77de46143d8b9277a55f5ad4d72', 'metadata': {}, 'qos_specs': None, 'retry': {'num_attempts': 2, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']}}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(b4281d17-c1d8-48d3-8a0b-8fd7d2f193cc),volume_id=b4281d17-c1d8-48d3-8a0b-8fd7d2f193cc,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:00:44Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-AttachVolumeShelveTestJSON-volume-7555594',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=b4281d17-c1d8-48d3-8a0b-8fd7d2f193cc,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='214a0f233f114d7aa22e64048a837c0b',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:03:02Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:03:02Z,use_quota=True,user_id='83b9d77de46143d8b9277a55f5ad4d72',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 16:05:21 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:00:44Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-AttachVolumeShelveTestJSON-volume-7555594',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=b4281d17-c1d8-48d3-8a0b-8fd7d2f193cc,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='214a0f233f114d7aa22e64048a837c0b',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:03:02Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:03:02Z,use_quota=True,user_id='83b9d77de46143d8b9277a55f5ad4d72',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:00:44Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-AttachVolumeShelveTestJSON-volume-7555594',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=b4281d17-c1d8-48d3-8a0b-8fd7d2f193cc,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='214a0f233f114d7aa22e64048a837c0b',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:03:02Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:03:02Z,use_quota=True,user_id='83b9d77de46143d8b9277a55f5ad4d72',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 16:05:21 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 16:05:21 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 16:05:21 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:05:21 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 16:05:21 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:05:21 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 16:05:21 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:05:21 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 16:05:21 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:05:21 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 16:05:21 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:05:21 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 16:05:21 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:05:21 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 16:05:21 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:05:21 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 16:05:21 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:05:21 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:05:21 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 16:05:21 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 16:05:21 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (528895d8-d3de-4ab5-be8d-6def8d3366a6) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:05:21 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (528895d8-d3de-4ab5-be8d-6def8d3366a6) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:05:21 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (b662187d-b791-4e86-92b5-dde5cb5f9604) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:05:21 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (b662187d-b791-4e86-92b5-dde5cb5f9604) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:05:21 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (a928dde2-794b-4453-83d3-3360d279ddd4) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:05:21 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (a928dde2-794b-4453-83d3-3360d279ddd4) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:05:21 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (1bd6d8b5-327c-4110-970e-7586cb828ec7) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:05:21 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Updating volume b4281d17-c1d8-48d3-8a0b-8fd7d2f193cc with {'scheduled_at': datetime.datetime(2023, 4, 17, 16, 5, 21, 236670), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 16:05:21 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Volume b4281d17-c1d8-48d3-8a0b-8fd7d2f193cc: re-scheduling SchedulerAPI.create_volume attempt 2 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 16:05:21 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Volume b4281d17-c1d8-48d3-8a0b-8fd7d2f193cc: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 16:05:21 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] deleating volume v_b4281d17-c1d8-48d3-8a0b-8fd7d2f193cc {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 16:05:21 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] get snapshots for volume v_b4281d17-c1d8-48d3-8a0b-8fd7d2f193cc {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 16:05:21 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_b4281d17-c1d8-48d3-8a0b-8fd7d2f193cc/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:05:21 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_b4281d17-c1d8-48d3-8a0b-8fd7d2f193cc/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:05:21 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Volume reschedule parameters: True retry: {'num_attempts': 3, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 16:05:21 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Flow 'volume_create_manager' (1cf046c1-1ada-4c3e-ac10-d4d428c2495a) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 16:05:21 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (0d3e40db-8638-46d2-91f8-796253e33629) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:05:21 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (0d3e40db-8638-46d2-91f8-796253e33629) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:00:44Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-AttachVolumeShelveTestJSON-volume-7555594',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=b4281d17-c1d8-48d3-8a0b-8fd7d2f193cc,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='214a0f233f114d7aa22e64048a837c0b',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:05:21Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:05:21Z,use_quota=True,user_id='83b9d77de46143d8b9277a55f5ad4d72',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:05:21 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (97d55fc0-4187-4795-998e-de3eeedad509) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:05:21 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (97d55fc0-4187-4795-998e-de3eeedad509) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:05:21 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (038bab7c-7596-429a-8925-90fc4b914f90) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:05:21 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (038bab7c-7596-429a-8925-90fc4b914f90) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': 'b4281d17-c1d8-48d3-8a0b-8fd7d2f193cc', 'volume_name': 'volume-b4281d17-c1d8-48d3-8a0b-8fd7d2f193cc', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:05:21 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (65a99498-25ff-4bc6-85c4-eeab1a143bd1) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:05:21 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (65a99498-25ff-4bc6-85c4-eeab1a143bd1) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:05:21 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (83fc3ab2-ce86-42ed-9afd-a44f55b27a7d) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:05:21 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Volume b4281d17-c1d8-48d3-8a0b-8fd7d2f193cc: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-b4281d17-c1d8-48d3-8a0b-8fd7d2f193cc', 'volume_size': 1} Apr 17 16:05:21 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] creating volume v_b4281d17-c1d8-48d3-8a0b-8fd7d2f193cc. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 16:05:21 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] create volume {'name': 'v_b4281d17-c1d8-48d3-8a0b-8fd7d2f193cc', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 16:05:21 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:05:21 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:05:24 user cinder-volume[84696]: DEBUG cinder.utils [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.955(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:05:24 user cinder-volume[84696]: DEBUG cinder.utils [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_b4281d17-c1d8-48d3-8a0b-8fd7d2f193cc/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:05:24 user cinder-volume[84696]: DEBUG cinder.utils [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.789(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:05:24 user cinder-volume[84696]: DEBUG cinder.utils [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:05:24 user cinder-volume[84696]: DEBUG cinder.utils [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 138.345(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:05:24 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:05:24 user cinder-volume[84696]: INFO cinder.message.api [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Creating message record for request_id = req-2f261810-f79c-48b0-b0ef-2727c4b52668 Apr 17 16:05:24 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 16:05:24 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (f8adf837-7376-42e2-8183-23cda829175a) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 16:05:24 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 16:05:24 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:03:06Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesListTestJSON-Volume-1309258736',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=4d960aac-48dd-470e-831f-eaafe64c9fdf,launched_at=None,metadata={Type='work'},migration_status=None,multiattach=False,previous_status=None,project_id='d1740791cf5e4c7087f45af9015c0a8f',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:03:06Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:03:06Z,use_quota=True,user_id='9091747614a945f6a88207d3662fea8e',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 16:05:24 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:03:06Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesListTestJSON-Volume-1309258736',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=4d960aac-48dd-470e-831f-eaafe64c9fdf,launched_at=None,metadata={Type='work'},migration_status=None,multiattach=False,previous_status=None,project_id='d1740791cf5e4c7087f45af9015c0a8f',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:03:06Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:03:06Z,use_quota=True,user_id='9091747614a945f6a88207d3662fea8e',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(4d960aac-48dd-470e-831f-eaafe64c9fdf),volume_id=4d960aac-48dd-470e-831f-eaafe64c9fdf,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': '4d960aac-48dd-470e-831f-eaafe64c9fdf', 'volume_name': 'volume-4d960aac-48dd-470e-831f-eaafe64c9fdf', 'volume_size': 1}} Apr 17 16:05:24 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'retry': {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']}, 'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': '4d960aac-48dd-470e-831f-eaafe64c9fdf', 'volume': {'id': '4d960aac-48dd-470e-831f-eaafe64c9fdf', '_name_id': None, 'ec2_id': None, 'user_id': '9091747614a945f6a88207d3662fea8e', 'project_id': 'd1740791cf5e4c7087f45af9015c0a8f', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-VolumesListTestJSON-Volume-1309258736', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {'Type': 'work'}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T16:03:05.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-4d960aac-48dd-470e-831f-eaafe64c9fdf', 'name_id': '4d960aac-48dd-470e-831f-eaafe64c9fdf', 'volume_metadata': [{'key': 'Type', 'value': 'work'}], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumesListTestJSON-Volume-1309258736', 'encryption_key_id': None, 'metadata': {'Type': 'work'}, 'multiattach': False, 'project_id': 'd1740791cf5e4c7087f45af9015c0a8f', 'qos_specs': None, 'reservations': ['61b9d19b-7196-4e0b-aa10-0b704c76275c', 'a121ac12-c1d8-4525-a222-e055f88dc125', '6e5c242e-6241-4c0f-9e09-305237e9921c', '20860b71-70c8-4b28-82e8-4b507fba5ed9'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': '9091747614a945f6a88207d3662fea8e', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumesListTestJSON-Volume-1309258736', 'encryption_key_id': None, 'metadata': {'Type': 'work'}, 'multiattach': False, 'project_id': 'd1740791cf5e4c7087f45af9015c0a8f', 'qos_specs': None, 'reservations': ['61b9d19b-7196-4e0b-aa10-0b704c76275c', 'a121ac12-c1d8-4525-a222-e055f88dc125', '6e5c242e-6241-4c0f-9e09-305237e9921c', '20860b71-70c8-4b28-82e8-4b507fba5ed9'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': '9091747614a945f6a88207d3662fea8e', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': '9091747614a945f6a88207d3662fea8e', 'metadata': {'Type': 'work'}, 'qos_specs': None}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(4d960aac-48dd-470e-831f-eaafe64c9fdf),volume_id=4d960aac-48dd-470e-831f-eaafe64c9fdf,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:03:06Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesListTestJSON-Volume-1309258736',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=4d960aac-48dd-470e-831f-eaafe64c9fdf,launched_at=None,metadata={Type='work'},migration_status=None,multiattach=False,previous_status=None,project_id='d1740791cf5e4c7087f45af9015c0a8f',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:03:06Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:03:06Z,use_quota=True,user_id='9091747614a945f6a88207d3662fea8e',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 16:05:24 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:03:06Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesListTestJSON-Volume-1309258736',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=4d960aac-48dd-470e-831f-eaafe64c9fdf,launched_at=None,metadata={Type='work'},migration_status=None,multiattach=False,previous_status=None,project_id='d1740791cf5e4c7087f45af9015c0a8f',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:03:06Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:03:06Z,use_quota=True,user_id='9091747614a945f6a88207d3662fea8e',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:03:06Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesListTestJSON-Volume-1309258736',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=4d960aac-48dd-470e-831f-eaafe64c9fdf,launched_at=None,metadata={Type='work'},migration_status=None,multiattach=False,previous_status=None,project_id='d1740791cf5e4c7087f45af9015c0a8f',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:03:06Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:03:06Z,use_quota=True,user_id='9091747614a945f6a88207d3662fea8e',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 16:05:24 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 16:05:24 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 16:05:24 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:05:24 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 16:05:24 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:05:24 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 16:05:24 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:05:24 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 16:05:24 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:05:24 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 16:05:24 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:05:24 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 16:05:24 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:05:24 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 16:05:24 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:05:24 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 16:05:24 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:05:24 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:05:24 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 16:05:24 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 16:05:24 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (f8adf837-7376-42e2-8183-23cda829175a) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:05:24 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (f8adf837-7376-42e2-8183-23cda829175a) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:05:24 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (e45ec5db-21cf-4496-a5d7-1a524ae82306) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:05:24 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (e45ec5db-21cf-4496-a5d7-1a524ae82306) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:05:24 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (171d181d-4cdf-41de-b293-bb2ec52ad422) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:05:24 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (171d181d-4cdf-41de-b293-bb2ec52ad422) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:05:24 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (d39f28e2-aa96-4bf8-bb1d-39acdf57545c) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:05:24 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Updating volume 4d960aac-48dd-470e-831f-eaafe64c9fdf with {'scheduled_at': datetime.datetime(2023, 4, 17, 16, 5, 24, 295329), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 16:05:24 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Volume 4d960aac-48dd-470e-831f-eaafe64c9fdf: re-scheduling SchedulerAPI.create_volume attempt 1 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 16:05:24 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Volume 4d960aac-48dd-470e-831f-eaafe64c9fdf: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 16:05:24 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] deleating volume v_4d960aac-48dd-470e-831f-eaafe64c9fdf {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 16:05:24 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] get snapshots for volume v_4d960aac-48dd-470e-831f-eaafe64c9fdf {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 16:05:24 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_4d960aac-48dd-470e-831f-eaafe64c9fdf/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:05:24 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_4d960aac-48dd-470e-831f-eaafe64c9fdf/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:05:24 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Volume reschedule parameters: True retry: {'num_attempts': 2, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 16:05:24 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Flow 'volume_create_manager' (063aef0d-532b-4252-971f-2ab93c167153) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 16:05:24 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (9c939d09-6904-4b5a-94e0-ff47e99f48a3) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:05:24 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (9c939d09-6904-4b5a-94e0-ff47e99f48a3) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:03:06Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesListTestJSON-Volume-1309258736',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=4d960aac-48dd-470e-831f-eaafe64c9fdf,launched_at=None,metadata={Type='work'},migration_status=None,multiattach=False,previous_status=None,project_id='d1740791cf5e4c7087f45af9015c0a8f',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:05:24Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:05:24Z,use_quota=True,user_id='9091747614a945f6a88207d3662fea8e',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:05:24 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (6f53bed4-0ed9-45a1-a45c-e1f27076bc9b) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:05:24 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (6f53bed4-0ed9-45a1-a45c-e1f27076bc9b) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:05:24 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (a137de86-8d66-433a-9ada-f8d3de4f9dfd) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:05:24 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (a137de86-8d66-433a-9ada-f8d3de4f9dfd) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': '4d960aac-48dd-470e-831f-eaafe64c9fdf', 'volume_name': 'volume-4d960aac-48dd-470e-831f-eaafe64c9fdf', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:05:24 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (a2334ea6-612f-41b1-8291-4525d4beafea) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:05:24 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (a2334ea6-612f-41b1-8291-4525d4beafea) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:05:24 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (f9a2d6a9-94b2-44ac-a82c-5c235bdbf08a) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:05:24 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Volume 4d960aac-48dd-470e-831f-eaafe64c9fdf: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-4d960aac-48dd-470e-831f-eaafe64c9fdf', 'volume_size': 1} Apr 17 16:05:24 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] creating volume v_4d960aac-48dd-470e-831f-eaafe64c9fdf. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 16:05:24 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] create volume {'name': 'v_4d960aac-48dd-470e-831f-eaafe64c9fdf', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 16:05:24 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:05:24 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:05:27 user cinder-volume[84696]: DEBUG cinder.utils [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.973(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:05:27 user cinder-volume[84696]: DEBUG cinder.utils [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_4d960aac-48dd-470e-831f-eaafe64c9fdf/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:05:27 user cinder-volume[84696]: DEBUG cinder.utils [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.863(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:05:27 user cinder-volume[84696]: DEBUG cinder.utils [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:05:27 user cinder-volume[84696]: DEBUG cinder.utils [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.034(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:05:27 user cinder-volume[84696]: DEBUG cinder.utils [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_b4281d17-c1d8-48d3-8a0b-8fd7d2f193cc/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:05:27 user cinder-volume[84696]: DEBUG cinder.utils [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.800(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:05:27 user cinder-volume[84696]: DEBUG cinder.utils [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:05:30 user cinder-volume[84696]: DEBUG cinder.utils [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.041(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:05:30 user cinder-volume[84696]: DEBUG cinder.utils [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_4d960aac-48dd-470e-831f-eaafe64c9fdf/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:05:30 user cinder-volume[84696]: DEBUG cinder.utils [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.866(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:05:30 user cinder-volume[84696]: DEBUG cinder.utils [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:05:33 user cinder-volume[84696]: DEBUG cinder.utils [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.172(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:05:33 user cinder-volume[84696]: DEBUG cinder.utils [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_b4281d17-c1d8-48d3-8a0b-8fd7d2f193cc/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:05:33 user cinder-volume[84696]: DEBUG cinder.utils [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.003(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:05:33 user cinder-volume[84696]: DEBUG cinder.utils [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:05:34 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:INSERT=3,cinder:SELECT=11,cinder:UPDATE=4 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:05:36 user cinder-volume[84696]: DEBUG cinder.utils [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.186(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:05:36 user cinder-volume[84696]: DEBUG cinder.utils [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_4d960aac-48dd-470e-831f-eaafe64c9fdf/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:05:36 user cinder-volume[84696]: DEBUG cinder.utils [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.012(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:05:36 user cinder-volume[84696]: DEBUG cinder.utils [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:05:39 user cinder-volume[84696]: DEBUG cinder.utils [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 74.676(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:05:39 user cinder-volume[84696]: DEBUG cinder.utils [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_26014b4e-ec84-4011-a69f-ed42c3424a24/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:05:39 user cinder-volume[84696]: DEBUG cinder.utils [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 74.481(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:05:39 user cinder-volume[84696]: DEBUG cinder.utils [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:05:42 user cinder-volume[84696]: DEBUG cinder.utils [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.387(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:05:42 user cinder-volume[84696]: DEBUG cinder.utils [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_b4281d17-c1d8-48d3-8a0b-8fd7d2f193cc/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:05:42 user cinder-volume[84696]: DEBUG cinder.utils [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.220(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:05:42 user cinder-volume[84696]: DEBUG cinder.utils [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:05:45 user cinder-volume[84696]: DEBUG cinder.utils [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.402(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:05:45 user cinder-volume[84696]: DEBUG cinder.utils [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_4d960aac-48dd-470e-831f-eaafe64c9fdf/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:05:45 user cinder-volume[84696]: DEBUG cinder.utils [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.227(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:05:45 user cinder-volume[84696]: DEBUG cinder.utils [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:05:45 user cinder-volume[84696]: DEBUG cinder.utils [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 139.184(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:05:45 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_bff6fdbb-a1fb-4b48-bca8-2f7efbc3f0ff/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:05:45 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (0fea95b6-4035-4c1d-b02e-1897705e16da) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 16:05:45 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (eca6f0b1-a41d-49c3-b746-77e961f97b80) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:05:45 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (eca6f0b1-a41d-49c3-b746-77e961f97b80) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:05:45 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Flow 'volume_create_manager' (925f5bf9-fd63-4750-b793-b971d34a6a40) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 16:05:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-fdf1a5ca-318c-4e88-a55c-392a5c930c01 tempest-VolumeRetypeWithoutMigrationTest-683339201 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 16:05:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 16:05:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 16:05:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 16:05:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 16:05:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 16:05:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 16:05:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 16:05:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 16:05:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 16:05:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 16:05:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 16:05:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 16:05:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 16:05:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 16:05:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 16:05:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 16:05:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 16:05:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 16:05:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 16:05:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 16:05:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 16:05:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 16:05:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 16:05:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:05:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 16:05:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:05:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 16:05:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:05:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 16:05:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:05:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 16:05:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:05:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 16:05:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:05:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 16:05:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:05:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 16:05:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:05:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:05:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 16:05:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 16:05:51 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 139.216(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:05:51 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:05:51 user cinder-volume[84696]: ERROR oslo_service.periodic_task [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Error during VolumeManager.publish_service_capabilities: TypeError: 'NoneType' object is not subscriptable Apr 17 16:05:51 user cinder-volume[84696]: ERROR oslo_service.periodic_task Traceback (most recent call last): Apr 17 16:05:51 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/usr/local/lib/python3.10/dist-packages/oslo_service/periodic_task.py", line 216, in run_periodic_tasks Apr 17 16:05:51 user cinder-volume[84696]: ERROR oslo_service.periodic_task task(self, context) Apr 17 16:05:51 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/manager.py", line 2876, in publish_service_capabilities Apr 17 16:05:51 user cinder-volume[84696]: ERROR oslo_service.periodic_task self._report_driver_status(context) Apr 17 16:05:51 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/manager.py", line 2755, in _report_driver_status Apr 17 16:05:51 user cinder-volume[84696]: ERROR oslo_service.periodic_task volume_stats = get_stats() Apr 17 16:05:51 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/usr/local/lib/python3.10/dist-packages/oslo_utils/timeutils.py", line 319, in wrapper Apr 17 16:05:51 user cinder-volume[84696]: ERROR oslo_service.periodic_task result = func(*args, **kwargs) Apr 17 16:05:51 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/manager.py", line 2753, in get_stats Apr 17 16:05:51 user cinder-volume[84696]: ERROR oslo_service.periodic_task return self.driver.get_volume_stats(refresh=True) Apr 17 16:05:51 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/driver.py", line 666, in get_volume_stats Apr 17 16:05:51 user cinder-volume[84696]: ERROR oslo_service.periodic_task self._update_volume_stats() Apr 17 16:05:51 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 726, in _update_volume_stats Apr 17 16:05:51 user cinder-volume[84696]: ERROR oslo_service.periodic_task pool_stats = self.ra.get_pool_stats() Apr 17 16:05:51 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 897, in get_pool_stats Apr 17 16:05:51 user cinder-volume[84696]: ERROR oslo_service.periodic_task if not resp["error"] and resp["code"] == 200: Apr 17 16:05:51 user cinder-volume[84696]: ERROR oslo_service.periodic_task TypeError: 'NoneType' object is not subscriptable Apr 17 16:05:51 user cinder-volume[84696]: ERROR oslo_service.periodic_task Apr 17 16:05:51 user cinder-volume[84696]: WARNING oslo.service.loopingcall [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Function 'cinder.service.Service.periodic_tasks' run outlasted interval by 79.22 sec Apr 17 16:05:51 user cinder-volume[84696]: DEBUG cinder.utils [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 139.194(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:05:51 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_fd313648-5fc1-4e6a-97a0-92ac803b36fd/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:05:51 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (63d555cc-1ab2-4e6a-b847-e940a6de6d3a) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 16:05:51 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (2c7909b5-a2dd-487d-8995-156c9934f822) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:05:51 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (2c7909b5-a2dd-487d-8995-156c9934f822) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:05:51 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Flow 'volume_create_manager' (858235c9-243c-43a5-b5d3-c0af8c60c14f) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 16:05:51 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 16:05:51 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 16:05:51 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 16:05:51 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 16:05:51 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 16:05:51 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 16:05:51 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 16:05:51 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 16:05:51 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 16:05:51 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 16:05:51 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 16:05:51 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 16:05:51 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 16:05:51 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 16:05:51 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 16:05:51 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 16:05:51 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 16:05:51 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 16:05:51 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 16:05:51 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 16:05:51 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 16:05:51 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 16:05:51 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 16:05:51 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 16:05:51 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:05:51 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 16:05:51 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1237, in execute Apr 17 16:05:51 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_from_image(context, Apr 17 16:05:51 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/utils.py", line 670, in _wrapper Apr 17 16:05:51 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return r(f, *args, **kwargs) Apr 17 16:05:51 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/tenacity/__init__.py", line 423, in __call__ Apr 17 16:05:51 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server do = self.iter(retry_state=retry_state) Apr 17 16:05:51 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/tenacity/__init__.py", line 360, in iter Apr 17 16:05:51 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return fut.result() Apr 17 16:05:51 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/lib/python3.10/concurrent/futures/_base.py", line 451, in result Apr 17 16:05:51 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self.__get_result() Apr 17 16:05:51 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result Apr 17 16:05:51 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self._exception Apr 17 16:05:51 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/tenacity/__init__.py", line 426, in __call__ Apr 17 16:05:51 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = fn(*args, **kwargs) Apr 17 16:05:51 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1125, in _create_from_image Apr 17 16:05:51 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_from_image_cache_or_download( Apr 17 16:05:51 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 944, in _create_from_image_cache_or_download Apr 17 16:05:51 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update, cloned = self._create_from_image_cache( Apr 17 16:05:51 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 843, in _create_from_image_cache Apr 17 16:05:51 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_from_source_volume( Apr 17 16:05:51 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 659, in _create_from_source_volume Apr 17 16:05:51 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self.driver.create_cloned_volume(volume, srcvol_ref) Apr 17 16:05:51 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 525, in create_cloned_volume Apr 17 16:05:51 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self._clone_object(vname, cvname) Apr 17 16:05:51 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 461, in _clone_object Apr 17 16:05:51 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_snapshot(oname, coname) Apr 17 16:05:51 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 639, in create_snapshot Apr 17 16:05:51 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201, 204): Apr 17 16:05:51 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 16:05:51 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 16:05:51 user cinder-volume[84696]: DEBUG cinder.utils [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 138.857(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:05:51 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_006f20ae-5306-44fe-afdf-e433f79284e8/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:05:51 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (faf27393-8156-4dee-bd89-b77e4119367a) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 16:05:51 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 16:05:51 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:58:54Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TestVolumeBootPattern-volume-origin-619636601',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=fd313648-5fc1-4e6a-97a0-92ac803b36fd,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='c8a836b47d954cb9a7c1cd7c0f790426',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:03:33Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:03:33Z,use_quota=True,user_id='a80e753322224ec1a897598b5b9ed6ea',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 16:05:51 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:58:54Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TestVolumeBootPattern-volume-origin-619636601',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=fd313648-5fc1-4e6a-97a0-92ac803b36fd,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='c8a836b47d954cb9a7c1cd7c0f790426',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:03:33Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:03:33Z,use_quota=True,user_id='a80e753322224ec1a897598b5b9ed6ea',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=b3f0dec1-7758-44a8-82e0-76ada8837428,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(fd313648-5fc1-4e6a-97a0-92ac803b36fd),volume_id=fd313648-5fc1-4e6a-97a0-92ac803b36fd,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'image', 'volume_id': 'fd313648-5fc1-4e6a-97a0-92ac803b36fd', 'volume_name': 'volume-fd313648-5fc1-4e6a-97a0-92ac803b36fd', 'volume_size': 1, 'image_id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'image_location': (None, None), 'image_meta': {'name': 'cirros-0.5.2-x86_64-disk', 'disk_format': 'qcow2', 'container_format': 'bare', 'visibility': 'public', 'size': 16300544, 'virtual_size': 117440512, 'status': 'active', 'checksum': 'b874c39491a2377b8490f5f1e89761a4', 'protected': False, 'min_ram': 0, 'min_disk': 0, 'owner': 'a43c211b4ed542538799349489b1d884', 'os_hidden': False, 'os_hash_algo': 'sha512', 'os_hash_value': '6b813aa46bb90b4da216a4d19376593fa3f4fc7e617f03a92b7fe11e9a3981cbe8f0959dbebe36225e5f53dc4492341a4863cac4ed1ee0909f3fc78ef9c3e869', 'id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'created_at': datetime.datetime(2023, 4, 17, 15, 19, 53, tzinfo=datetime.timezone.utc), 'updated_at': datetime.datetime(2023, 4, 17, 15, 19, 55, tzinfo=datetime.timezone.utc), 'tags': [], 'file': '/v2/images/b3f0dec1-7758-44a8-82e0-76ada8837428/file', 'properties': {'hw_rng_model': 'virtio', 'owner_specified.openstack.md5': '', 'owner_specified.openstack.object': 'images/cirros-0.5.2-x86_64-disk', 'owner_specified.openstack.sha256': ''}}, 'image_service': }} Apr 17 16:05:51 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': 'b3f0dec1-7758-44a8-82e0-76ada8837428', 'snapshot_id': None, 'source_volid': None, 'volume_id': 'fd313648-5fc1-4e6a-97a0-92ac803b36fd', 'volume': {'id': 'fd313648-5fc1-4e6a-97a0-92ac803b36fd', '_name_id': None, 'ec2_id': None, 'user_id': 'a80e753322224ec1a897598b5b9ed6ea', 'project_id': 'c8a836b47d954cb9a7c1cd7c0f790426', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-TestVolumeBootPattern-volume-origin-619636601', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'admin_metadata': {}, 'glance_metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T15:58:53.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-fd313648-5fc1-4e6a-97a0-92ac803b36fd', 'name_id': 'fd313648-5fc1-4e6a-97a0-92ac803b36fd', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-TestVolumeBootPattern-volume-origin-619636601', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': 'c8a836b47d954cb9a7c1cd7c0f790426', 'qos_specs': None, 'reservations': ['85aec33d-518b-4853-aab2-fb3033224b8f', '56b9d1aa-d720-448b-857d-904a1f5517db', 'acdbd69e-bf21-488b-9caa-1525a1d521b2', 'd0d43e22-774a-4a41-86f9-bcde7e88d4b7'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'a80e753322224ec1a897598b5b9ed6ea', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-TestVolumeBootPattern-volume-origin-619636601', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': 'c8a836b47d954cb9a7c1cd7c0f790426', 'qos_specs': None, 'reservations': ['85aec33d-518b-4853-aab2-fb3033224b8f', '56b9d1aa-d720-448b-857d-904a1f5517db', 'acdbd69e-bf21-488b-9caa-1525a1d521b2', 'd0d43e22-774a-4a41-86f9-bcde7e88d4b7'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'a80e753322224ec1a897598b5b9ed6ea', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': 'a80e753322224ec1a897598b5b9ed6ea', 'metadata': {}, 'qos_specs': None, 'retry': {'num_attempts': 3, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']}}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=b3f0dec1-7758-44a8-82e0-76ada8837428,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(fd313648-5fc1-4e6a-97a0-92ac803b36fd),volume_id=fd313648-5fc1-4e6a-97a0-92ac803b36fd,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:58:54Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TestVolumeBootPattern-volume-origin-619636601',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=fd313648-5fc1-4e6a-97a0-92ac803b36fd,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='c8a836b47d954cb9a7c1cd7c0f790426',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:03:33Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:03:33Z,use_quota=True,user_id='a80e753322224ec1a897598b5b9ed6ea',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 16:05:51 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:58:54Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TestVolumeBootPattern-volume-origin-619636601',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=fd313648-5fc1-4e6a-97a0-92ac803b36fd,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='c8a836b47d954cb9a7c1cd7c0f790426',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:03:33Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:03:33Z,use_quota=True,user_id='a80e753322224ec1a897598b5b9ed6ea',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:58:54Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TestVolumeBootPattern-volume-origin-619636601',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=fd313648-5fc1-4e6a-97a0-92ac803b36fd,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='c8a836b47d954cb9a7c1cd7c0f790426',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:03:33Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:03:33Z,use_quota=True,user_id='a80e753322224ec1a897598b5b9ed6ea',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 16:05:51 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 16:05:51 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 16:05:51 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:05:51 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 16:05:51 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1237, in execute Apr 17 16:05:51 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_from_image(context, Apr 17 16:05:51 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/utils.py", line 670, in _wrapper Apr 17 16:05:51 user cinder-volume[84696]: ERROR cinder.volume.manager return r(f, *args, **kwargs) Apr 17 16:05:51 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/tenacity/__init__.py", line 423, in __call__ Apr 17 16:05:51 user cinder-volume[84696]: ERROR cinder.volume.manager do = self.iter(retry_state=retry_state) Apr 17 16:05:51 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/tenacity/__init__.py", line 360, in iter Apr 17 16:05:51 user cinder-volume[84696]: ERROR cinder.volume.manager return fut.result() Apr 17 16:05:51 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/lib/python3.10/concurrent/futures/_base.py", line 451, in result Apr 17 16:05:51 user cinder-volume[84696]: ERROR cinder.volume.manager return self.__get_result() Apr 17 16:05:51 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result Apr 17 16:05:51 user cinder-volume[84696]: ERROR cinder.volume.manager raise self._exception Apr 17 16:05:51 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/tenacity/__init__.py", line 426, in __call__ Apr 17 16:05:51 user cinder-volume[84696]: ERROR cinder.volume.manager result = fn(*args, **kwargs) Apr 17 16:05:51 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1125, in _create_from_image Apr 17 16:05:51 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_from_image_cache_or_download( Apr 17 16:05:51 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 944, in _create_from_image_cache_or_download Apr 17 16:05:51 user cinder-volume[84696]: ERROR cinder.volume.manager model_update, cloned = self._create_from_image_cache( Apr 17 16:05:51 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 843, in _create_from_image_cache Apr 17 16:05:51 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_from_source_volume( Apr 17 16:05:51 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 659, in _create_from_source_volume Apr 17 16:05:51 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self.driver.create_cloned_volume(volume, srcvol_ref) Apr 17 16:05:51 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 525, in create_cloned_volume Apr 17 16:05:51 user cinder-volume[84696]: ERROR cinder.volume.manager self._clone_object(vname, cvname) Apr 17 16:05:51 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 461, in _clone_object Apr 17 16:05:51 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_snapshot(oname, coname) Apr 17 16:05:51 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 639, in create_snapshot Apr 17 16:05:51 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201, 204): Apr 17 16:05:51 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 16:05:51 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 16:05:51 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (faf27393-8156-4dee-bd89-b77e4119367a) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:05:51 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (faf27393-8156-4dee-bd89-b77e4119367a) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:05:51 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (9ed763ba-f71f-4aab-89eb-ac3e5159ed81) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:05:51 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (9ed763ba-f71f-4aab-89eb-ac3e5159ed81) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:05:51 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (0e04621c-ff7e-420c-9e64-068fc655bc83) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:05:51 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (0e04621c-ff7e-420c-9e64-068fc655bc83) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:05:51 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (468f07b1-f1bb-48dd-b6ea-bcae9f98018c) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:05:51 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Updating volume fd313648-5fc1-4e6a-97a0-92ac803b36fd with {'scheduled_at': datetime.datetime(2023, 4, 17, 16, 5, 51, 929980), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 16:05:51 user cinder-volume[84696]: DEBUG oslo_service.periodic_task [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Running periodic task VolumeManager.publish_service_capabilities {{(pid=84696) run_periodic_tasks /usr/local/lib/python3.10/dist-packages/oslo_service/periodic_task.py:210}} Apr 17 16:05:51 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Updating volume stats {{(pid=84696) _update_volume_stats /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:724}} Apr 17 16:05:51 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Get pool Pool-0 fsprops {{(pid=84696) get_pool_stats /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:894}} Apr 17 16:05:51 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:05:51 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:05:51 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=14,cinder:INSERT=1,cinder:UPDATE=4 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:05:51 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Volume fd313648-5fc1-4e6a-97a0-92ac803b36fd: re-scheduling SchedulerAPI.create_volume attempt 3 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 16:05:51 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Volume fd313648-5fc1-4e6a-97a0-92ac803b36fd: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 16:05:51 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] deleating volume v_fd313648-5fc1-4e6a-97a0-92ac803b36fd {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 16:05:51 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] get snapshots for volume v_fd313648-5fc1-4e6a-97a0-92ac803b36fd {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 16:05:51 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_fd313648-5fc1-4e6a-97a0-92ac803b36fd/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:05:51 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_fd313648-5fc1-4e6a-97a0-92ac803b36fd/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:05:54 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 3.006(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:05:54 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:05:54 user cinder-volume[84696]: DEBUG cinder.utils [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 71.602(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:05:54 user cinder-volume[84696]: DEBUG cinder.utils [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_9a399a62-6a8f-4b6d-8ecf-8dcdf57e5fbb/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:05:54 user cinder-volume[84696]: DEBUG cinder.utils [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.984(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:05:54 user cinder-volume[84696]: DEBUG cinder.utils [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_fd313648-5fc1-4e6a-97a0-92ac803b36fd/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:05:55 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Volume reschedule parameters: True retry: {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 16:05:55 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Flow 'volume_create_manager' (0572e5ad-45d0-4358-9be6-8f98fe771e9b) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 16:05:55 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (8af09775-aa19-414c-9fc5-d5c8ebe3e6e1) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:05:55 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (8af09775-aa19-414c-9fc5-d5c8ebe3e6e1) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:05:55Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-AttachVolumeTestJSON-volume-1580982693',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=65a642c3-b1e1-4b80-9bd2-f00151b6304a,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='2eb998c84b5b41e599af9636b929295d',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:05:56Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:05:56Z,use_quota=True,user_id='b9d4a317838b47d6a2a2e7eab7a453dd',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:05:55 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (69725152-efb8-4ceb-b719-32a1332715d2) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:05:55 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (69725152-efb8-4ceb-b719-32a1332715d2) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:05:55 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (ed007a2f-e22a-4ae0-a101-6944585cfb78) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:05:55 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (ed007a2f-e22a-4ae0-a101-6944585cfb78) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': '65a642c3-b1e1-4b80-9bd2-f00151b6304a', 'volume_name': 'volume-65a642c3-b1e1-4b80-9bd2-f00151b6304a', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:05:55 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (9e245c48-21e1-45a6-a58e-ba1a497620cb) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:05:55 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (9e245c48-21e1-45a6-a58e-ba1a497620cb) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:05:55 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (d65e06e5-e3d2-47ff-9ea3-963b4990bcc6) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:05:55 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Volume 65a642c3-b1e1-4b80-9bd2-f00151b6304a: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-65a642c3-b1e1-4b80-9bd2-f00151b6304a', 'volume_size': 1} Apr 17 16:05:55 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] creating volume v_65a642c3-b1e1-4b80-9bd2-f00151b6304a. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 16:05:55 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] create volume {'name': 'v_65a642c3-b1e1-4b80-9bd2-f00151b6304a', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 16:05:55 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:05:55 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:05:56 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-008ec7fb-052b-43ae-952b-3c2d087ab270 req-6f2f6ab3-3788-404c-8060-061f30d95d4f tempest-ServerActionsTestJSON-645547919 None] terminate connection for 1072d05f-2ebb-4398-902c-902df25464e9 {{(pid=84696) terminate_connection /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:1032}} Apr 17 16:05:56 user cinder-volume[84696]: INFO cinder.volume.manager [req-008ec7fb-052b-43ae-952b-3c2d087ab270 req-6f2f6ab3-3788-404c-8060-061f30d95d4f tempest-ServerActionsTestJSON-645547919 None] Terminate volume connection completed successfully. Apr 17 16:05:56 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-008ec7fb-052b-43ae-952b-3c2d087ab270 req-6f2f6ab3-3788-404c-8060-061f30d95d4f tempest-ServerActionsTestJSON-645547919 None] Deleting attachment a68f5fc4-dea3-48a4-bdb0-1d89db356c29. {{(pid=84696) attachment_delete /opt/stack/cinder/cinder/volume/manager.py:5006}} Apr 17 16:05:56 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-008ec7fb-052b-43ae-952b-3c2d087ab270 req-6f2f6ab3-3788-404c-8060-061f30d95d4f tempest-ServerActionsTestJSON-645547919 None] remove_export for volume: 1072d05f-2ebb-4398-902c-902df25464e9. {{(pid=84696) remove_export /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:718}} Apr 17 16:05:56 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-008ec7fb-052b-43ae-952b-3c2d087ab270 req-6f2f6ab3-3788-404c-8060-061f30d95d4f tempest-ServerActionsTestJSON-645547919 None] remove export {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:912}} Apr 17 16:05:56 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-008ec7fb-052b-43ae-952b-3c2d087ab270 req-6f2f6ab3-3788-404c-8060-061f30d95d4f tempest-ServerActionsTestJSON-645547919 None] detach volume:Volume(_name_id=None,admin_metadata=,attach_status='attached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:27:39Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-ServerActionsTestJSON-volume-1520907984',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=1072d05f-2ebb-4398-902c-902df25464e9,launched_at=2023-04-17T15:27:40Z,metadata=,migration_status=None,multiattach=False,previous_status=None,project_id='cb5f6b5ed2fc43b69c47e1fc18646545',provider_auth='CHAP NKISaVyG nHFlRwNr8b5KS2',provider_geometry=None,provider_id=None,provider_location='172.16.0.220:3260,1 iqn.2010-10.org.openstack:1072d05f-2ebb-4398-902c-902df25464e9 0',replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:27:39Z,service_uuid='e628352e-c4cf-40eb-8389-cce40663026a',shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='in-use',terminated_at=None,updated_at=2023-04-17T15:32:47Z,use_quota=True,user_id='638a4f535e9740fc9d810639a5eb24d7',volume_attachment=,volume_type=,volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c) from target:iqn.2010-10.org.openstack:1072d05f-2ebb-4398-902c-902df25464e9. {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:913}} Apr 17 16:05:56 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [req-008ec7fb-052b-43ae-952b-3c2d087ab270 req-6f2f6ab3-3788-404c-8060-061f30d95d4f tempest-ServerActionsTestJSON-645547919 None] detach volume v_1072d05f-2ebb-4398-902c-902df25464e9 from target iqn.2010-10.org.openstack:1072d05f-2ebb-4398-902c-902df25464e9 {{(pid=84696) detach_target_vol /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:606}} Apr 17 16:05:56 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-008ec7fb-052b-43ae-952b-3c2d087ab270 req-6f2f6ab3-3788-404c-8060-061f30d95d4f tempest-ServerActionsTestJSON-645547919 None] Sending pool request DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:1072d05f-2ebb-4398-902c-902df25464e9/luns/v_1072d05f-2ebb-4398-902c-902df25464e9 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:05:56 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-008ec7fb-052b-43ae-952b-3c2d087ab270 req-6f2f6ab3-3788-404c-8060-061f30d95d4f tempest-ServerActionsTestJSON-645547919 None] Sending DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:1072d05f-2ebb-4398-902c-902df25464e9/luns/v_1072d05f-2ebb-4398-902c-902df25464e9 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:05:58 user cinder-volume[84696]: DEBUG cinder.utils [req-008ec7fb-052b-43ae-952b-3c2d087ab270 req-6f2f6ab3-3788-404c-8060-061f30d95d4f tempest-ServerActionsTestJSON-645547919 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 1.648(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:05:58 user cinder-volume[84696]: DEBUG cinder.utils [req-008ec7fb-052b-43ae-952b-3c2d087ab270 req-6f2f6ab3-3788-404c-8060-061f30d95d4f tempest-ServerActionsTestJSON-645547919 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:1072d05f-2ebb-4398-902c-902df25464e9/luns/v_1072d05f-2ebb-4398-902c-902df25464e9 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:05:58 user cinder-volume[84696]: DEBUG cinder.utils [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 139.196(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:05:58 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_63b2c7c3-8482-49d4-bb89-c4ccf3f2db82/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:05:58 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (a81e2362-7b10-448f-b218-2b651857d634) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 16:05:58 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (86e0addb-999b-4b63-9589-3d1a3f018737) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:05:58 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (86e0addb-999b-4b63-9589-3d1a3f018737) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:05:58 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Flow 'volume_create_manager' (f23579c3-74ef-405a-935d-f3d1e46c2757) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 16:05:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 16:05:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 16:05:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 16:05:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 16:05:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 16:05:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 16:05:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 16:05:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 16:05:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 16:05:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 16:05:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 16:05:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 16:05:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 16:05:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 16:05:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 16:05:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 16:05:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 16:05:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 16:05:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 16:05:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 16:05:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 16:05:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 16:05:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 16:05:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 16:05:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:05:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 16:05:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:05:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 16:05:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:05:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 16:05:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:05:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 16:05:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:05:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 16:05:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:05:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 16:05:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:05:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 16:05:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:05:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:05:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 16:05:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 16:05:58 user cinder-volume[84696]: DEBUG cinder.utils [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 139.014(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:05:58 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:05:58 user cinder-volume[84696]: INFO cinder.message.api [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Creating message record for request_id = req-a293a946-8aa4-47bc-8290-fe9922d7ae21 Apr 17 16:05:58 user cinder-volume[84696]: DEBUG cinder.utils [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.388(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:05:58 user cinder-volume[84696]: DEBUG cinder.utils [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:05:58 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.105(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:05:58 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:05:58 user cinder-volume[84696]: DEBUG cinder.utils [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.080(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:05:58 user cinder-volume[84696]: DEBUG cinder.utils [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_fd313648-5fc1-4e6a-97a0-92ac803b36fd/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:05:58 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 16:05:58 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (3d9e112f-3299-4b47-8eba-5907b33f5509) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 16:05:58 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 16:05:58 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:01:19Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-AttachVolumeNegativeTest-volume-1267367334',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=63b2c7c3-8482-49d4-bb89-c4ccf3f2db82,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='8586cc10142d4743874f43cdb0f37aba',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:03:39Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:03:39Z,use_quota=True,user_id='9fdc7c80b4654692a52c8a4c4149324f',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 16:05:58 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:01:19Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-AttachVolumeNegativeTest-volume-1267367334',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=63b2c7c3-8482-49d4-bb89-c4ccf3f2db82,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='8586cc10142d4743874f43cdb0f37aba',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:03:39Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:03:39Z,use_quota=True,user_id='9fdc7c80b4654692a52c8a4c4149324f',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(63b2c7c3-8482-49d4-bb89-c4ccf3f2db82),volume_id=63b2c7c3-8482-49d4-bb89-c4ccf3f2db82,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': '63b2c7c3-8482-49d4-bb89-c4ccf3f2db82', 'volume_name': 'volume-63b2c7c3-8482-49d4-bb89-c4ccf3f2db82', 'volume_size': 1}} Apr 17 16:05:58 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': '63b2c7c3-8482-49d4-bb89-c4ccf3f2db82', 'volume': {'id': '63b2c7c3-8482-49d4-bb89-c4ccf3f2db82', '_name_id': None, 'ec2_id': None, 'user_id': '9fdc7c80b4654692a52c8a4c4149324f', 'project_id': '8586cc10142d4743874f43cdb0f37aba', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-AttachVolumeNegativeTest-volume-1267367334', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'admin_metadata': {}, 'glance_metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T16:01:19.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-63b2c7c3-8482-49d4-bb89-c4ccf3f2db82', 'name_id': '63b2c7c3-8482-49d4-bb89-c4ccf3f2db82', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-AttachVolumeNegativeTest-volume-1267367334', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '8586cc10142d4743874f43cdb0f37aba', 'qos_specs': None, 'reservations': ['60706a3b-0211-488a-91cd-03061d7521a9', '3ccbaa06-3cae-4fa4-b8ac-ab913175fa13', '74bd9b1b-1b22-4085-aa81-b422f52940be', '3a23250a-64d8-4fa2-826b-865608d68d21'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': '9fdc7c80b4654692a52c8a4c4149324f', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-AttachVolumeNegativeTest-volume-1267367334', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '8586cc10142d4743874f43cdb0f37aba', 'qos_specs': None, 'reservations': ['60706a3b-0211-488a-91cd-03061d7521a9', '3ccbaa06-3cae-4fa4-b8ac-ab913175fa13', '74bd9b1b-1b22-4085-aa81-b422f52940be', '3a23250a-64d8-4fa2-826b-865608d68d21'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': '9fdc7c80b4654692a52c8a4c4149324f', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': '9fdc7c80b4654692a52c8a4c4149324f', 'metadata': {}, 'qos_specs': None, 'retry': {'num_attempts': 2, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']}}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(63b2c7c3-8482-49d4-bb89-c4ccf3f2db82),volume_id=63b2c7c3-8482-49d4-bb89-c4ccf3f2db82,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:01:19Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-AttachVolumeNegativeTest-volume-1267367334',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=63b2c7c3-8482-49d4-bb89-c4ccf3f2db82,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='8586cc10142d4743874f43cdb0f37aba',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:03:39Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:03:39Z,use_quota=True,user_id='9fdc7c80b4654692a52c8a4c4149324f',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 16:05:58 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:01:19Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-AttachVolumeNegativeTest-volume-1267367334',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=63b2c7c3-8482-49d4-bb89-c4ccf3f2db82,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='8586cc10142d4743874f43cdb0f37aba',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:03:39Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:03:39Z,use_quota=True,user_id='9fdc7c80b4654692a52c8a4c4149324f',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:01:19Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-AttachVolumeNegativeTest-volume-1267367334',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=63b2c7c3-8482-49d4-bb89-c4ccf3f2db82,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='8586cc10142d4743874f43cdb0f37aba',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:03:39Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:03:39Z,use_quota=True,user_id='9fdc7c80b4654692a52c8a4c4149324f',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 16:05:58 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 16:05:58 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 16:05:58 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:05:58 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 16:05:58 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:05:58 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 16:05:58 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:05:58 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 16:05:58 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:05:58 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 16:05:58 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:05:58 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 16:05:58 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:05:58 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 16:05:58 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:05:58 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 16:05:58 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:05:58 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:05:58 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 16:05:58 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 16:05:58 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (3d9e112f-3299-4b47-8eba-5907b33f5509) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:05:58 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (3d9e112f-3299-4b47-8eba-5907b33f5509) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:05:58 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (f547cce5-2b90-460c-ad6e-9add1931f684) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:05:58 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (f547cce5-2b90-460c-ad6e-9add1931f684) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:05:58 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (592eb3de-507d-4938-87d0-adb3814a7e3b) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:05:58 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (592eb3de-507d-4938-87d0-adb3814a7e3b) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:05:58 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (6903556b-4102-4597-bd53-3965cfeb9219) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:05:58 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Updating volume 63b2c7c3-8482-49d4-bb89-c4ccf3f2db82 with {'scheduled_at': datetime.datetime(2023, 4, 17, 16, 5, 58, 99717), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 16:05:58 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Volume 63b2c7c3-8482-49d4-bb89-c4ccf3f2db82: re-scheduling SchedulerAPI.create_volume attempt 2 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 16:05:58 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Volume 63b2c7c3-8482-49d4-bb89-c4ccf3f2db82: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 16:05:58 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] deleating volume v_63b2c7c3-8482-49d4-bb89-c4ccf3f2db82 {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 16:05:58 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] get snapshots for volume v_63b2c7c3-8482-49d4-bb89-c4ccf3f2db82 {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 16:05:58 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_63b2c7c3-8482-49d4-bb89-c4ccf3f2db82/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:05:58 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_63b2c7c3-8482-49d4-bb89-c4ccf3f2db82/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:05:58 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Volume reschedule parameters: True retry: {'num_attempts': 3, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 16:05:58 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Flow 'volume_create_manager' (4c14647e-2769-45ca-8abd-b0a1a8f00fed) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 16:05:58 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (bf5fe72f-b2c5-448c-901a-3175747e95c0) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:05:58 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (bf5fe72f-b2c5-448c-901a-3175747e95c0) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:01:19Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-AttachVolumeNegativeTest-volume-1267367334',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=63b2c7c3-8482-49d4-bb89-c4ccf3f2db82,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='8586cc10142d4743874f43cdb0f37aba',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:05:58Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:05:58Z,use_quota=True,user_id='9fdc7c80b4654692a52c8a4c4149324f',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:05:58 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (ae393033-9a6b-4c77-a97c-754828be17e4) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:05:58 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (ae393033-9a6b-4c77-a97c-754828be17e4) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:05:58 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (4f04b455-a4c0-4ec3-b802-67aef6d02b08) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:05:58 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (4f04b455-a4c0-4ec3-b802-67aef6d02b08) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': '63b2c7c3-8482-49d4-bb89-c4ccf3f2db82', 'volume_name': 'volume-63b2c7c3-8482-49d4-bb89-c4ccf3f2db82', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:05:58 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (2f6ededf-ecea-4107-85b8-57d0f74076ad) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:05:58 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (2f6ededf-ecea-4107-85b8-57d0f74076ad) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:05:58 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (39af473a-c09b-4053-a9b3-4e1083fb4a01) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:05:58 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Volume 63b2c7c3-8482-49d4-bb89-c4ccf3f2db82: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-63b2c7c3-8482-49d4-bb89-c4ccf3f2db82', 'volume_size': 1} Apr 17 16:05:58 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] creating volume v_63b2c7c3-8482-49d4-bb89-c4ccf3f2db82. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 16:05:58 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] create volume {'name': 'v_63b2c7c3-8482-49d4-bb89-c4ccf3f2db82', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 16:05:58 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:05:58 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:06:01 user cinder-volume[84696]: DEBUG cinder.utils [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.924(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:06:01 user cinder-volume[84696]: DEBUG cinder.utils [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_63b2c7c3-8482-49d4-bb89-c4ccf3f2db82/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:06:01 user cinder-volume[84696]: DEBUG cinder.utils [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.738(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:06:01 user cinder-volume[84696]: DEBUG cinder.utils [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:06:01 user cinder-volume[84696]: DEBUG cinder.utils [req-008ec7fb-052b-43ae-952b-3c2d087ab270 req-6f2f6ab3-3788-404c-8060-061f30d95d4f tempest-ServerActionsTestJSON-645547919 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 4.728(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:06:01 user cinder-volume[84696]: DEBUG cinder.utils [req-008ec7fb-052b-43ae-952b-3c2d087ab270 req-6f2f6ab3-3788-404c-8060-061f30d95d4f tempest-ServerActionsTestJSON-645547919 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:1072d05f-2ebb-4398-902c-902df25464e9/luns/v_1072d05f-2ebb-4398-902c-902df25464e9 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:06:01 user cinder-volume[84696]: DEBUG cinder.utils [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.658(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:06:01 user cinder-volume[84696]: DEBUG cinder.utils [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:06:01 user cinder-volume[84696]: DEBUG cinder.utils [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.828(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:06:01 user cinder-volume[84696]: DEBUG cinder.utils [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_b4281d17-c1d8-48d3-8a0b-8fd7d2f193cc/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:06:01 user cinder-volume[84696]: DEBUG cinder.utils [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.445(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:06:01 user cinder-volume[84696]: DEBUG cinder.utils [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:06:04 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.222(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:06:04 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:06:04 user cinder-volume[84696]: DEBUG cinder.utils [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.658(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:06:04 user cinder-volume[84696]: DEBUG cinder.utils [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:06:04 user cinder-volume[84696]: DEBUG cinder.utils [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.836(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:06:04 user cinder-volume[84696]: DEBUG cinder.utils [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_4d960aac-48dd-470e-831f-eaafe64c9fdf/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:06:04 user cinder-volume[84696]: DEBUG cinder.utils [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.807(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:06:04 user cinder-volume[84696]: DEBUG cinder.utils [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:06:04 user cinder-volume[84696]: DEBUG cinder.utils [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.995(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:06:04 user cinder-volume[84696]: DEBUG cinder.utils [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_63b2c7c3-8482-49d4-bb89-c4ccf3f2db82/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:06:04 user cinder-volume[84696]: DEBUG cinder.utils [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.201(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:06:04 user cinder-volume[84696]: DEBUG cinder.utils [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_fd313648-5fc1-4e6a-97a0-92ac803b36fd/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:06:07 user cinder-volume[84696]: DEBUG cinder.utils [req-008ec7fb-052b-43ae-952b-3c2d087ab270 req-6f2f6ab3-3788-404c-8060-061f30d95d4f tempest-ServerActionsTestJSON-645547919 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 10.868(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:06:07 user cinder-volume[84696]: DEBUG cinder.utils [req-008ec7fb-052b-43ae-952b-3c2d087ab270 req-6f2f6ab3-3788-404c-8060-061f30d95d4f tempest-ServerActionsTestJSON-645547919 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:1072d05f-2ebb-4398-902c-902df25464e9/luns/v_1072d05f-2ebb-4398-902c-902df25464e9 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:06:07 user cinder-volume[84696]: DEBUG cinder.utils [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 11.586(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:06:07 user cinder-volume[84696]: DEBUG cinder.utils [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:06:10 user cinder-volume[84696]: DEBUG cinder.utils [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 11.948(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:06:10 user cinder-volume[84696]: DEBUG cinder.utils [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:06:10 user cinder-volume[84696]: DEBUG cinder.utils [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.137(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:06:10 user cinder-volume[84696]: DEBUG cinder.utils [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_63b2c7c3-8482-49d4-bb89-c4ccf3f2db82/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:06:13 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.438(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:06:13 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:06:13 user cinder-volume[84696]: DEBUG cinder.utils [req-008ec7fb-052b-43ae-952b-3c2d087ab270 req-843599cc-4c80-4fe3-81ef-b83c113ecd84 tempest-ServerActionsTestJSON-645547919 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 137.048(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:06:13 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-008ec7fb-052b-43ae-952b-3c2d087ab270 req-843599cc-4c80-4fe3-81ef-b83c113ecd84 tempest-ServerActionsTestJSON-645547919 None] Geting None from DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:1072d05f-2ebb-4398-902c-902df25464e9/luns/v_1072d05f-2ebb-4398-902c-902df25464e9 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:06:13 user cinder-volume[84696]: WARNING cinder.volume.manager [req-008ec7fb-052b-43ae-952b-3c2d087ab270 req-843599cc-4c80-4fe3-81ef-b83c113ecd84 tempest-ServerActionsTestJSON-645547919 None] Failed to detach volume on the backend, ignoring failure 'NoneType' object is not subscriptable Apr 17 16:06:13 user cinder-volume[84696]: DEBUG cinder.utils [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.416(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:06:13 user cinder-volume[84696]: DEBUG cinder.utils [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_fd313648-5fc1-4e6a-97a0-92ac803b36fd/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:06:14 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=15,cinder:UPDATE=2,cinder:INSERT=3 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:06:16 user cinder-volume[84696]: DEBUG cinder.utils [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 20.795(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:06:16 user cinder-volume[84696]: DEBUG cinder.utils [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:06:16 user cinder-volume[84696]: DEBUG cinder.utils [req-008ec7fb-052b-43ae-952b-3c2d087ab270 req-6f2f6ab3-3788-404c-8060-061f30d95d4f tempest-ServerActionsTestJSON-645547919 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 20.081(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:06:16 user cinder-volume[84696]: DEBUG cinder.utils [req-008ec7fb-052b-43ae-952b-3c2d087ab270 req-6f2f6ab3-3788-404c-8060-061f30d95d4f tempest-ServerActionsTestJSON-645547919 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:1072d05f-2ebb-4398-902c-902df25464e9/luns/v_1072d05f-2ebb-4398-902c-902df25464e9 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:06:19 user cinder-volume[84696]: DEBUG cinder.utils [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.164(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:06:19 user cinder-volume[84696]: DEBUG cinder.utils [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:06:19 user cinder-volume[84696]: DEBUG cinder.utils [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.353(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:06:19 user cinder-volume[84696]: DEBUG cinder.utils [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_63b2c7c3-8482-49d4-bb89-c4ccf3f2db82/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:06:24 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=17,cinder:UPDATE=8 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:06:32 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 40.512(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:06:32 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:06:32 user cinder-volume[84696]: DEBUG cinder.utils [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 40.487(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:06:32 user cinder-volume[84696]: DEBUG cinder.utils [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_fd313648-5fc1-4e6a-97a0-92ac803b36fd/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:06:35 user cinder-volume[84696]: DEBUG cinder.utils [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.867(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:06:35 user cinder-volume[84696]: DEBUG cinder.utils [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:06:35 user cinder-volume[84696]: DEBUG cinder.utils [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 74.253(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:06:35 user cinder-volume[84696]: DEBUG cinder.utils [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_b4281d17-c1d8-48d3-8a0b-8fd7d2f193cc/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:06:35 user cinder-volume[84696]: DEBUG cinder.utils [req-008ec7fb-052b-43ae-952b-3c2d087ab270 req-6f2f6ab3-3788-404c-8060-061f30d95d4f tempest-ServerActionsTestJSON-645547919 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.155(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:06:35 user cinder-volume[84696]: DEBUG cinder.utils [req-008ec7fb-052b-43ae-952b-3c2d087ab270 req-6f2f6ab3-3788-404c-8060-061f30d95d4f tempest-ServerActionsTestJSON-645547919 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:1072d05f-2ebb-4398-902c-902df25464e9/luns/v_1072d05f-2ebb-4398-902c-902df25464e9 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:06:35 user cinder-volume[84696]: DEBUG cinder.utils [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 74.085(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:06:35 user cinder-volume[84696]: DEBUG cinder.utils [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:06:38 user cinder-volume[84696]: DEBUG cinder.utils [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 40.236(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:06:38 user cinder-volume[84696]: DEBUG cinder.utils [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:06:38 user cinder-volume[84696]: DEBUG cinder.utils [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 40.425(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:06:38 user cinder-volume[84696]: DEBUG cinder.utils [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_63b2c7c3-8482-49d4-bb89-c4ccf3f2db82/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:06:38 user cinder-volume[84696]: DEBUG cinder.utils [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 74.268(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:06:38 user cinder-volume[84696]: DEBUG cinder.utils [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_4d960aac-48dd-470e-831f-eaafe64c9fdf/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:06:38 user cinder-volume[84696]: DEBUG cinder.utils [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 74.092(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:06:38 user cinder-volume[84696]: DEBUG cinder.utils [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:06:45 user cinder-volume[84696]: DEBUG cinder.utils [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 140.016(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:06:45 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:06:45 user cinder-volume[84696]: INFO cinder.message.api [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Creating message record for request_id = req-8969563c-70db-45ae-8d53-74d80da7efc1 Apr 17 16:06:45 user cinder-volume[84696]: DEBUG cinder.utils [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 140.218(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:06:45 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_26014b4e-ec84-4011-a69f-ed42c3424a24/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:06:45 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (a35054f0-be95-43c7-93ec-8f4d0735bc3d) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 16:06:45 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (ae4d6c1a-cf85-41e0-85a5-b0980a5d748c) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:06:45 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (ae4d6c1a-cf85-41e0-85a5-b0980a5d748c) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:06:45 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Flow 'volume_create_manager' (e561083c-5024-4447-8401-bbcd3d4cddcc) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 16:06:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 16:06:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 16:06:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 16:06:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 16:06:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 16:06:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 16:06:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 16:06:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 16:06:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 16:06:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 16:06:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 16:06:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 16:06:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 16:06:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 16:06:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 16:06:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 16:06:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 16:06:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 16:06:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 16:06:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 16:06:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 16:06:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 16:06:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 16:06:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 16:06:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:06:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 16:06:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:06:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 16:06:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:06:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 16:06:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:06:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 16:06:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:06:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 16:06:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:06:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 16:06:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:06:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 16:06:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:06:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:06:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 16:06:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 16:06:45 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 16:06:45 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=8,cinder:UPDATE=3,cinder:INSERT=1 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:06:45 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (79508278-836c-4b32-9b74-a94b07600a38) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 16:06:45 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 16:06:45 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:59:46Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesListAdminTestJSON-Volume-237491364',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=26014b4e-ec84-4011-a69f-ed42c3424a24,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='73222838c1d4451281d075e1f9c092b3',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:04:25Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:04:25Z,use_quota=True,user_id='6a809fbfe4264147a2716c1cc5914a28',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 16:06:45 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:59:46Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesListAdminTestJSON-Volume-237491364',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=26014b4e-ec84-4011-a69f-ed42c3424a24,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='73222838c1d4451281d075e1f9c092b3',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:04:25Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:04:25Z,use_quota=True,user_id='6a809fbfe4264147a2716c1cc5914a28',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(26014b4e-ec84-4011-a69f-ed42c3424a24),volume_id=26014b4e-ec84-4011-a69f-ed42c3424a24,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': '26014b4e-ec84-4011-a69f-ed42c3424a24', 'volume_name': 'volume-26014b4e-ec84-4011-a69f-ed42c3424a24', 'volume_size': 1}} Apr 17 16:06:45 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': '26014b4e-ec84-4011-a69f-ed42c3424a24', 'volume': {'id': '26014b4e-ec84-4011-a69f-ed42c3424a24', '_name_id': None, 'ec2_id': None, 'user_id': '6a809fbfe4264147a2716c1cc5914a28', 'project_id': '73222838c1d4451281d075e1f9c092b3', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-VolumesListAdminTestJSON-Volume-237491364', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'admin_metadata': {}, 'glance_metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T15:59:45.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-26014b4e-ec84-4011-a69f-ed42c3424a24', 'name_id': '26014b4e-ec84-4011-a69f-ed42c3424a24', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumesListAdminTestJSON-Volume-237491364', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '73222838c1d4451281d075e1f9c092b3', 'qos_specs': None, 'reservations': ['aae7b3c8-9b21-42c2-b7d1-365a54453839', '40630276-bfc3-4923-b711-7d6a45c12c52', '251f733a-f47a-4155-8d77-c0c8abe4451b', '7007e6ad-6925-4dc6-bc99-8f9d477393a9'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': '6a809fbfe4264147a2716c1cc5914a28', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumesListAdminTestJSON-Volume-237491364', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '73222838c1d4451281d075e1f9c092b3', 'qos_specs': None, 'reservations': ['aae7b3c8-9b21-42c2-b7d1-365a54453839', '40630276-bfc3-4923-b711-7d6a45c12c52', '251f733a-f47a-4155-8d77-c0c8abe4451b', '7007e6ad-6925-4dc6-bc99-8f9d477393a9'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': '6a809fbfe4264147a2716c1cc5914a28', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': '6a809fbfe4264147a2716c1cc5914a28', 'metadata': {}, 'qos_specs': None, 'retry': {'num_attempts': 3, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']}}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(26014b4e-ec84-4011-a69f-ed42c3424a24),volume_id=26014b4e-ec84-4011-a69f-ed42c3424a24,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:59:46Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesListAdminTestJSON-Volume-237491364',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=26014b4e-ec84-4011-a69f-ed42c3424a24,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='73222838c1d4451281d075e1f9c092b3',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:04:25Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:04:25Z,use_quota=True,user_id='6a809fbfe4264147a2716c1cc5914a28',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 16:06:45 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:59:46Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesListAdminTestJSON-Volume-237491364',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=26014b4e-ec84-4011-a69f-ed42c3424a24,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='73222838c1d4451281d075e1f9c092b3',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:04:25Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:04:25Z,use_quota=True,user_id='6a809fbfe4264147a2716c1cc5914a28',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:59:46Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesListAdminTestJSON-Volume-237491364',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=26014b4e-ec84-4011-a69f-ed42c3424a24,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='73222838c1d4451281d075e1f9c092b3',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:04:25Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:04:25Z,use_quota=True,user_id='6a809fbfe4264147a2716c1cc5914a28',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 16:06:45 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 16:06:45 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 16:06:45 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:06:45 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 16:06:45 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:06:45 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 16:06:45 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:06:45 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 16:06:45 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:06:45 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 16:06:45 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:06:45 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 16:06:45 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:06:45 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 16:06:45 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:06:45 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 16:06:45 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:06:45 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:06:45 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 16:06:45 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 16:06:45 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (79508278-836c-4b32-9b74-a94b07600a38) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:06:45 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (79508278-836c-4b32-9b74-a94b07600a38) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:06:45 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (39ffc2ad-3cb4-47af-8cb6-005975eb2062) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:06:45 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (39ffc2ad-3cb4-47af-8cb6-005975eb2062) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:06:45 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (18195d25-eb30-419f-bf9e-82839ffdfac9) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:06:45 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (18195d25-eb30-419f-bf9e-82839ffdfac9) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:06:45 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (b17bc9b9-8b5d-4670-9061-37ba32d86af2) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:06:45 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Updating volume 26014b4e-ec84-4011-a69f-ed42c3424a24 with {'scheduled_at': datetime.datetime(2023, 4, 17, 16, 6, 45, 209477), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 16:06:45 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Volume 26014b4e-ec84-4011-a69f-ed42c3424a24: re-scheduling SchedulerAPI.create_volume attempt 3 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 16:06:45 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Volume 26014b4e-ec84-4011-a69f-ed42c3424a24: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 16:06:45 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] deleating volume v_26014b4e-ec84-4011-a69f-ed42c3424a24 {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 16:06:45 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] get snapshots for volume v_26014b4e-ec84-4011-a69f-ed42c3424a24 {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 16:06:45 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_26014b4e-ec84-4011-a69f-ed42c3424a24/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:06:45 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_26014b4e-ec84-4011-a69f-ed42c3424a24/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:06:48 user cinder-volume[84696]: DEBUG cinder.utils [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.953(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:06:48 user cinder-volume[84696]: DEBUG cinder.utils [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_26014b4e-ec84-4011-a69f-ed42c3424a24/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:06:51 user cinder-volume[84696]: DEBUG cinder.utils [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.024(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:06:51 user cinder-volume[84696]: DEBUG cinder.utils [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_26014b4e-ec84-4011-a69f-ed42c3424a24/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:06:58 user cinder-volume[84696]: DEBUG cinder.utils [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.904(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:06:58 user cinder-volume[84696]: DEBUG cinder.utils [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_26014b4e-ec84-4011-a69f-ed42c3424a24/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:07:01 user cinder-volume[84696]: DEBUG cinder.utils [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 137.877(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:07:01 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_9a399a62-6a8f-4b6d-8ecf-8dcdf57e5fbb/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:07:01 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (06e44fc5-64be-4eb1-a659-1214b2f82192) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 16:07:01 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (5d2f7300-9d37-486b-a275-c5f243b63c15) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:07:01 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (5d2f7300-9d37-486b-a275-c5f243b63c15) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:07:01 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Flow 'volume_create_manager' (41be6685-be50-442b-9510-ccce73f8da2b) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 16:07:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-dc8764a6-903f-44ac-86a4-76a8d1914785 tempest-VolumeRevertTests-63962236 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 16:07:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 16:07:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 16:07:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 16:07:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 16:07:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 16:07:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 16:07:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 16:07:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 16:07:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 16:07:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 16:07:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 16:07:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 16:07:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 16:07:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 16:07:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 16:07:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 16:07:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 16:07:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 16:07:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 16:07:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 16:07:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 16:07:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 16:07:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 16:07:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:07:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 16:07:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:07:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 16:07:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:07:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 16:07:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:07:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 16:07:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:07:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 16:07:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:07:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 16:07:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:07:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 16:07:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:07:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:07:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 16:07:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 16:07:04 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=4,cinder:UPDATE=1,cinder:INSERT=1 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:07:07 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 75.422(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:07:07 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:07:07 user cinder-volume[84696]: DEBUG cinder.utils [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 75.398(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:07:07 user cinder-volume[84696]: DEBUG cinder.utils [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_fd313648-5fc1-4e6a-97a0-92ac803b36fd/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:07:07 user cinder-volume[84696]: DEBUG cinder.utils [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 22.122(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:07:07 user cinder-volume[84696]: DEBUG cinder.utils [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_26014b4e-ec84-4011-a69f-ed42c3424a24/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:07:10 user cinder-volume[84696]: DEBUG cinder.utils [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 74.779(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:07:10 user cinder-volume[84696]: DEBUG cinder.utils [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:07:10 user cinder-volume[84696]: DEBUG cinder.utils [req-008ec7fb-052b-43ae-952b-3c2d087ab270 req-6f2f6ab3-3788-404c-8060-061f30d95d4f tempest-ServerActionsTestJSON-645547919 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 74.065(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:07:10 user cinder-volume[84696]: DEBUG cinder.utils [req-008ec7fb-052b-43ae-952b-3c2d087ab270 req-6f2f6ab3-3788-404c-8060-061f30d95d4f tempest-ServerActionsTestJSON-645547919 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:1072d05f-2ebb-4398-902c-902df25464e9/luns/v_1072d05f-2ebb-4398-902c-902df25464e9 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:07:13 user cinder-volume[84696]: DEBUG cinder.utils [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 75.336(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:07:13 user cinder-volume[84696]: DEBUG cinder.utils [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_63b2c7c3-8482-49d4-bb89-c4ccf3f2db82/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:07:13 user cinder-volume[84696]: DEBUG cinder.utils [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 75.152(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:07:13 user cinder-volume[84696]: DEBUG cinder.utils [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:07:24 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=8,cinder:UPDATE=4 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:07:26 user cinder-volume[84696]: DEBUG cinder.utils [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 41.000(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:07:26 user cinder-volume[84696]: DEBUG cinder.utils [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_26014b4e-ec84-4011-a69f-ed42c3424a24/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:07:34 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=6,cinder:UPDATE=1 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:07:42 user cinder-volume[84696]: DEBUG cinder.utils [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 141.154(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:07:42 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:07:42 user cinder-volume[84696]: INFO cinder.message.api [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Creating message record for request_id = req-39f8a1bb-4b23-4f63-8942-bcd810255374 Apr 17 16:07:42 user cinder-volume[84696]: DEBUG cinder.utils [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 141.327(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:07:42 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_b4281d17-c1d8-48d3-8a0b-8fd7d2f193cc/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:07:42 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (1bd6d8b5-327c-4110-970e-7586cb828ec7) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 16:07:42 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (ceb3ed76-d61c-43e2-a2f6-5b95b4b62d52) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:07:42 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (ceb3ed76-d61c-43e2-a2f6-5b95b4b62d52) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:07:42 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Flow 'volume_create_manager' (3263a579-5dd0-479b-a8d4-fb022da8fc88) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 16:07:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 16:07:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 16:07:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 16:07:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 16:07:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 16:07:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 16:07:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 16:07:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 16:07:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 16:07:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 16:07:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 16:07:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 16:07:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 16:07:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 16:07:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 16:07:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 16:07:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 16:07:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 16:07:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 16:07:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 16:07:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 16:07:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 16:07:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 16:07:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 16:07:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:07:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 16:07:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:07:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 16:07:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:07:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 16:07:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:07:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 16:07:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:07:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 16:07:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:07:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 16:07:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:07:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 16:07:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:07:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:07:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 16:07:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 16:07:42 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 16:07:42 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (83fc3ab2-ce86-42ed-9afd-a44f55b27a7d) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 16:07:42 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 16:07:42 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:00:44Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-AttachVolumeShelveTestJSON-volume-7555594',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=b4281d17-c1d8-48d3-8a0b-8fd7d2f193cc,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='214a0f233f114d7aa22e64048a837c0b',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:05:21Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:05:21Z,use_quota=True,user_id='83b9d77de46143d8b9277a55f5ad4d72',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 16:07:42 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:00:44Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-AttachVolumeShelveTestJSON-volume-7555594',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=b4281d17-c1d8-48d3-8a0b-8fd7d2f193cc,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='214a0f233f114d7aa22e64048a837c0b',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:05:21Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:05:21Z,use_quota=True,user_id='83b9d77de46143d8b9277a55f5ad4d72',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(b4281d17-c1d8-48d3-8a0b-8fd7d2f193cc),volume_id=b4281d17-c1d8-48d3-8a0b-8fd7d2f193cc,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': 'b4281d17-c1d8-48d3-8a0b-8fd7d2f193cc', 'volume_name': 'volume-b4281d17-c1d8-48d3-8a0b-8fd7d2f193cc', 'volume_size': 1}} Apr 17 16:07:42 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': 'b4281d17-c1d8-48d3-8a0b-8fd7d2f193cc', 'volume': {'id': 'b4281d17-c1d8-48d3-8a0b-8fd7d2f193cc', '_name_id': None, 'ec2_id': None, 'user_id': '83b9d77de46143d8b9277a55f5ad4d72', 'project_id': '214a0f233f114d7aa22e64048a837c0b', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-AttachVolumeShelveTestJSON-volume-7555594', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'admin_metadata': {}, 'glance_metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T16:00:44.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-b4281d17-c1d8-48d3-8a0b-8fd7d2f193cc', 'name_id': 'b4281d17-c1d8-48d3-8a0b-8fd7d2f193cc', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-AttachVolumeShelveTestJSON-volume-7555594', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '214a0f233f114d7aa22e64048a837c0b', 'qos_specs': None, 'reservations': ['d9fcea02-28d2-4dba-a68b-5de4e249400a', 'dc207600-1b77-4709-9acd-aacbcf21051f', '891fc9c0-b5b9-44e3-a1c8-d30fcbfd5cc4', '8f4a2d2a-cb05-4ac0-b722-22369386396e'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': '83b9d77de46143d8b9277a55f5ad4d72', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-AttachVolumeShelveTestJSON-volume-7555594', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '214a0f233f114d7aa22e64048a837c0b', 'qos_specs': None, 'reservations': ['d9fcea02-28d2-4dba-a68b-5de4e249400a', 'dc207600-1b77-4709-9acd-aacbcf21051f', '891fc9c0-b5b9-44e3-a1c8-d30fcbfd5cc4', '8f4a2d2a-cb05-4ac0-b722-22369386396e'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': '83b9d77de46143d8b9277a55f5ad4d72', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': '83b9d77de46143d8b9277a55f5ad4d72', 'metadata': {}, 'qos_specs': None, 'retry': {'num_attempts': 3, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']}}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(b4281d17-c1d8-48d3-8a0b-8fd7d2f193cc),volume_id=b4281d17-c1d8-48d3-8a0b-8fd7d2f193cc,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:00:44Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-AttachVolumeShelveTestJSON-volume-7555594',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=b4281d17-c1d8-48d3-8a0b-8fd7d2f193cc,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='214a0f233f114d7aa22e64048a837c0b',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:05:21Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:05:21Z,use_quota=True,user_id='83b9d77de46143d8b9277a55f5ad4d72',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 16:07:42 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:00:44Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-AttachVolumeShelveTestJSON-volume-7555594',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=b4281d17-c1d8-48d3-8a0b-8fd7d2f193cc,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='214a0f233f114d7aa22e64048a837c0b',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:05:21Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:05:21Z,use_quota=True,user_id='83b9d77de46143d8b9277a55f5ad4d72',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:00:44Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-AttachVolumeShelveTestJSON-volume-7555594',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=b4281d17-c1d8-48d3-8a0b-8fd7d2f193cc,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='214a0f233f114d7aa22e64048a837c0b',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:05:21Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:05:21Z,use_quota=True,user_id='83b9d77de46143d8b9277a55f5ad4d72',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 16:07:42 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 16:07:42 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 16:07:42 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:07:42 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 16:07:42 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:07:42 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 16:07:42 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:07:42 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 16:07:42 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:07:42 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 16:07:42 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:07:42 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 16:07:42 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:07:42 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 16:07:42 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:07:42 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 16:07:42 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:07:42 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:07:42 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 16:07:42 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 16:07:42 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (83fc3ab2-ce86-42ed-9afd-a44f55b27a7d) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:07:42 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (83fc3ab2-ce86-42ed-9afd-a44f55b27a7d) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:07:42 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (65a99498-25ff-4bc6-85c4-eeab1a143bd1) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:07:42 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (65a99498-25ff-4bc6-85c4-eeab1a143bd1) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:07:42 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (038bab7c-7596-429a-8925-90fc4b914f90) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:07:42 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (038bab7c-7596-429a-8925-90fc4b914f90) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:07:42 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (97d55fc0-4187-4795-998e-de3eeedad509) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:07:42 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Updating volume b4281d17-c1d8-48d3-8a0b-8fd7d2f193cc with {'scheduled_at': datetime.datetime(2023, 4, 17, 16, 7, 42, 662643), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 16:07:42 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Volume b4281d17-c1d8-48d3-8a0b-8fd7d2f193cc: re-scheduling SchedulerAPI.create_volume attempt 3 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 16:07:42 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Volume b4281d17-c1d8-48d3-8a0b-8fd7d2f193cc: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 16:07:42 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] deleating volume v_b4281d17-c1d8-48d3-8a0b-8fd7d2f193cc {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 16:07:42 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] get snapshots for volume v_b4281d17-c1d8-48d3-8a0b-8fd7d2f193cc {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 16:07:42 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_b4281d17-c1d8-48d3-8a0b-8fd7d2f193cc/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:07:42 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_b4281d17-c1d8-48d3-8a0b-8fd7d2f193cc/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:07:45 user cinder-volume[84696]: DEBUG cinder.utils [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.966(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:07:45 user cinder-volume[84696]: DEBUG cinder.utils [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_b4281d17-c1d8-48d3-8a0b-8fd7d2f193cc/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:07:45 user cinder-volume[84696]: DEBUG cinder.utils [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 141.163(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:07:45 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:07:45 user cinder-volume[84696]: INFO cinder.message.api [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Creating message record for request_id = req-2f261810-f79c-48b0-b0ef-2727c4b52668 Apr 17 16:07:45 user cinder-volume[84696]: DEBUG cinder.utils [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 141.343(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:07:45 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_4d960aac-48dd-470e-831f-eaafe64c9fdf/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:07:45 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (d39f28e2-aa96-4bf8-bb1d-39acdf57545c) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 16:07:45 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (3ccbe5ca-8953-48f7-86fc-86df8aaeb3cf) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:07:45 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (3ccbe5ca-8953-48f7-86fc-86df8aaeb3cf) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:07:45 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Flow 'volume_create_manager' (d27fdfd8-f28b-4321-a429-36deaa0990f5) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 16:07:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 16:07:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 16:07:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 16:07:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 16:07:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 16:07:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 16:07:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 16:07:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 16:07:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 16:07:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 16:07:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 16:07:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 16:07:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 16:07:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 16:07:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 16:07:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 16:07:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 16:07:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 16:07:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 16:07:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 16:07:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 16:07:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 16:07:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 16:07:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 16:07:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:07:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 16:07:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:07:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 16:07:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:07:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 16:07:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:07:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 16:07:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:07:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 16:07:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:07:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 16:07:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:07:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 16:07:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:07:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:07:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 16:07:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 16:07:45 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 16:07:45 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (f9a2d6a9-94b2-44ac-a82c-5c235bdbf08a) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 16:07:45 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 16:07:45 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:03:06Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesListTestJSON-Volume-1309258736',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=4d960aac-48dd-470e-831f-eaafe64c9fdf,launched_at=None,metadata={Type='work'},migration_status=None,multiattach=False,previous_status=None,project_id='d1740791cf5e4c7087f45af9015c0a8f',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:05:24Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:05:24Z,use_quota=True,user_id='9091747614a945f6a88207d3662fea8e',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 16:07:45 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:03:06Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesListTestJSON-Volume-1309258736',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=4d960aac-48dd-470e-831f-eaafe64c9fdf,launched_at=None,metadata={Type='work'},migration_status=None,multiattach=False,previous_status=None,project_id='d1740791cf5e4c7087f45af9015c0a8f',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:05:24Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:05:24Z,use_quota=True,user_id='9091747614a945f6a88207d3662fea8e',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(4d960aac-48dd-470e-831f-eaafe64c9fdf),volume_id=4d960aac-48dd-470e-831f-eaafe64c9fdf,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': '4d960aac-48dd-470e-831f-eaafe64c9fdf', 'volume_name': 'volume-4d960aac-48dd-470e-831f-eaafe64c9fdf', 'volume_size': 1}} Apr 17 16:07:45 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': '4d960aac-48dd-470e-831f-eaafe64c9fdf', 'volume': {'id': '4d960aac-48dd-470e-831f-eaafe64c9fdf', '_name_id': None, 'ec2_id': None, 'user_id': '9091747614a945f6a88207d3662fea8e', 'project_id': 'd1740791cf5e4c7087f45af9015c0a8f', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-VolumesListTestJSON-Volume-1309258736', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {'Type': 'work'}, 'admin_metadata': {}, 'glance_metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T16:03:05.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-4d960aac-48dd-470e-831f-eaafe64c9fdf', 'name_id': '4d960aac-48dd-470e-831f-eaafe64c9fdf', 'volume_metadata': [{'key': 'Type', 'value': 'work'}], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumesListTestJSON-Volume-1309258736', 'encryption_key_id': None, 'metadata': {'Type': 'work'}, 'multiattach': False, 'project_id': 'd1740791cf5e4c7087f45af9015c0a8f', 'qos_specs': None, 'reservations': ['61b9d19b-7196-4e0b-aa10-0b704c76275c', 'a121ac12-c1d8-4525-a222-e055f88dc125', '6e5c242e-6241-4c0f-9e09-305237e9921c', '20860b71-70c8-4b28-82e8-4b507fba5ed9'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': '9091747614a945f6a88207d3662fea8e', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumesListTestJSON-Volume-1309258736', 'encryption_key_id': None, 'metadata': {'Type': 'work'}, 'multiattach': False, 'project_id': 'd1740791cf5e4c7087f45af9015c0a8f', 'qos_specs': None, 'reservations': ['61b9d19b-7196-4e0b-aa10-0b704c76275c', 'a121ac12-c1d8-4525-a222-e055f88dc125', '6e5c242e-6241-4c0f-9e09-305237e9921c', '20860b71-70c8-4b28-82e8-4b507fba5ed9'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': '9091747614a945f6a88207d3662fea8e', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': '9091747614a945f6a88207d3662fea8e', 'metadata': {'Type': 'work'}, 'qos_specs': None, 'retry': {'num_attempts': 2, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']}}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(4d960aac-48dd-470e-831f-eaafe64c9fdf),volume_id=4d960aac-48dd-470e-831f-eaafe64c9fdf,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:03:06Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesListTestJSON-Volume-1309258736',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=4d960aac-48dd-470e-831f-eaafe64c9fdf,launched_at=None,metadata={Type='work'},migration_status=None,multiattach=False,previous_status=None,project_id='d1740791cf5e4c7087f45af9015c0a8f',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:05:24Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:05:24Z,use_quota=True,user_id='9091747614a945f6a88207d3662fea8e',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 16:07:45 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:03:06Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesListTestJSON-Volume-1309258736',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=4d960aac-48dd-470e-831f-eaafe64c9fdf,launched_at=None,metadata={Type='work'},migration_status=None,multiattach=False,previous_status=None,project_id='d1740791cf5e4c7087f45af9015c0a8f',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:05:24Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:05:24Z,use_quota=True,user_id='9091747614a945f6a88207d3662fea8e',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:03:06Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesListTestJSON-Volume-1309258736',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=4d960aac-48dd-470e-831f-eaafe64c9fdf,launched_at=None,metadata={Type='work'},migration_status=None,multiattach=False,previous_status=None,project_id='d1740791cf5e4c7087f45af9015c0a8f',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:05:24Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:05:24Z,use_quota=True,user_id='9091747614a945f6a88207d3662fea8e',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 16:07:45 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 16:07:45 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 16:07:45 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:07:45 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 16:07:45 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:07:45 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 16:07:45 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:07:45 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 16:07:45 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:07:45 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 16:07:45 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:07:45 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 16:07:45 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:07:45 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 16:07:45 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:07:45 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 16:07:45 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:07:45 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:07:45 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 16:07:45 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 16:07:45 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (f9a2d6a9-94b2-44ac-a82c-5c235bdbf08a) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:07:45 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (f9a2d6a9-94b2-44ac-a82c-5c235bdbf08a) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:07:45 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (a2334ea6-612f-41b1-8291-4525d4beafea) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:07:45 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (a2334ea6-612f-41b1-8291-4525d4beafea) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:07:45 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (a137de86-8d66-433a-9ada-f8d3de4f9dfd) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:07:45 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (a137de86-8d66-433a-9ada-f8d3de4f9dfd) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:07:45 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (6f53bed4-0ed9-45a1-a45c-e1f27076bc9b) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:07:45 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Updating volume 4d960aac-48dd-470e-831f-eaafe64c9fdf with {'scheduled_at': datetime.datetime(2023, 4, 17, 16, 7, 45, 743801), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 16:07:45 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Volume 4d960aac-48dd-470e-831f-eaafe64c9fdf: re-scheduling SchedulerAPI.create_volume attempt 2 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 16:07:45 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Volume 4d960aac-48dd-470e-831f-eaafe64c9fdf: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 16:07:45 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] deleating volume v_4d960aac-48dd-470e-831f-eaafe64c9fdf {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 16:07:45 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] get snapshots for volume v_4d960aac-48dd-470e-831f-eaafe64c9fdf {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 16:07:45 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_4d960aac-48dd-470e-831f-eaafe64c9fdf/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:07:45 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_4d960aac-48dd-470e-831f-eaafe64c9fdf/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:07:45 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Volume reschedule parameters: True retry: {'num_attempts': 3, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 16:07:45 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Flow 'volume_create_manager' (4836f35e-faae-495a-9e9f-3b924cbf44b9) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 16:07:45 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (0212d309-5066-4b3c-aee3-0e008e1bcfa3) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:07:45 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (0212d309-5066-4b3c-aee3-0e008e1bcfa3) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:03:06Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesListTestJSON-Volume-1309258736',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=4d960aac-48dd-470e-831f-eaafe64c9fdf,launched_at=None,metadata={Type='work'},migration_status=None,multiattach=False,previous_status=None,project_id='d1740791cf5e4c7087f45af9015c0a8f',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:07:46Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:07:46Z,use_quota=True,user_id='9091747614a945f6a88207d3662fea8e',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:07:45 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (0c9c6108-3ef2-48ba-8fbd-83c6bc609fe4) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:07:45 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (0c9c6108-3ef2-48ba-8fbd-83c6bc609fe4) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:07:45 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (1c2cecdd-8308-4439-929b-9a18b2914675) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:07:45 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (1c2cecdd-8308-4439-929b-9a18b2914675) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': '4d960aac-48dd-470e-831f-eaafe64c9fdf', 'volume_name': 'volume-4d960aac-48dd-470e-831f-eaafe64c9fdf', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:07:45 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (b0ba8568-c132-4d6f-a6e8-f4f50b2880be) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:07:45 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (b0ba8568-c132-4d6f-a6e8-f4f50b2880be) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:07:45 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (01f5ecc6-a5e6-4864-9c1e-8dc3da5366b1) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:07:45 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Volume 4d960aac-48dd-470e-831f-eaafe64c9fdf: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-4d960aac-48dd-470e-831f-eaafe64c9fdf', 'volume_size': 1} Apr 17 16:07:45 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] creating volume v_4d960aac-48dd-470e-831f-eaafe64c9fdf. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 16:07:45 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] create volume {'name': 'v_4d960aac-48dd-470e-831f-eaafe64c9fdf', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 16:07:45 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:07:45 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:07:48 user cinder-volume[84696]: DEBUG cinder.utils [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.962(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:07:48 user cinder-volume[84696]: DEBUG cinder.utils [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_4d960aac-48dd-470e-831f-eaafe64c9fdf/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:07:48 user cinder-volume[84696]: DEBUG cinder.utils [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.770(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:07:48 user cinder-volume[84696]: DEBUG cinder.utils [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:07:48 user cinder-volume[84696]: DEBUG cinder.utils [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.042(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:07:48 user cinder-volume[84696]: DEBUG cinder.utils [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_b4281d17-c1d8-48d3-8a0b-8fd7d2f193cc/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:07:51 user cinder-volume[84696]: DEBUG cinder.utils [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.035(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:07:51 user cinder-volume[84696]: DEBUG cinder.utils [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_4d960aac-48dd-470e-831f-eaafe64c9fdf/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:07:51 user cinder-volume[84696]: DEBUG cinder.utils [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.842(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:07:51 user cinder-volume[84696]: DEBUG cinder.utils [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:07:54 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=11,cinder:UPDATE=3,cinder:INSERT=1 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:07:54 user cinder-volume[84696]: DEBUG cinder.utils [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.187(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:07:54 user cinder-volume[84696]: DEBUG cinder.utils [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_b4281d17-c1d8-48d3-8a0b-8fd7d2f193cc/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:07:56 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-008ec7fb-052b-43ae-952b-3c2d087ab270 req-de825ae0-7573-4fdc-8780-dc5a66cd78ec tempest-ServerActionsTestJSON-645547919 None] terminate connection for 1072d05f-2ebb-4398-902c-902df25464e9 {{(pid=84696) terminate_connection /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:1032}} Apr 17 16:07:56 user cinder-volume[84696]: INFO cinder.volume.manager [req-008ec7fb-052b-43ae-952b-3c2d087ab270 req-de825ae0-7573-4fdc-8780-dc5a66cd78ec tempest-ServerActionsTestJSON-645547919 None] Terminate volume connection completed successfully. Apr 17 16:07:56 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-008ec7fb-052b-43ae-952b-3c2d087ab270 req-de825ae0-7573-4fdc-8780-dc5a66cd78ec tempest-ServerActionsTestJSON-645547919 None] Deleting attachment a68f5fc4-dea3-48a4-bdb0-1d89db356c29. {{(pid=84696) attachment_delete /opt/stack/cinder/cinder/volume/manager.py:5006}} Apr 17 16:07:56 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-008ec7fb-052b-43ae-952b-3c2d087ab270 req-de825ae0-7573-4fdc-8780-dc5a66cd78ec tempest-ServerActionsTestJSON-645547919 None] remove_export for volume: 1072d05f-2ebb-4398-902c-902df25464e9. {{(pid=84696) remove_export /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:718}} Apr 17 16:07:56 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-008ec7fb-052b-43ae-952b-3c2d087ab270 req-de825ae0-7573-4fdc-8780-dc5a66cd78ec tempest-ServerActionsTestJSON-645547919 None] remove export {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:912}} Apr 17 16:07:56 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-008ec7fb-052b-43ae-952b-3c2d087ab270 req-de825ae0-7573-4fdc-8780-dc5a66cd78ec tempest-ServerActionsTestJSON-645547919 None] detach volume:Volume(_name_id=None,admin_metadata=,attach_status='attached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:27:39Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-ServerActionsTestJSON-volume-1520907984',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=1072d05f-2ebb-4398-902c-902df25464e9,launched_at=2023-04-17T15:27:40Z,metadata=,migration_status=None,multiattach=False,previous_status=None,project_id='cb5f6b5ed2fc43b69c47e1fc18646545',provider_auth='CHAP NKISaVyG nHFlRwNr8b5KS2',provider_geometry=None,provider_id=None,provider_location='172.16.0.220:3260,1 iqn.2010-10.org.openstack:1072d05f-2ebb-4398-902c-902df25464e9 0',replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:27:39Z,service_uuid='e628352e-c4cf-40eb-8389-cce40663026a',shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='in-use',terminated_at=None,updated_at=2023-04-17T15:32:47Z,use_quota=True,user_id='638a4f535e9740fc9d810639a5eb24d7',volume_attachment=,volume_type=,volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c) from target:iqn.2010-10.org.openstack:1072d05f-2ebb-4398-902c-902df25464e9. {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:913}} Apr 17 16:07:56 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [req-008ec7fb-052b-43ae-952b-3c2d087ab270 req-de825ae0-7573-4fdc-8780-dc5a66cd78ec tempest-ServerActionsTestJSON-645547919 None] detach volume v_1072d05f-2ebb-4398-902c-902df25464e9 from target iqn.2010-10.org.openstack:1072d05f-2ebb-4398-902c-902df25464e9 {{(pid=84696) detach_target_vol /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:606}} Apr 17 16:07:56 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-008ec7fb-052b-43ae-952b-3c2d087ab270 req-de825ae0-7573-4fdc-8780-dc5a66cd78ec tempest-ServerActionsTestJSON-645547919 None] Sending pool request DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:1072d05f-2ebb-4398-902c-902df25464e9/luns/v_1072d05f-2ebb-4398-902c-902df25464e9 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:07:56 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-008ec7fb-052b-43ae-952b-3c2d087ab270 req-de825ae0-7573-4fdc-8780-dc5a66cd78ec tempest-ServerActionsTestJSON-645547919 None] Sending DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:1072d05f-2ebb-4398-902c-902df25464e9/luns/v_1072d05f-2ebb-4398-902c-902df25464e9 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:07:57 user cinder-volume[84696]: DEBUG cinder.utils [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.178(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:07:57 user cinder-volume[84696]: DEBUG cinder.utils [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_4d960aac-48dd-470e-831f-eaafe64c9fdf/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:07:57 user cinder-volume[84696]: DEBUG cinder.utils [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 11.985(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:07:57 user cinder-volume[84696]: DEBUG cinder.utils [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:07:57 user cinder-volume[84696]: DEBUG cinder.utils [req-008ec7fb-052b-43ae-952b-3c2d087ab270 req-de825ae0-7573-4fdc-8780-dc5a66cd78ec tempest-ServerActionsTestJSON-645547919 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 1.479(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:07:57 user cinder-volume[84696]: DEBUG cinder.utils [req-008ec7fb-052b-43ae-952b-3c2d087ab270 req-de825ae0-7573-4fdc-8780-dc5a66cd78ec tempest-ServerActionsTestJSON-645547919 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:1072d05f-2ebb-4398-902c-902df25464e9/luns/v_1072d05f-2ebb-4398-902c-902df25464e9 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:08:01 user cinder-volume[84696]: DEBUG cinder.utils [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 75.784(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:08:01 user cinder-volume[84696]: DEBUG cinder.utils [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_26014b4e-ec84-4011-a69f-ed42c3424a24/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:08:01 user cinder-volume[84696]: DEBUG cinder.utils [req-008ec7fb-052b-43ae-952b-3c2d087ab270 req-de825ae0-7573-4fdc-8780-dc5a66cd78ec tempest-ServerActionsTestJSON-645547919 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 4.550(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:08:01 user cinder-volume[84696]: DEBUG cinder.utils [req-008ec7fb-052b-43ae-952b-3c2d087ab270 req-de825ae0-7573-4fdc-8780-dc5a66cd78ec tempest-ServerActionsTestJSON-645547919 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:1072d05f-2ebb-4398-902c-902df25464e9/luns/v_1072d05f-2ebb-4398-902c-902df25464e9 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:08:04 user cinder-volume[84696]: DEBUG cinder.utils [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.399(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:08:04 user cinder-volume[84696]: DEBUG cinder.utils [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_b4281d17-c1d8-48d3-8a0b-8fd7d2f193cc/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:08:04 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:INSERT=3,cinder:UPDATE=5,cinder:SELECT=11 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:08:07 user cinder-volume[84696]: DEBUG cinder.utils [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.394(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:08:07 user cinder-volume[84696]: DEBUG cinder.utils [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_4d960aac-48dd-470e-831f-eaafe64c9fdf/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:08:07 user cinder-volume[84696]: DEBUG cinder.utils [req-008ec7fb-052b-43ae-952b-3c2d087ab270 req-de825ae0-7573-4fdc-8780-dc5a66cd78ec tempest-ServerActionsTestJSON-645547919 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 10.694(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:08:07 user cinder-volume[84696]: DEBUG cinder.utils [req-008ec7fb-052b-43ae-952b-3c2d087ab270 req-de825ae0-7573-4fdc-8780-dc5a66cd78ec tempest-ServerActionsTestJSON-645547919 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:1072d05f-2ebb-4398-902c-902df25464e9/luns/v_1072d05f-2ebb-4398-902c-902df25464e9 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:08:07 user cinder-volume[84696]: DEBUG cinder.utils [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.202(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:08:07 user cinder-volume[84696]: DEBUG cinder.utils [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:08:14 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 142.274(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:08:14 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:08:14 user cinder-volume[84696]: ERROR oslo_service.periodic_task [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Error during VolumeManager.publish_service_capabilities: TypeError: 'NoneType' object is not subscriptable Apr 17 16:08:14 user cinder-volume[84696]: ERROR oslo_service.periodic_task Traceback (most recent call last): Apr 17 16:08:14 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/usr/local/lib/python3.10/dist-packages/oslo_service/periodic_task.py", line 216, in run_periodic_tasks Apr 17 16:08:14 user cinder-volume[84696]: ERROR oslo_service.periodic_task task(self, context) Apr 17 16:08:14 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/manager.py", line 2876, in publish_service_capabilities Apr 17 16:08:14 user cinder-volume[84696]: ERROR oslo_service.periodic_task self._report_driver_status(context) Apr 17 16:08:14 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/manager.py", line 2755, in _report_driver_status Apr 17 16:08:14 user cinder-volume[84696]: ERROR oslo_service.periodic_task volume_stats = get_stats() Apr 17 16:08:14 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/usr/local/lib/python3.10/dist-packages/oslo_utils/timeutils.py", line 319, in wrapper Apr 17 16:08:14 user cinder-volume[84696]: ERROR oslo_service.periodic_task result = func(*args, **kwargs) Apr 17 16:08:14 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/manager.py", line 2753, in get_stats Apr 17 16:08:14 user cinder-volume[84696]: ERROR oslo_service.periodic_task return self.driver.get_volume_stats(refresh=True) Apr 17 16:08:14 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/driver.py", line 666, in get_volume_stats Apr 17 16:08:14 user cinder-volume[84696]: ERROR oslo_service.periodic_task self._update_volume_stats() Apr 17 16:08:14 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 726, in _update_volume_stats Apr 17 16:08:14 user cinder-volume[84696]: ERROR oslo_service.periodic_task pool_stats = self.ra.get_pool_stats() Apr 17 16:08:14 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 897, in get_pool_stats Apr 17 16:08:14 user cinder-volume[84696]: ERROR oslo_service.periodic_task if not resp["error"] and resp["code"] == 200: Apr 17 16:08:14 user cinder-volume[84696]: ERROR oslo_service.periodic_task TypeError: 'NoneType' object is not subscriptable Apr 17 16:08:14 user cinder-volume[84696]: ERROR oslo_service.periodic_task Apr 17 16:08:14 user cinder-volume[84696]: WARNING oslo.service.loopingcall [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Function 'cinder.service.Service.periodic_tasks' run outlasted interval by 82.28 sec Apr 17 16:08:14 user cinder-volume[84696]: DEBUG cinder.utils [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 142.251(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:08:14 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_fd313648-5fc1-4e6a-97a0-92ac803b36fd/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:08:14 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (468f07b1-f1bb-48dd-b6ea-bcae9f98018c) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 16:08:14 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (6f3d172e-3e4a-41b8-a0b3-11718edd69d2) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:08:14 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (6f3d172e-3e4a-41b8-a0b3-11718edd69d2) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:08:14 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Flow 'volume_create_manager' (d3929d41-9b1a-4572-a977-36878f2f92a3) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 16:08:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-349c88aa-88c4-45fd-8d6e-2f56f9e5f4f3 tempest-TestVolumeBootPattern-1545678105 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 16:08:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 16:08:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 16:08:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 16:08:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 16:08:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 16:08:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 16:08:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 16:08:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 16:08:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 16:08:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 16:08:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 16:08:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 16:08:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 16:08:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 16:08:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 16:08:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 16:08:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 16:08:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 16:08:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 16:08:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 16:08:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 16:08:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 16:08:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 16:08:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:08:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 16:08:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1237, in execute Apr 17 16:08:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_from_image(context, Apr 17 16:08:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/utils.py", line 670, in _wrapper Apr 17 16:08:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return r(f, *args, **kwargs) Apr 17 16:08:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/tenacity/__init__.py", line 423, in __call__ Apr 17 16:08:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server do = self.iter(retry_state=retry_state) Apr 17 16:08:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/tenacity/__init__.py", line 360, in iter Apr 17 16:08:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return fut.result() Apr 17 16:08:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/lib/python3.10/concurrent/futures/_base.py", line 451, in result Apr 17 16:08:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self.__get_result() Apr 17 16:08:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result Apr 17 16:08:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self._exception Apr 17 16:08:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/tenacity/__init__.py", line 426, in __call__ Apr 17 16:08:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = fn(*args, **kwargs) Apr 17 16:08:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1125, in _create_from_image Apr 17 16:08:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_from_image_cache_or_download( Apr 17 16:08:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 944, in _create_from_image_cache_or_download Apr 17 16:08:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update, cloned = self._create_from_image_cache( Apr 17 16:08:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 843, in _create_from_image_cache Apr 17 16:08:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_from_source_volume( Apr 17 16:08:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 659, in _create_from_source_volume Apr 17 16:08:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self.driver.create_cloned_volume(volume, srcvol_ref) Apr 17 16:08:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 525, in create_cloned_volume Apr 17 16:08:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self._clone_object(vname, cvname) Apr 17 16:08:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 461, in _clone_object Apr 17 16:08:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_snapshot(oname, coname) Apr 17 16:08:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 639, in create_snapshot Apr 17 16:08:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201, 204): Apr 17 16:08:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 16:08:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 16:08:14 user cinder-volume[84696]: DEBUG oslo_service.periodic_task [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Running periodic task VolumeManager.publish_service_capabilities {{(pid=84696) run_periodic_tasks /usr/local/lib/python3.10/dist-packages/oslo_service/periodic_task.py:210}} Apr 17 16:08:14 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Updating volume stats {{(pid=84696) _update_volume_stats /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:724}} Apr 17 16:08:14 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Get pool Pool-0 fsprops {{(pid=84696) get_pool_stats /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:894}} Apr 17 16:08:14 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:08:14 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:08:17 user cinder-volume[84696]: DEBUG cinder.utils [req-008ec7fb-052b-43ae-952b-3c2d087ab270 req-de825ae0-7573-4fdc-8780-dc5a66cd78ec tempest-ServerActionsTestJSON-645547919 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 20.805(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:08:17 user cinder-volume[84696]: DEBUG cinder.utils [req-008ec7fb-052b-43ae-952b-3c2d087ab270 req-de825ae0-7573-4fdc-8780-dc5a66cd78ec tempest-ServerActionsTestJSON-645547919 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:1072d05f-2ebb-4398-902c-902df25464e9/luns/v_1072d05f-2ebb-4398-902c-902df25464e9 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:08:17 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 3.047(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:08:17 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:08:17 user cinder-volume[84696]: DEBUG cinder.utils [req-008ec7fb-052b-43ae-952b-3c2d087ab270 req-6f2f6ab3-3788-404c-8060-061f30d95d4f tempest-ServerActionsTestJSON-645547919 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 140.915(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:08:17 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-008ec7fb-052b-43ae-952b-3c2d087ab270 req-6f2f6ab3-3788-404c-8060-061f30d95d4f tempest-ServerActionsTestJSON-645547919 None] Geting None from DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:1072d05f-2ebb-4398-902c-902df25464e9/luns/v_1072d05f-2ebb-4398-902c-902df25464e9 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:08:17 user cinder-volume[84696]: WARNING cinder.volume.manager [req-008ec7fb-052b-43ae-952b-3c2d087ab270 req-6f2f6ab3-3788-404c-8060-061f30d95d4f tempest-ServerActionsTestJSON-645547919 None] Failed to detach volume on the backend, ignoring failure 'NoneType' object is not subscriptable Apr 17 16:08:17 user cinder-volume[84696]: DEBUG cinder.utils [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 141.632(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:08:17 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:08:17 user cinder-volume[84696]: INFO cinder.message.api [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Creating message record for request_id = req-251ec93b-05ae-4736-ab3c-338a5d9efb63 Apr 17 16:08:17 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 16:08:17 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (d65e06e5-e3d2-47ff-9ea3-963b4990bcc6) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 16:08:17 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 16:08:17 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:05:55Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-AttachVolumeTestJSON-volume-1580982693',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=65a642c3-b1e1-4b80-9bd2-f00151b6304a,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='2eb998c84b5b41e599af9636b929295d',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:05:56Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:05:56Z,use_quota=True,user_id='b9d4a317838b47d6a2a2e7eab7a453dd',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 16:08:17 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:05:55Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-AttachVolumeTestJSON-volume-1580982693',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=65a642c3-b1e1-4b80-9bd2-f00151b6304a,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='2eb998c84b5b41e599af9636b929295d',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:05:56Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:05:56Z,use_quota=True,user_id='b9d4a317838b47d6a2a2e7eab7a453dd',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(65a642c3-b1e1-4b80-9bd2-f00151b6304a),volume_id=65a642c3-b1e1-4b80-9bd2-f00151b6304a,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': '65a642c3-b1e1-4b80-9bd2-f00151b6304a', 'volume_name': 'volume-65a642c3-b1e1-4b80-9bd2-f00151b6304a', 'volume_size': 1}} Apr 17 16:08:17 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'retry': {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']}, 'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': '65a642c3-b1e1-4b80-9bd2-f00151b6304a', 'volume': {'id': '65a642c3-b1e1-4b80-9bd2-f00151b6304a', '_name_id': None, 'ec2_id': None, 'user_id': 'b9d4a317838b47d6a2a2e7eab7a453dd', 'project_id': '2eb998c84b5b41e599af9636b929295d', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-AttachVolumeTestJSON-volume-1580982693', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T16:05:55.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-65a642c3-b1e1-4b80-9bd2-f00151b6304a', 'name_id': '65a642c3-b1e1-4b80-9bd2-f00151b6304a', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-AttachVolumeTestJSON-volume-1580982693', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '2eb998c84b5b41e599af9636b929295d', 'qos_specs': None, 'reservations': ['1201d95e-040b-4401-9092-4910db00f9ec', '61d42360-766f-47d3-8707-d968887b405f', '54625198-5e49-42b5-a0f1-c764a5a3aaeb', '917c67ca-7481-4181-951b-961fc40b1138'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'b9d4a317838b47d6a2a2e7eab7a453dd', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-AttachVolumeTestJSON-volume-1580982693', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '2eb998c84b5b41e599af9636b929295d', 'qos_specs': None, 'reservations': ['1201d95e-040b-4401-9092-4910db00f9ec', '61d42360-766f-47d3-8707-d968887b405f', '54625198-5e49-42b5-a0f1-c764a5a3aaeb', '917c67ca-7481-4181-951b-961fc40b1138'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'b9d4a317838b47d6a2a2e7eab7a453dd', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': 'b9d4a317838b47d6a2a2e7eab7a453dd', 'metadata': {}, 'qos_specs': None}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(65a642c3-b1e1-4b80-9bd2-f00151b6304a),volume_id=65a642c3-b1e1-4b80-9bd2-f00151b6304a,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:05:55Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-AttachVolumeTestJSON-volume-1580982693',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=65a642c3-b1e1-4b80-9bd2-f00151b6304a,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='2eb998c84b5b41e599af9636b929295d',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:05:56Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:05:56Z,use_quota=True,user_id='b9d4a317838b47d6a2a2e7eab7a453dd',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 16:08:17 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:05:55Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-AttachVolumeTestJSON-volume-1580982693',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=65a642c3-b1e1-4b80-9bd2-f00151b6304a,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='2eb998c84b5b41e599af9636b929295d',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:05:56Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:05:56Z,use_quota=True,user_id='b9d4a317838b47d6a2a2e7eab7a453dd',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:05:55Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-AttachVolumeTestJSON-volume-1580982693',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=65a642c3-b1e1-4b80-9bd2-f00151b6304a,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='2eb998c84b5b41e599af9636b929295d',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:05:56Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:05:56Z,use_quota=True,user_id='b9d4a317838b47d6a2a2e7eab7a453dd',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 16:08:17 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 16:08:17 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 16:08:17 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:08:17 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 16:08:17 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:08:17 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 16:08:17 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:08:17 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 16:08:17 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:08:17 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 16:08:17 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:08:17 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 16:08:17 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:08:17 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 16:08:17 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:08:17 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 16:08:17 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:08:17 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:08:17 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 16:08:17 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 16:08:17 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (d65e06e5-e3d2-47ff-9ea3-963b4990bcc6) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:08:17 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (d65e06e5-e3d2-47ff-9ea3-963b4990bcc6) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:08:17 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (9e245c48-21e1-45a6-a58e-ba1a497620cb) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:08:17 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (9e245c48-21e1-45a6-a58e-ba1a497620cb) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:08:17 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (ed007a2f-e22a-4ae0-a101-6944585cfb78) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:08:17 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (ed007a2f-e22a-4ae0-a101-6944585cfb78) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:08:17 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (69725152-efb8-4ceb-b719-32a1332715d2) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:08:17 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Updating volume 65a642c3-b1e1-4b80-9bd2-f00151b6304a with {'scheduled_at': datetime.datetime(2023, 4, 17, 16, 8, 17, 338879), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 16:08:17 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Volume 65a642c3-b1e1-4b80-9bd2-f00151b6304a: re-scheduling SchedulerAPI.create_volume attempt 1 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 16:08:17 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Volume 65a642c3-b1e1-4b80-9bd2-f00151b6304a: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 16:08:17 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] deleating volume v_65a642c3-b1e1-4b80-9bd2-f00151b6304a {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 16:08:17 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] get snapshots for volume v_65a642c3-b1e1-4b80-9bd2-f00151b6304a {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 16:08:17 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_65a642c3-b1e1-4b80-9bd2-f00151b6304a/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:08:17 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_65a642c3-b1e1-4b80-9bd2-f00151b6304a/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:08:17 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Volume reschedule parameters: True retry: {'num_attempts': 2, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 16:08:17 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Flow 'volume_create_manager' (c7250c61-21e3-4f2f-87b2-a4e7a77f94cf) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 16:08:17 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (8257a320-a94c-48aa-88f3-d98a64bd5017) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:08:17 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (8257a320-a94c-48aa-88f3-d98a64bd5017) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:05:55Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-AttachVolumeTestJSON-volume-1580982693',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=65a642c3-b1e1-4b80-9bd2-f00151b6304a,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='2eb998c84b5b41e599af9636b929295d',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:08:17Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:08:17Z,use_quota=True,user_id='b9d4a317838b47d6a2a2e7eab7a453dd',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:08:17 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (8e18a790-1ba9-497b-a9bd-8edd30a199fa) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:08:17 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (8e18a790-1ba9-497b-a9bd-8edd30a199fa) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:08:17 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (9cae5485-cfd8-4eef-893b-ad692eee180d) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:08:17 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (9cae5485-cfd8-4eef-893b-ad692eee180d) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': '65a642c3-b1e1-4b80-9bd2-f00151b6304a', 'volume_name': 'volume-65a642c3-b1e1-4b80-9bd2-f00151b6304a', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:08:17 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (02ee8a81-0d54-47a3-857a-8bfc890ca5c2) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:08:17 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (02ee8a81-0d54-47a3-857a-8bfc890ca5c2) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:08:17 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (d77e7fda-51c7-4be1-93d7-91ea34f7f935) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:08:17 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Volume 65a642c3-b1e1-4b80-9bd2-f00151b6304a: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-65a642c3-b1e1-4b80-9bd2-f00151b6304a', 'volume_size': 1} Apr 17 16:08:17 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] creating volume v_65a642c3-b1e1-4b80-9bd2-f00151b6304a. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 16:08:17 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] create volume {'name': 'v_65a642c3-b1e1-4b80-9bd2-f00151b6304a', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 16:08:17 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:08:17 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:08:20 user cinder-volume[84696]: DEBUG cinder.utils [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 142.183(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:08:20 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_63b2c7c3-8482-49d4-bb89-c4ccf3f2db82/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:08:20 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (6903556b-4102-4597-bd53-3965cfeb9219) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 16:08:20 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (e83b71b6-5e69-4568-bb40-d5a1f1bd4d64) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:08:20 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (e83b71b6-5e69-4568-bb40-d5a1f1bd4d64) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:08:20 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Flow 'volume_create_manager' (5e9ae34d-c22c-4075-81de-33ec1630d1ff) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 16:08:20 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 16:08:20 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 16:08:20 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 16:08:20 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 16:08:20 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 16:08:20 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 16:08:20 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 16:08:20 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 16:08:20 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 16:08:20 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 16:08:20 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 16:08:20 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 16:08:20 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 16:08:20 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 16:08:20 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 16:08:20 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 16:08:20 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 16:08:20 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 16:08:20 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 16:08:20 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 16:08:20 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 16:08:20 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 16:08:20 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 16:08:20 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 16:08:20 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:08:20 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 16:08:20 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:08:20 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 16:08:20 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:08:20 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 16:08:20 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:08:20 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 16:08:20 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:08:20 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 16:08:20 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:08:20 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 16:08:20 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:08:20 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 16:08:20 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:08:20 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:08:20 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 16:08:20 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 16:08:20 user cinder-volume[84696]: DEBUG cinder.utils [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.826(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:08:20 user cinder-volume[84696]: DEBUG cinder.utils [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:08:20 user cinder-volume[84696]: DEBUG cinder.utils [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 142.016(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:08:20 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:08:20 user cinder-volume[84696]: INFO cinder.message.api [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Creating message record for request_id = req-a293a946-8aa4-47bc-8290-fe9922d7ae21 Apr 17 16:08:20 user cinder-volume[84696]: DEBUG cinder.utils [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 3.004(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:08:20 user cinder-volume[84696]: DEBUG cinder.utils [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_65a642c3-b1e1-4b80-9bd2-f00151b6304a/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:08:20 user cinder-volume[84696]: DEBUG cinder.utils [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 37.678(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:08:20 user cinder-volume[84696]: DEBUG cinder.utils [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_b4281d17-c1d8-48d3-8a0b-8fd7d2f193cc/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:08:20 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.142(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:08:20 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:08:20 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 16:08:20 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (39af473a-c09b-4053-a9b3-4e1083fb4a01) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 16:08:20 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 16:08:20 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:01:19Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-AttachVolumeNegativeTest-volume-1267367334',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=63b2c7c3-8482-49d4-bb89-c4ccf3f2db82,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='8586cc10142d4743874f43cdb0f37aba',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:05:58Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:05:58Z,use_quota=True,user_id='9fdc7c80b4654692a52c8a4c4149324f',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 16:08:20 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:01:19Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-AttachVolumeNegativeTest-volume-1267367334',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=63b2c7c3-8482-49d4-bb89-c4ccf3f2db82,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='8586cc10142d4743874f43cdb0f37aba',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:05:58Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:05:58Z,use_quota=True,user_id='9fdc7c80b4654692a52c8a4c4149324f',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(63b2c7c3-8482-49d4-bb89-c4ccf3f2db82),volume_id=63b2c7c3-8482-49d4-bb89-c4ccf3f2db82,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': '63b2c7c3-8482-49d4-bb89-c4ccf3f2db82', 'volume_name': 'volume-63b2c7c3-8482-49d4-bb89-c4ccf3f2db82', 'volume_size': 1}} Apr 17 16:08:20 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': '63b2c7c3-8482-49d4-bb89-c4ccf3f2db82', 'volume': {'id': '63b2c7c3-8482-49d4-bb89-c4ccf3f2db82', '_name_id': None, 'ec2_id': None, 'user_id': '9fdc7c80b4654692a52c8a4c4149324f', 'project_id': '8586cc10142d4743874f43cdb0f37aba', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-AttachVolumeNegativeTest-volume-1267367334', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'admin_metadata': {}, 'glance_metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T16:01:19.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-63b2c7c3-8482-49d4-bb89-c4ccf3f2db82', 'name_id': '63b2c7c3-8482-49d4-bb89-c4ccf3f2db82', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-AttachVolumeNegativeTest-volume-1267367334', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '8586cc10142d4743874f43cdb0f37aba', 'qos_specs': None, 'reservations': ['60706a3b-0211-488a-91cd-03061d7521a9', '3ccbaa06-3cae-4fa4-b8ac-ab913175fa13', '74bd9b1b-1b22-4085-aa81-b422f52940be', '3a23250a-64d8-4fa2-826b-865608d68d21'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': '9fdc7c80b4654692a52c8a4c4149324f', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-AttachVolumeNegativeTest-volume-1267367334', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '8586cc10142d4743874f43cdb0f37aba', 'qos_specs': None, 'reservations': ['60706a3b-0211-488a-91cd-03061d7521a9', '3ccbaa06-3cae-4fa4-b8ac-ab913175fa13', '74bd9b1b-1b22-4085-aa81-b422f52940be', '3a23250a-64d8-4fa2-826b-865608d68d21'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': '9fdc7c80b4654692a52c8a4c4149324f', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': '9fdc7c80b4654692a52c8a4c4149324f', 'metadata': {}, 'qos_specs': None, 'retry': {'num_attempts': 3, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']}}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(63b2c7c3-8482-49d4-bb89-c4ccf3f2db82),volume_id=63b2c7c3-8482-49d4-bb89-c4ccf3f2db82,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:01:19Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-AttachVolumeNegativeTest-volume-1267367334',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=63b2c7c3-8482-49d4-bb89-c4ccf3f2db82,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='8586cc10142d4743874f43cdb0f37aba',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:05:58Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:05:58Z,use_quota=True,user_id='9fdc7c80b4654692a52c8a4c4149324f',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 16:08:20 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:01:19Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-AttachVolumeNegativeTest-volume-1267367334',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=63b2c7c3-8482-49d4-bb89-c4ccf3f2db82,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='8586cc10142d4743874f43cdb0f37aba',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:05:58Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:05:58Z,use_quota=True,user_id='9fdc7c80b4654692a52c8a4c4149324f',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:01:19Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-AttachVolumeNegativeTest-volume-1267367334',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=63b2c7c3-8482-49d4-bb89-c4ccf3f2db82,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='8586cc10142d4743874f43cdb0f37aba',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:05:58Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:05:58Z,use_quota=True,user_id='9fdc7c80b4654692a52c8a4c4149324f',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 16:08:20 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 16:08:20 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 16:08:20 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:08:20 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 16:08:20 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:08:20 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 16:08:20 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:08:20 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 16:08:20 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:08:20 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 16:08:20 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:08:20 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 16:08:20 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:08:20 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 16:08:20 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:08:20 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 16:08:20 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:08:20 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:08:20 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 16:08:20 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 16:08:20 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (39af473a-c09b-4053-a9b3-4e1083fb4a01) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:08:20 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (39af473a-c09b-4053-a9b3-4e1083fb4a01) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:08:20 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (2f6ededf-ecea-4107-85b8-57d0f74076ad) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:08:20 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (2f6ededf-ecea-4107-85b8-57d0f74076ad) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:08:20 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (4f04b455-a4c0-4ec3-b802-67aef6d02b08) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:08:20 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (4f04b455-a4c0-4ec3-b802-67aef6d02b08) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:08:20 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (ae393033-9a6b-4c77-a97c-754828be17e4) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:08:20 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Updating volume 63b2c7c3-8482-49d4-bb89-c4ccf3f2db82 with {'scheduled_at': datetime.datetime(2023, 4, 17, 16, 8, 20, 448742), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 16:08:20 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Volume 63b2c7c3-8482-49d4-bb89-c4ccf3f2db82: re-scheduling SchedulerAPI.create_volume attempt 3 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 16:08:20 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Volume 63b2c7c3-8482-49d4-bb89-c4ccf3f2db82: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 16:08:20 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] deleating volume v_63b2c7c3-8482-49d4-bb89-c4ccf3f2db82 {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 16:08:20 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] get snapshots for volume v_63b2c7c3-8482-49d4-bb89-c4ccf3f2db82 {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 16:08:20 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_63b2c7c3-8482-49d4-bb89-c4ccf3f2db82/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:08:20 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_63b2c7c3-8482-49d4-bb89-c4ccf3f2db82/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:08:23 user cinder-volume[84696]: DEBUG cinder.utils [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.947(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:08:23 user cinder-volume[84696]: DEBUG cinder.utils [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_63b2c7c3-8482-49d4-bb89-c4ccf3f2db82/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:08:23 user cinder-volume[84696]: DEBUG cinder.utils [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.056(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:08:23 user cinder-volume[84696]: DEBUG cinder.utils [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_65a642c3-b1e1-4b80-9bd2-f00151b6304a/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:08:23 user cinder-volume[84696]: DEBUG cinder.utils [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.883(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:08:23 user cinder-volume[84696]: DEBUG cinder.utils [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:08:23 user cinder-volume[84696]: DEBUG cinder.utils [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 37.459(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:08:23 user cinder-volume[84696]: DEBUG cinder.utils [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:08:23 user cinder-volume[84696]: DEBUG cinder.utils [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 37.655(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:08:23 user cinder-volume[84696]: DEBUG cinder.utils [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_4d960aac-48dd-470e-831f-eaafe64c9fdf/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:08:24 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=18,cinder:UPDATE=3,cinder:INSERT=2 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:08:26 user cinder-volume[84696]: DEBUG cinder.utils [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.020(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:08:26 user cinder-volume[84696]: DEBUG cinder.utils [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_63b2c7c3-8482-49d4-bb89-c4ccf3f2db82/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:08:26 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.264(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:08:26 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:08:29 user cinder-volume[84696]: DEBUG cinder.utils [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.198(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:08:29 user cinder-volume[84696]: DEBUG cinder.utils [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_65a642c3-b1e1-4b80-9bd2-f00151b6304a/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:08:29 user cinder-volume[84696]: DEBUG cinder.utils [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.026(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:08:29 user cinder-volume[84696]: DEBUG cinder.utils [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:08:32 user cinder-volume[84696]: DEBUG cinder.utils [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.163(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:08:32 user cinder-volume[84696]: DEBUG cinder.utils [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_63b2c7c3-8482-49d4-bb89-c4ccf3f2db82/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:08:35 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.477(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:08:35 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:08:35 user cinder-volume[84696]: DEBUG cinder.utils [req-008ec7fb-052b-43ae-952b-3c2d087ab270 req-de825ae0-7573-4fdc-8780-dc5a66cd78ec tempest-ServerActionsTestJSON-645547919 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.238(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:08:35 user cinder-volume[84696]: DEBUG cinder.utils [req-008ec7fb-052b-43ae-952b-3c2d087ab270 req-de825ae0-7573-4fdc-8780-dc5a66cd78ec tempest-ServerActionsTestJSON-645547919 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:1072d05f-2ebb-4398-902c-902df25464e9/luns/v_1072d05f-2ebb-4398-902c-902df25464e9 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:08:38 user cinder-volume[84696]: DEBUG cinder.utils [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.414(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:08:38 user cinder-volume[84696]: DEBUG cinder.utils [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_65a642c3-b1e1-4b80-9bd2-f00151b6304a/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:08:38 user cinder-volume[84696]: DEBUG cinder.utils [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.242(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:08:38 user cinder-volume[84696]: DEBUG cinder.utils [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:08:41 user cinder-volume[84696]: DEBUG cinder.utils [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.379(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:08:41 user cinder-volume[84696]: DEBUG cinder.utils [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_63b2c7c3-8482-49d4-bb89-c4ccf3f2db82/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:08:44 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=8,cinder:UPDATE=6,cinder:INSERT=2 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:08:49 user cinder-volume[84696]: INFO cinder.volume.manager [None req-989d7e12-21bd-4d79-b382-2b050fa61450 tempest-GroupsV320Test-1696456310 None] Group tempest-GroupsV320Test-Group-1010458796: creating Apr 17 16:08:49 user cinder-volume[84696]: INFO cinder.volume.manager [None req-989d7e12-21bd-4d79-b382-2b050fa61450 tempest-GroupsV320Test-1696456310 None] Group tempest-GroupsV320Test-Group-1010458796: created successfully Apr 17 16:08:49 user cinder-volume[84696]: INFO cinder.volume.manager [None req-989d7e12-21bd-4d79-b382-2b050fa61450 tempest-GroupsV320Test-1696456310 None] Create group completed successfully. Apr 17 16:08:51 user cinder-volume[84696]: DEBUG cinder.quota [None req-e0496134-2447-4744-8a98-847f63a20a7b tempest-GroupsV320Test-1696456310 None] Created reservations ['b4dd68e7-69d2-4826-b934-5829a5787cca'] {{(pid=84696) reserve /opt/stack/cinder/cinder/quota.py:755}} Apr 17 16:08:51 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-e0496134-2447-4744-8a98-847f63a20a7b tempest-GroupsV320Test-1696456310 None] Updating volume stats {{(pid=84696) _update_volume_stats /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:724}} Apr 17 16:08:51 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-e0496134-2447-4744-8a98-847f63a20a7b tempest-GroupsV320Test-1696456310 None] Get pool Pool-0 fsprops {{(pid=84696) get_pool_stats /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:894}} Apr 17 16:08:51 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-e0496134-2447-4744-8a98-847f63a20a7b tempest-GroupsV320Test-1696456310 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:08:51 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-e0496134-2447-4744-8a98-847f63a20a7b tempest-GroupsV320Test-1696456310 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:08:54 user cinder-volume[84696]: DEBUG cinder.utils [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 71.447(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:08:54 user cinder-volume[84696]: DEBUG cinder.utils [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_b4281d17-c1d8-48d3-8a0b-8fd7d2f193cc/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:08:54 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.911(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:08:54 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:08:54 user cinder-volume[84696]: DEBUG cinder.utils [None req-e0496134-2447-4744-8a98-847f63a20a7b tempest-GroupsV320Test-1696456310 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.954(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:08:54 user cinder-volume[84696]: DEBUG cinder.utils [None req-e0496134-2447-4744-8a98-847f63a20a7b tempest-GroupsV320Test-1696456310 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:08:54 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=21,cinder:UPDATE=5,cinder:INSERT=1 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:08:57 user cinder-volume[84696]: DEBUG cinder.utils [None req-e0496134-2447-4744-8a98-847f63a20a7b tempest-GroupsV320Test-1696456310 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.028(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:08:57 user cinder-volume[84696]: DEBUG cinder.utils [None req-e0496134-2447-4744-8a98-847f63a20a7b tempest-GroupsV320Test-1696456310 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:08:57 user cinder-volume[84696]: DEBUG cinder.utils [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.678(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:08:57 user cinder-volume[84696]: DEBUG cinder.utils [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:08:57 user cinder-volume[84696]: DEBUG cinder.utils [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.852(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:08:57 user cinder-volume[84696]: DEBUG cinder.utils [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_65a642c3-b1e1-4b80-9bd2-f00151b6304a/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:08:57 user cinder-volume[84696]: DEBUG cinder.utils [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 71.255(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:08:57 user cinder-volume[84696]: DEBUG cinder.utils [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:08:57 user cinder-volume[84696]: DEBUG cinder.utils [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 71.450(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:08:57 user cinder-volume[84696]: DEBUG cinder.utils [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_4d960aac-48dd-470e-831f-eaafe64c9fdf/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:08:57 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Volume reschedule parameters: True retry: {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 16:08:57 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Flow 'volume_create_manager' (2bedb8fd-4b3a-49a4-aa70-0a68a6e5f424) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 16:08:57 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (78aba441-05df-4a59-bfb0-daad5876261a) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:08:57 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (78aba441-05df-4a59-bfb0-daad5876261a) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:08:58Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesTransfersV355Test-Volume-1916167074',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=49effcda-e787-4983-92b2-f8261052732d,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='9796cc8ccc334a5b86ff098b10a0c413',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:08:58Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:08:58Z,use_quota=True,user_id='de05a598447a4e9fb3ff488ec2d84071',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:08:57 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (85e8bd62-0b6e-4718-b1b6-4e9d77ee4eeb) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:08:57 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (85e8bd62-0b6e-4718-b1b6-4e9d77ee4eeb) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:08:57 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (331f78e6-704c-4c85-9cf4-7b1b63e706b8) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:08:57 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (331f78e6-704c-4c85-9cf4-7b1b63e706b8) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': '49effcda-e787-4983-92b2-f8261052732d', 'volume_name': 'volume-49effcda-e787-4983-92b2-f8261052732d', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:08:57 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (7c6d64c8-6145-4ee2-ba3b-04a2ed0734d5) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:08:57 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (7c6d64c8-6145-4ee2-ba3b-04a2ed0734d5) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:08:57 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (346283e3-48d0-4d43-87e5-6196db961c87) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:08:57 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Volume 49effcda-e787-4983-92b2-f8261052732d: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-49effcda-e787-4983-92b2-f8261052732d', 'volume_size': 1} Apr 17 16:08:57 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] creating volume v_49effcda-e787-4983-92b2-f8261052732d. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 16:08:57 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] create volume {'name': 'v_49effcda-e787-4983-92b2-f8261052732d', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 16:08:57 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:08:57 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:09:00 user cinder-volume[84696]: DEBUG cinder.utils [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.811(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:09:00 user cinder-volume[84696]: DEBUG cinder.utils [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_63b2c7c3-8482-49d4-bb89-c4ccf3f2db82/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:09:00 user cinder-volume[84696]: DEBUG cinder.utils [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.394(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:09:00 user cinder-volume[84696]: DEBUG cinder.utils [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:09:03 user cinder-volume[84696]: DEBUG cinder.utils [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.465(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:09:03 user cinder-volume[84696]: DEBUG cinder.utils [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:09:03 user cinder-volume[84696]: DEBUG cinder.utils [None req-e0496134-2447-4744-8a98-847f63a20a7b tempest-GroupsV320Test-1696456310 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.170(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:09:03 user cinder-volume[84696]: DEBUG cinder.utils [None req-e0496134-2447-4744-8a98-847f63a20a7b tempest-GroupsV320Test-1696456310 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:09:06 user cinder-volume[84696]: DEBUG cinder.utils [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 141.196(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:09:06 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_26014b4e-ec84-4011-a69f-ed42c3424a24/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:09:06 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (b17bc9b9-8b5d-4670-9061-37ba32d86af2) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 16:09:06 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (1d3529f0-9a7a-4d7c-9d56-b03414fdc4d1) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:09:06 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (1d3529f0-9a7a-4d7c-9d56-b03414fdc4d1) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:09:06 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Flow 'volume_create_manager' (0dcce114-7d1d-4690-ad6b-a588f3750e32) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 16:09:06 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-8969563c-70db-45ae-8d53-74d80da7efc1 tempest-VolumesListAdminTestJSON-1808324659 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 16:09:06 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 16:09:06 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 16:09:06 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 16:09:06 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 16:09:06 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 16:09:06 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 16:09:06 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 16:09:06 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 16:09:06 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 16:09:06 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 16:09:06 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 16:09:06 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 16:09:06 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 16:09:06 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 16:09:06 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 16:09:06 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 16:09:06 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 16:09:06 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 16:09:06 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 16:09:06 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 16:09:06 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 16:09:06 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 16:09:06 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 16:09:06 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:09:06 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 16:09:06 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:09:06 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 16:09:06 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:09:06 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 16:09:06 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:09:06 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 16:09:06 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:09:06 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 16:09:06 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:09:06 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 16:09:06 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:09:06 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 16:09:06 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:09:06 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:09:06 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 16:09:06 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 16:09:09 user cinder-volume[84696]: DEBUG cinder.utils [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 11.609(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:09:09 user cinder-volume[84696]: DEBUG cinder.utils [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:09:09 user cinder-volume[84696]: DEBUG cinder.utils [req-008ec7fb-052b-43ae-952b-3c2d087ab270 req-de825ae0-7573-4fdc-8780-dc5a66cd78ec tempest-ServerActionsTestJSON-645547919 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 73.030(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:09:09 user cinder-volume[84696]: DEBUG cinder.utils [req-008ec7fb-052b-43ae-952b-3c2d087ab270 req-de825ae0-7573-4fdc-8780-dc5a66cd78ec tempest-ServerActionsTestJSON-645547919 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:1072d05f-2ebb-4398-902c-902df25464e9/luns/v_1072d05f-2ebb-4398-902c-902df25464e9 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:09:12 user cinder-volume[84696]: DEBUG cinder.utils [None req-e0496134-2447-4744-8a98-847f63a20a7b tempest-GroupsV320Test-1696456310 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.384(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:09:12 user cinder-volume[84696]: DEBUG cinder.utils [None req-e0496134-2447-4744-8a98-847f63a20a7b tempest-GroupsV320Test-1696456310 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:09:14 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=4,cinder:UPDATE=1 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:09:14 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=19,cinder:UPDATE=5 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:09:18 user cinder-volume[84696]: DEBUG cinder.utils [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 20.829(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:09:18 user cinder-volume[84696]: DEBUG cinder.utils [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:09:28 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 74.023(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:09:28 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:09:31 user cinder-volume[84696]: DEBUG cinder.utils [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 73.786(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:09:31 user cinder-volume[84696]: DEBUG cinder.utils [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:09:31 user cinder-volume[84696]: DEBUG cinder.utils [None req-e0496134-2447-4744-8a98-847f63a20a7b tempest-GroupsV320Test-1696456310 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 40.141(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:09:31 user cinder-volume[84696]: DEBUG cinder.utils [None req-e0496134-2447-4744-8a98-847f63a20a7b tempest-GroupsV320Test-1696456310 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:09:31 user cinder-volume[84696]: DEBUG cinder.utils [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 73.963(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:09:31 user cinder-volume[84696]: DEBUG cinder.utils [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_65a642c3-b1e1-4b80-9bd2-f00151b6304a/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:09:34 user cinder-volume[84696]: DEBUG cinder.utils [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 73.923(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:09:34 user cinder-volume[84696]: DEBUG cinder.utils [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_63b2c7c3-8482-49d4-bb89-c4ccf3f2db82/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:09:34 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=1 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:09:34 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=4,cinder:UPDATE=2 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:09:37 user cinder-volume[84696]: DEBUG cinder.utils [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.577(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:09:37 user cinder-volume[84696]: DEBUG cinder.utils [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:09:54 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=5,cinder:UPDATE=1 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:09:56 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-008ec7fb-052b-43ae-952b-3c2d087ab270 req-15d5486a-4efe-4884-8b8c-05a078b8fa5f tempest-ServerActionsTestJSON-645547919 None] terminate connection for 1072d05f-2ebb-4398-902c-902df25464e9 {{(pid=84696) terminate_connection /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:1032}} Apr 17 16:09:56 user cinder-volume[84696]: INFO cinder.volume.manager [req-008ec7fb-052b-43ae-952b-3c2d087ab270 req-15d5486a-4efe-4884-8b8c-05a078b8fa5f tempest-ServerActionsTestJSON-645547919 None] Terminate volume connection completed successfully. Apr 17 16:09:56 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-008ec7fb-052b-43ae-952b-3c2d087ab270 req-15d5486a-4efe-4884-8b8c-05a078b8fa5f tempest-ServerActionsTestJSON-645547919 None] Deleting attachment a68f5fc4-dea3-48a4-bdb0-1d89db356c29. {{(pid=84696) attachment_delete /opt/stack/cinder/cinder/volume/manager.py:5006}} Apr 17 16:09:56 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-008ec7fb-052b-43ae-952b-3c2d087ab270 req-15d5486a-4efe-4884-8b8c-05a078b8fa5f tempest-ServerActionsTestJSON-645547919 None] remove_export for volume: 1072d05f-2ebb-4398-902c-902df25464e9. {{(pid=84696) remove_export /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:718}} Apr 17 16:09:56 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-008ec7fb-052b-43ae-952b-3c2d087ab270 req-15d5486a-4efe-4884-8b8c-05a078b8fa5f tempest-ServerActionsTestJSON-645547919 None] remove export {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:912}} Apr 17 16:09:56 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-008ec7fb-052b-43ae-952b-3c2d087ab270 req-15d5486a-4efe-4884-8b8c-05a078b8fa5f tempest-ServerActionsTestJSON-645547919 None] detach volume:Volume(_name_id=None,admin_metadata=,attach_status='attached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:27:39Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-ServerActionsTestJSON-volume-1520907984',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=1072d05f-2ebb-4398-902c-902df25464e9,launched_at=2023-04-17T15:27:40Z,metadata=,migration_status=None,multiattach=False,previous_status=None,project_id='cb5f6b5ed2fc43b69c47e1fc18646545',provider_auth='CHAP NKISaVyG nHFlRwNr8b5KS2',provider_geometry=None,provider_id=None,provider_location='172.16.0.220:3260,1 iqn.2010-10.org.openstack:1072d05f-2ebb-4398-902c-902df25464e9 0',replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:27:39Z,service_uuid='e628352e-c4cf-40eb-8389-cce40663026a',shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='in-use',terminated_at=None,updated_at=2023-04-17T15:32:47Z,use_quota=True,user_id='638a4f535e9740fc9d810639a5eb24d7',volume_attachment=,volume_type=,volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c) from target:iqn.2010-10.org.openstack:1072d05f-2ebb-4398-902c-902df25464e9. {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:913}} Apr 17 16:09:56 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [req-008ec7fb-052b-43ae-952b-3c2d087ab270 req-15d5486a-4efe-4884-8b8c-05a078b8fa5f tempest-ServerActionsTestJSON-645547919 None] detach volume v_1072d05f-2ebb-4398-902c-902df25464e9 from target iqn.2010-10.org.openstack:1072d05f-2ebb-4398-902c-902df25464e9 {{(pid=84696) detach_target_vol /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:606}} Apr 17 16:09:56 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-008ec7fb-052b-43ae-952b-3c2d087ab270 req-15d5486a-4efe-4884-8b8c-05a078b8fa5f tempest-ServerActionsTestJSON-645547919 None] Sending pool request DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:1072d05f-2ebb-4398-902c-902df25464e9/luns/v_1072d05f-2ebb-4398-902c-902df25464e9 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:09:56 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-008ec7fb-052b-43ae-952b-3c2d087ab270 req-15d5486a-4efe-4884-8b8c-05a078b8fa5f tempest-ServerActionsTestJSON-645547919 None] Sending DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:1072d05f-2ebb-4398-902c-902df25464e9/luns/v_1072d05f-2ebb-4398-902c-902df25464e9 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:09:57 user cinder-volume[84696]: DEBUG cinder.utils [req-008ec7fb-052b-43ae-952b-3c2d087ab270 req-15d5486a-4efe-4884-8b8c-05a078b8fa5f tempest-ServerActionsTestJSON-645547919 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 0.818(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:09:57 user cinder-volume[84696]: DEBUG cinder.utils [req-008ec7fb-052b-43ae-952b-3c2d087ab270 req-15d5486a-4efe-4884-8b8c-05a078b8fa5f tempest-ServerActionsTestJSON-645547919 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:1072d05f-2ebb-4398-902c-902df25464e9/luns/v_1072d05f-2ebb-4398-902c-902df25464e9 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:10:01 user cinder-volume[84696]: DEBUG cinder.utils [req-008ec7fb-052b-43ae-952b-3c2d087ab270 req-15d5486a-4efe-4884-8b8c-05a078b8fa5f tempest-ServerActionsTestJSON-645547919 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 4.694(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:10:01 user cinder-volume[84696]: DEBUG cinder.utils [req-008ec7fb-052b-43ae-952b-3c2d087ab270 req-15d5486a-4efe-4884-8b8c-05a078b8fa5f tempest-ServerActionsTestJSON-645547919 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:1072d05f-2ebb-4398-902c-902df25464e9/luns/v_1072d05f-2ebb-4398-902c-902df25464e9 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:10:01 user cinder-volume[84696]: DEBUG cinder.utils [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 138.525(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:10:01 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_b4281d17-c1d8-48d3-8a0b-8fd7d2f193cc/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:10:01 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (97d55fc0-4187-4795-998e-de3eeedad509) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 16:10:01 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (0d3e40db-8638-46d2-91f8-796253e33629) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:10:01 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (0d3e40db-8638-46d2-91f8-796253e33629) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:10:01 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Flow 'volume_create_manager' (1cf046c1-1ada-4c3e-ac10-d4d428c2495a) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 16:10:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-39f8a1bb-4b23-4f63-8942-bcd810255374 tempest-AttachVolumeShelveTestJSON-1745617049 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 16:10:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 16:10:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 16:10:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 16:10:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 16:10:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 16:10:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 16:10:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 16:10:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 16:10:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 16:10:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 16:10:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 16:10:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 16:10:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 16:10:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 16:10:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 16:10:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 16:10:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 16:10:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 16:10:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 16:10:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 16:10:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 16:10:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 16:10:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 16:10:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:10:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 16:10:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:10:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 16:10:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:10:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 16:10:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:10:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 16:10:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:10:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 16:10:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:10:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 16:10:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:10:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 16:10:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:10:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:10:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 16:10:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 16:10:04 user cinder-volume[84696]: DEBUG cinder.utils [None req-e0496134-2447-4744-8a98-847f63a20a7b tempest-GroupsV320Test-1696456310 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 73.096(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:10:04 user cinder-volume[84696]: DEBUG cinder.utils [None req-e0496134-2447-4744-8a98-847f63a20a7b tempest-GroupsV320Test-1696456310 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:10:04 user cinder-volume[84696]: DEBUG cinder.utils [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 138.321(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:10:04 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:10:04 user cinder-volume[84696]: INFO cinder.message.api [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Creating message record for request_id = req-2f261810-f79c-48b0-b0ef-2727c4b52668 Apr 17 16:10:04 user cinder-volume[84696]: DEBUG cinder.utils [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 138.522(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:10:04 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_4d960aac-48dd-470e-831f-eaafe64c9fdf/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:10:04 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (6f53bed4-0ed9-45a1-a45c-e1f27076bc9b) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 16:10:04 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (9c939d09-6904-4b5a-94e0-ff47e99f48a3) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:10:04 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (9c939d09-6904-4b5a-94e0-ff47e99f48a3) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:10:04 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Flow 'volume_create_manager' (063aef0d-532b-4252-971f-2ab93c167153) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 16:10:04 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 16:10:04 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 16:10:04 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 16:10:04 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 16:10:04 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 16:10:04 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 16:10:04 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 16:10:04 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 16:10:04 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 16:10:04 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 16:10:04 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 16:10:04 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 16:10:04 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 16:10:04 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 16:10:04 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 16:10:04 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 16:10:04 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 16:10:04 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 16:10:04 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 16:10:04 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 16:10:04 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 16:10:04 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 16:10:04 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 16:10:04 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 16:10:04 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:10:04 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 16:10:04 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:10:04 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 16:10:04 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:10:04 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 16:10:04 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:10:04 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 16:10:04 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:10:04 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 16:10:04 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:10:04 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 16:10:04 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:10:04 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 16:10:04 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:10:04 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:10:04 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 16:10:04 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 16:10:04 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 16:10:04 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (01f5ecc6-a5e6-4864-9c1e-8dc3da5366b1) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 16:10:04 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 16:10:04 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:03:06Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesListTestJSON-Volume-1309258736',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=4d960aac-48dd-470e-831f-eaafe64c9fdf,launched_at=None,metadata={Type='work'},migration_status=None,multiattach=False,previous_status=None,project_id='d1740791cf5e4c7087f45af9015c0a8f',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:07:46Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:07:46Z,use_quota=True,user_id='9091747614a945f6a88207d3662fea8e',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 16:10:04 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:03:06Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesListTestJSON-Volume-1309258736',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=4d960aac-48dd-470e-831f-eaafe64c9fdf,launched_at=None,metadata={Type='work'},migration_status=None,multiattach=False,previous_status=None,project_id='d1740791cf5e4c7087f45af9015c0a8f',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:07:46Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:07:46Z,use_quota=True,user_id='9091747614a945f6a88207d3662fea8e',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(4d960aac-48dd-470e-831f-eaafe64c9fdf),volume_id=4d960aac-48dd-470e-831f-eaafe64c9fdf,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': '4d960aac-48dd-470e-831f-eaafe64c9fdf', 'volume_name': 'volume-4d960aac-48dd-470e-831f-eaafe64c9fdf', 'volume_size': 1}} Apr 17 16:10:04 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': '4d960aac-48dd-470e-831f-eaafe64c9fdf', 'volume': {'id': '4d960aac-48dd-470e-831f-eaafe64c9fdf', '_name_id': None, 'ec2_id': None, 'user_id': '9091747614a945f6a88207d3662fea8e', 'project_id': 'd1740791cf5e4c7087f45af9015c0a8f', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-VolumesListTestJSON-Volume-1309258736', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {'Type': 'work'}, 'admin_metadata': {}, 'glance_metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T16:03:05.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-4d960aac-48dd-470e-831f-eaafe64c9fdf', 'name_id': '4d960aac-48dd-470e-831f-eaafe64c9fdf', 'volume_metadata': [{'key': 'Type', 'value': 'work'}], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumesListTestJSON-Volume-1309258736', 'encryption_key_id': None, 'metadata': {'Type': 'work'}, 'multiattach': False, 'project_id': 'd1740791cf5e4c7087f45af9015c0a8f', 'qos_specs': None, 'reservations': ['61b9d19b-7196-4e0b-aa10-0b704c76275c', 'a121ac12-c1d8-4525-a222-e055f88dc125', '6e5c242e-6241-4c0f-9e09-305237e9921c', '20860b71-70c8-4b28-82e8-4b507fba5ed9'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': '9091747614a945f6a88207d3662fea8e', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumesListTestJSON-Volume-1309258736', 'encryption_key_id': None, 'metadata': {'Type': 'work'}, 'multiattach': False, 'project_id': 'd1740791cf5e4c7087f45af9015c0a8f', 'qos_specs': None, 'reservations': ['61b9d19b-7196-4e0b-aa10-0b704c76275c', 'a121ac12-c1d8-4525-a222-e055f88dc125', '6e5c242e-6241-4c0f-9e09-305237e9921c', '20860b71-70c8-4b28-82e8-4b507fba5ed9'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': '9091747614a945f6a88207d3662fea8e', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': '9091747614a945f6a88207d3662fea8e', 'metadata': {'Type': 'work'}, 'qos_specs': None, 'retry': {'num_attempts': 3, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']}}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(4d960aac-48dd-470e-831f-eaafe64c9fdf),volume_id=4d960aac-48dd-470e-831f-eaafe64c9fdf,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:03:06Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesListTestJSON-Volume-1309258736',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=4d960aac-48dd-470e-831f-eaafe64c9fdf,launched_at=None,metadata={Type='work'},migration_status=None,multiattach=False,previous_status=None,project_id='d1740791cf5e4c7087f45af9015c0a8f',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:07:46Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:07:46Z,use_quota=True,user_id='9091747614a945f6a88207d3662fea8e',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 16:10:04 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:03:06Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesListTestJSON-Volume-1309258736',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=4d960aac-48dd-470e-831f-eaafe64c9fdf,launched_at=None,metadata={Type='work'},migration_status=None,multiattach=False,previous_status=None,project_id='d1740791cf5e4c7087f45af9015c0a8f',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:07:46Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:07:46Z,use_quota=True,user_id='9091747614a945f6a88207d3662fea8e',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:03:06Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesListTestJSON-Volume-1309258736',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=4d960aac-48dd-470e-831f-eaafe64c9fdf,launched_at=None,metadata={Type='work'},migration_status=None,multiattach=False,previous_status=None,project_id='d1740791cf5e4c7087f45af9015c0a8f',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:07:46Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:07:46Z,use_quota=True,user_id='9091747614a945f6a88207d3662fea8e',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 16:10:04 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 16:10:04 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 16:10:04 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:10:04 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 16:10:04 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:10:04 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 16:10:04 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:10:04 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 16:10:04 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:10:04 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 16:10:04 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:10:04 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 16:10:04 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:10:04 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 16:10:04 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:10:04 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 16:10:04 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:10:04 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:10:04 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 16:10:04 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 16:10:04 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (01f5ecc6-a5e6-4864-9c1e-8dc3da5366b1) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:10:04 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (01f5ecc6-a5e6-4864-9c1e-8dc3da5366b1) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:10:04 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (b0ba8568-c132-4d6f-a6e8-f4f50b2880be) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:10:04 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (b0ba8568-c132-4d6f-a6e8-f4f50b2880be) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:10:04 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (1c2cecdd-8308-4439-929b-9a18b2914675) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:10:04 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (1c2cecdd-8308-4439-929b-9a18b2914675) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:10:04 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (0c9c6108-3ef2-48ba-8fbd-83c6bc609fe4) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:10:04 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Updating volume 4d960aac-48dd-470e-831f-eaafe64c9fdf with {'scheduled_at': datetime.datetime(2023, 4, 17, 16, 10, 4, 371358), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 16:10:04 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Volume 4d960aac-48dd-470e-831f-eaafe64c9fdf: re-scheduling SchedulerAPI.create_volume attempt 3 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 16:10:04 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Volume 4d960aac-48dd-470e-831f-eaafe64c9fdf: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 16:10:04 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] deleating volume v_4d960aac-48dd-470e-831f-eaafe64c9fdf {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 16:10:04 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] get snapshots for volume v_4d960aac-48dd-470e-831f-eaafe64c9fdf {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 16:10:04 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_4d960aac-48dd-470e-831f-eaafe64c9fdf/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:10:04 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_4d960aac-48dd-470e-831f-eaafe64c9fdf/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:10:07 user cinder-volume[84696]: DEBUG cinder.utils [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.956(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:10:07 user cinder-volume[84696]: DEBUG cinder.utils [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_4d960aac-48dd-470e-831f-eaafe64c9fdf/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:10:07 user cinder-volume[84696]: DEBUG cinder.utils [req-008ec7fb-052b-43ae-952b-3c2d087ab270 req-15d5486a-4efe-4884-8b8c-05a078b8fa5f tempest-ServerActionsTestJSON-645547919 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 10.836(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:10:07 user cinder-volume[84696]: DEBUG cinder.utils [req-008ec7fb-052b-43ae-952b-3c2d087ab270 req-15d5486a-4efe-4884-8b8c-05a078b8fa5f tempest-ServerActionsTestJSON-645547919 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:1072d05f-2ebb-4398-902c-902df25464e9/luns/v_1072d05f-2ebb-4398-902c-902df25464e9 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:10:10 user cinder-volume[84696]: DEBUG cinder.utils [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.032(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:10:10 user cinder-volume[84696]: DEBUG cinder.utils [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_4d960aac-48dd-470e-831f-eaafe64c9fdf/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:10:10 user cinder-volume[84696]: DEBUG cinder.utils [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 72.543(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:10:10 user cinder-volume[84696]: DEBUG cinder.utils [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:10:14 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=9,cinder:UPDATE=3 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:10:16 user cinder-volume[84696]: DEBUG cinder.utils [req-008ec7fb-052b-43ae-952b-3c2d087ab270 req-de825ae0-7573-4fdc-8780-dc5a66cd78ec tempest-ServerActionsTestJSON-645547919 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 140.100(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:10:16 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-008ec7fb-052b-43ae-952b-3c2d087ab270 req-de825ae0-7573-4fdc-8780-dc5a66cd78ec tempest-ServerActionsTestJSON-645547919 None] Geting None from DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:1072d05f-2ebb-4398-902c-902df25464e9/luns/v_1072d05f-2ebb-4398-902c-902df25464e9 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:10:16 user cinder-volume[84696]: WARNING cinder.volume.manager [req-008ec7fb-052b-43ae-952b-3c2d087ab270 req-de825ae0-7573-4fdc-8780-dc5a66cd78ec tempest-ServerActionsTestJSON-645547919 None] Failed to detach volume on the backend, ignoring failure 'NoneType' object is not subscriptable Apr 17 16:10:16 user cinder-volume[84696]: DEBUG cinder.utils [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.174(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:10:16 user cinder-volume[84696]: DEBUG cinder.utils [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_4d960aac-48dd-470e-831f-eaafe64c9fdf/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:10:16 user cinder-volume[84696]: DEBUG cinder.utils [req-008ec7fb-052b-43ae-952b-3c2d087ab270 req-15d5486a-4efe-4884-8b8c-05a078b8fa5f tempest-ServerActionsTestJSON-645547919 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 20.053(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:10:16 user cinder-volume[84696]: DEBUG cinder.utils [req-008ec7fb-052b-43ae-952b-3c2d087ab270 req-15d5486a-4efe-4884-8b8c-05a078b8fa5f tempest-ServerActionsTestJSON-645547919 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:1072d05f-2ebb-4398-902c-902df25464e9/luns/v_1072d05f-2ebb-4398-902c-902df25464e9 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:10:26 user cinder-volume[84696]: DEBUG cinder.utils [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.900(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:10:26 user cinder-volume[84696]: DEBUG cinder.utils [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_4d960aac-48dd-470e-831f-eaafe64c9fdf/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:10:33 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 139.077(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:10:33 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:10:33 user cinder-volume[84696]: ERROR oslo_service.periodic_task [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Error during VolumeManager.publish_service_capabilities: TypeError: 'NoneType' object is not subscriptable Apr 17 16:10:33 user cinder-volume[84696]: ERROR oslo_service.periodic_task Traceback (most recent call last): Apr 17 16:10:33 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/usr/local/lib/python3.10/dist-packages/oslo_service/periodic_task.py", line 216, in run_periodic_tasks Apr 17 16:10:33 user cinder-volume[84696]: ERROR oslo_service.periodic_task task(self, context) Apr 17 16:10:33 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/manager.py", line 2876, in publish_service_capabilities Apr 17 16:10:33 user cinder-volume[84696]: ERROR oslo_service.periodic_task self._report_driver_status(context) Apr 17 16:10:33 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/manager.py", line 2755, in _report_driver_status Apr 17 16:10:33 user cinder-volume[84696]: ERROR oslo_service.periodic_task volume_stats = get_stats() Apr 17 16:10:33 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/usr/local/lib/python3.10/dist-packages/oslo_utils/timeutils.py", line 319, in wrapper Apr 17 16:10:33 user cinder-volume[84696]: ERROR oslo_service.periodic_task result = func(*args, **kwargs) Apr 17 16:10:33 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/manager.py", line 2753, in get_stats Apr 17 16:10:33 user cinder-volume[84696]: ERROR oslo_service.periodic_task return self.driver.get_volume_stats(refresh=True) Apr 17 16:10:33 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/driver.py", line 666, in get_volume_stats Apr 17 16:10:33 user cinder-volume[84696]: ERROR oslo_service.periodic_task self._update_volume_stats() Apr 17 16:10:33 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 726, in _update_volume_stats Apr 17 16:10:33 user cinder-volume[84696]: ERROR oslo_service.periodic_task pool_stats = self.ra.get_pool_stats() Apr 17 16:10:33 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 897, in get_pool_stats Apr 17 16:10:33 user cinder-volume[84696]: ERROR oslo_service.periodic_task if not resp["error"] and resp["code"] == 200: Apr 17 16:10:33 user cinder-volume[84696]: ERROR oslo_service.periodic_task TypeError: 'NoneType' object is not subscriptable Apr 17 16:10:33 user cinder-volume[84696]: ERROR oslo_service.periodic_task Apr 17 16:10:33 user cinder-volume[84696]: DEBUG cinder.utils [req-008ec7fb-052b-43ae-952b-3c2d087ab270 req-15d5486a-4efe-4884-8b8c-05a078b8fa5f tempest-ServerActionsTestJSON-645547919 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 36.788(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:10:33 user cinder-volume[84696]: DEBUG cinder.utils [req-008ec7fb-052b-43ae-952b-3c2d087ab270 req-15d5486a-4efe-4884-8b8c-05a078b8fa5f tempest-ServerActionsTestJSON-645547919 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:1072d05f-2ebb-4398-902c-902df25464e9/luns/v_1072d05f-2ebb-4398-902c-902df25464e9 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:10:33 user cinder-volume[84696]: WARNING oslo.service.loopingcall [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Function 'cinder.service.Service.periodic_tasks' run outlasted interval by 79.08 sec Apr 17 16:10:33 user cinder-volume[84696]: DEBUG oslo_service.periodic_task [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Running periodic task VolumeManager.publish_service_capabilities {{(pid=84696) run_periodic_tasks /usr/local/lib/python3.10/dist-packages/oslo_service/periodic_task.py:210}} Apr 17 16:10:33 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Updating volume stats {{(pid=84696) _update_volume_stats /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:724}} Apr 17 16:10:33 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Get pool Pool-0 fsprops {{(pid=84696) get_pool_stats /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:894}} Apr 17 16:10:33 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:10:33 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:10:34 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=11,cinder:INSERT=2,cinder:UPDATE=2 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:10:36 user cinder-volume[84696]: DEBUG cinder.utils [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 139.014(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:10:36 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_65a642c3-b1e1-4b80-9bd2-f00151b6304a/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:10:36 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (69725152-efb8-4ceb-b719-32a1332715d2) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 16:10:36 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (8af09775-aa19-414c-9fc5-d5c8ebe3e6e1) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:10:36 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (8af09775-aa19-414c-9fc5-d5c8ebe3e6e1) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:10:36 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Flow 'volume_create_manager' (0572e5ad-45d0-4358-9be6-8f98fe771e9b) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 16:10:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 16:10:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 16:10:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 16:10:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 16:10:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 16:10:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 16:10:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 16:10:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 16:10:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 16:10:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 16:10:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 16:10:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 16:10:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 16:10:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 16:10:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 16:10:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 16:10:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 16:10:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 16:10:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 16:10:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 16:10:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 16:10:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 16:10:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 16:10:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 16:10:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:10:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 16:10:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:10:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 16:10:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:10:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 16:10:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:10:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 16:10:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:10:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 16:10:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:10:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 16:10:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:10:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 16:10:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:10:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:10:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 16:10:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 16:10:36 user cinder-volume[84696]: DEBUG cinder.utils [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 138.862(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:10:36 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:10:36 user cinder-volume[84696]: INFO cinder.message.api [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Creating message record for request_id = req-251ec93b-05ae-4736-ab3c-338a5d9efb63 Apr 17 16:10:36 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 3.091(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:10:36 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:10:36 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 16:10:36 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (d77e7fda-51c7-4be1-93d7-91ea34f7f935) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 16:10:36 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 16:10:36 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:05:55Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-AttachVolumeTestJSON-volume-1580982693',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=65a642c3-b1e1-4b80-9bd2-f00151b6304a,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='2eb998c84b5b41e599af9636b929295d',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:08:17Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:08:17Z,use_quota=True,user_id='b9d4a317838b47d6a2a2e7eab7a453dd',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 16:10:36 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:05:55Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-AttachVolumeTestJSON-volume-1580982693',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=65a642c3-b1e1-4b80-9bd2-f00151b6304a,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='2eb998c84b5b41e599af9636b929295d',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:08:17Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:08:17Z,use_quota=True,user_id='b9d4a317838b47d6a2a2e7eab7a453dd',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(65a642c3-b1e1-4b80-9bd2-f00151b6304a),volume_id=65a642c3-b1e1-4b80-9bd2-f00151b6304a,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': '65a642c3-b1e1-4b80-9bd2-f00151b6304a', 'volume_name': 'volume-65a642c3-b1e1-4b80-9bd2-f00151b6304a', 'volume_size': 1}} Apr 17 16:10:36 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': '65a642c3-b1e1-4b80-9bd2-f00151b6304a', 'volume': {'id': '65a642c3-b1e1-4b80-9bd2-f00151b6304a', '_name_id': None, 'ec2_id': None, 'user_id': 'b9d4a317838b47d6a2a2e7eab7a453dd', 'project_id': '2eb998c84b5b41e599af9636b929295d', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-AttachVolumeTestJSON-volume-1580982693', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'admin_metadata': {}, 'glance_metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T16:05:55.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-65a642c3-b1e1-4b80-9bd2-f00151b6304a', 'name_id': '65a642c3-b1e1-4b80-9bd2-f00151b6304a', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-AttachVolumeTestJSON-volume-1580982693', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '2eb998c84b5b41e599af9636b929295d', 'qos_specs': None, 'reservations': ['1201d95e-040b-4401-9092-4910db00f9ec', '61d42360-766f-47d3-8707-d968887b405f', '54625198-5e49-42b5-a0f1-c764a5a3aaeb', '917c67ca-7481-4181-951b-961fc40b1138'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'b9d4a317838b47d6a2a2e7eab7a453dd', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-AttachVolumeTestJSON-volume-1580982693', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '2eb998c84b5b41e599af9636b929295d', 'qos_specs': None, 'reservations': ['1201d95e-040b-4401-9092-4910db00f9ec', '61d42360-766f-47d3-8707-d968887b405f', '54625198-5e49-42b5-a0f1-c764a5a3aaeb', '917c67ca-7481-4181-951b-961fc40b1138'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'b9d4a317838b47d6a2a2e7eab7a453dd', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': 'b9d4a317838b47d6a2a2e7eab7a453dd', 'metadata': {}, 'qos_specs': None, 'retry': {'num_attempts': 2, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']}}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(65a642c3-b1e1-4b80-9bd2-f00151b6304a),volume_id=65a642c3-b1e1-4b80-9bd2-f00151b6304a,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:05:55Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-AttachVolumeTestJSON-volume-1580982693',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=65a642c3-b1e1-4b80-9bd2-f00151b6304a,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='2eb998c84b5b41e599af9636b929295d',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:08:17Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:08:17Z,use_quota=True,user_id='b9d4a317838b47d6a2a2e7eab7a453dd',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 16:10:36 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:05:55Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-AttachVolumeTestJSON-volume-1580982693',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=65a642c3-b1e1-4b80-9bd2-f00151b6304a,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='2eb998c84b5b41e599af9636b929295d',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:08:17Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:08:17Z,use_quota=True,user_id='b9d4a317838b47d6a2a2e7eab7a453dd',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:05:55Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-AttachVolumeTestJSON-volume-1580982693',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=65a642c3-b1e1-4b80-9bd2-f00151b6304a,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='2eb998c84b5b41e599af9636b929295d',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:08:17Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:08:17Z,use_quota=True,user_id='b9d4a317838b47d6a2a2e7eab7a453dd',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 16:10:36 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 16:10:36 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 16:10:36 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:10:36 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 16:10:36 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:10:36 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 16:10:36 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:10:36 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 16:10:36 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:10:36 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 16:10:36 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:10:36 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 16:10:36 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:10:36 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 16:10:36 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:10:36 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 16:10:36 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:10:36 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:10:36 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 16:10:36 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 16:10:36 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (d77e7fda-51c7-4be1-93d7-91ea34f7f935) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:10:36 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (d77e7fda-51c7-4be1-93d7-91ea34f7f935) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:10:36 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (02ee8a81-0d54-47a3-857a-8bfc890ca5c2) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:10:36 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (02ee8a81-0d54-47a3-857a-8bfc890ca5c2) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:10:36 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (9cae5485-cfd8-4eef-893b-ad692eee180d) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:10:36 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (9cae5485-cfd8-4eef-893b-ad692eee180d) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:10:36 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (8e18a790-1ba9-497b-a9bd-8edd30a199fa) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:10:36 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Updating volume 65a642c3-b1e1-4b80-9bd2-f00151b6304a with {'scheduled_at': datetime.datetime(2023, 4, 17, 16, 10, 36, 463244), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 16:10:36 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Volume 65a642c3-b1e1-4b80-9bd2-f00151b6304a: re-scheduling SchedulerAPI.create_volume attempt 2 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 16:10:36 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Volume 65a642c3-b1e1-4b80-9bd2-f00151b6304a: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 16:10:36 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] deleating volume v_65a642c3-b1e1-4b80-9bd2-f00151b6304a {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 16:10:36 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] get snapshots for volume v_65a642c3-b1e1-4b80-9bd2-f00151b6304a {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 16:10:36 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_65a642c3-b1e1-4b80-9bd2-f00151b6304a/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:10:36 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_65a642c3-b1e1-4b80-9bd2-f00151b6304a/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:10:36 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Volume reschedule parameters: True retry: {'num_attempts': 3, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 16:10:36 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Flow 'volume_create_manager' (3f4b9fa0-2892-458f-906b-35570f6450e1) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 16:10:36 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (7efe3b15-9aa2-436c-8283-68e86b986737) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:10:36 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (7efe3b15-9aa2-436c-8283-68e86b986737) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:05:55Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-AttachVolumeTestJSON-volume-1580982693',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=65a642c3-b1e1-4b80-9bd2-f00151b6304a,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='2eb998c84b5b41e599af9636b929295d',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:10:37Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:10:37Z,use_quota=True,user_id='b9d4a317838b47d6a2a2e7eab7a453dd',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:10:36 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (23d34cdc-882e-4b5d-abc8-2a3d8ec2a97f) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:10:36 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (23d34cdc-882e-4b5d-abc8-2a3d8ec2a97f) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:10:36 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (00e215bb-de2b-4c68-9321-ea88043a137f) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:10:36 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (00e215bb-de2b-4c68-9321-ea88043a137f) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': '65a642c3-b1e1-4b80-9bd2-f00151b6304a', 'volume_name': 'volume-65a642c3-b1e1-4b80-9bd2-f00151b6304a', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:10:36 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (1631cbb1-e31d-4ba1-9964-bc84509c09a7) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:10:36 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (1631cbb1-e31d-4ba1-9964-bc84509c09a7) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:10:36 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (97d3fe9c-7f5d-49bb-b63c-18f72dd43d16) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:10:36 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Volume 65a642c3-b1e1-4b80-9bd2-f00151b6304a: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-65a642c3-b1e1-4b80-9bd2-f00151b6304a', 'volume_size': 1} Apr 17 16:10:36 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] creating volume v_65a642c3-b1e1-4b80-9bd2-f00151b6304a. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 16:10:36 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] create volume {'name': 'v_65a642c3-b1e1-4b80-9bd2-f00151b6304a', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 16:10:36 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:10:36 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:10:39 user cinder-volume[84696]: DEBUG cinder.utils [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.798(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:10:39 user cinder-volume[84696]: DEBUG cinder.utils [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:10:39 user cinder-volume[84696]: DEBUG cinder.utils [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.968(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:10:39 user cinder-volume[84696]: DEBUG cinder.utils [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_65a642c3-b1e1-4b80-9bd2-f00151b6304a/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:10:39 user cinder-volume[84696]: DEBUG cinder.utils [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 138.986(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:10:39 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_63b2c7c3-8482-49d4-bb89-c4ccf3f2db82/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:10:39 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (ae393033-9a6b-4c77-a97c-754828be17e4) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 16:10:39 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (bf5fe72f-b2c5-448c-901a-3175747e95c0) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:10:39 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (bf5fe72f-b2c5-448c-901a-3175747e95c0) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:10:39 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Flow 'volume_create_manager' (4c14647e-2769-45ca-8abd-b0a1a8f00fed) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 16:10:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-a293a946-8aa4-47bc-8290-fe9922d7ae21 tempest-AttachVolumeNegativeTest-331389537 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 16:10:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 16:10:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 16:10:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 16:10:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 16:10:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 16:10:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 16:10:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 16:10:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 16:10:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 16:10:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 16:10:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 16:10:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 16:10:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 16:10:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 16:10:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 16:10:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 16:10:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 16:10:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 16:10:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 16:10:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 16:10:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 16:10:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 16:10:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 16:10:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:10:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 16:10:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:10:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 16:10:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:10:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 16:10:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:10:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 16:10:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:10:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 16:10:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:10:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 16:10:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:10:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 16:10:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:10:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:10:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 16:10:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 16:10:39 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.163(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:10:39 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:10:42 user cinder-volume[84696]: DEBUG cinder.utils [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 38.124(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:10:42 user cinder-volume[84696]: DEBUG cinder.utils [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_4d960aac-48dd-470e-831f-eaafe64c9fdf/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:10:42 user cinder-volume[84696]: DEBUG cinder.utils [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.036(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:10:42 user cinder-volume[84696]: DEBUG cinder.utils [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_65a642c3-b1e1-4b80-9bd2-f00151b6304a/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:10:42 user cinder-volume[84696]: DEBUG cinder.utils [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.868(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:10:42 user cinder-volume[84696]: DEBUG cinder.utils [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:10:44 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=10,cinder:UPDATE=6 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:10:45 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.283(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:10:45 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:10:48 user cinder-volume[84696]: DEBUG cinder.utils [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.178(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:10:48 user cinder-volume[84696]: DEBUG cinder.utils [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_65a642c3-b1e1-4b80-9bd2-f00151b6304a/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:10:48 user cinder-volume[84696]: DEBUG cinder.utils [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.012(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:10:48 user cinder-volume[84696]: DEBUG cinder.utils [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:10:54 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.499(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:10:54 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:10:57 user cinder-volume[84696]: DEBUG cinder.utils [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.394(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:10:57 user cinder-volume[84696]: DEBUG cinder.utils [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_65a642c3-b1e1-4b80-9bd2-f00151b6304a/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:10:57 user cinder-volume[84696]: DEBUG cinder.utils [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.228(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:10:57 user cinder-volume[84696]: DEBUG cinder.utils [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:11:04 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=10,cinder:INSERT=2,cinder:UPDATE=2 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:11:07 user cinder-volume[84696]: DEBUG cinder.utils [req-008ec7fb-052b-43ae-952b-3c2d087ab270 req-15d5486a-4efe-4884-8b8c-05a078b8fa5f tempest-ServerActionsTestJSON-645547919 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 70.898(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:11:07 user cinder-volume[84696]: DEBUG cinder.utils [req-008ec7fb-052b-43ae-952b-3c2d087ab270 req-15d5486a-4efe-4884-8b8c-05a078b8fa5f tempest-ServerActionsTestJSON-645547919 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:1072d05f-2ebb-4398-902c-902df25464e9/luns/v_1072d05f-2ebb-4398-902c-902df25464e9 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:11:10 user cinder-volume[84696]: DEBUG cinder.utils [None req-e0496134-2447-4744-8a98-847f63a20a7b tempest-GroupsV320Test-1696456310 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 139.304(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:11:10 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-e0496134-2447-4744-8a98-847f63a20a7b tempest-GroupsV320Test-1696456310 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:11:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-e0496134-2447-4744-8a98-847f63a20a7b tempest-GroupsV320Test-1696456310 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 16:11:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 16:11:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 16:11:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 16:11:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 16:11:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 16:11:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 16:11:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 16:11:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 3864, in delete_group Apr 17 16:11:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.publish_service_capabilities(context) Apr 17 16:11:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 2876, in publish_service_capabilities Apr 17 16:11:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self._report_driver_status(context) Apr 17 16:11:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 2755, in _report_driver_status Apr 17 16:11:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server volume_stats = get_stats() Apr 17 16:11:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/timeutils.py", line 319, in wrapper Apr 17 16:11:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(*args, **kwargs) Apr 17 16:11:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 2753, in get_stats Apr 17 16:11:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self.driver.get_volume_stats(refresh=True) Apr 17 16:11:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/driver.py", line 666, in get_volume_stats Apr 17 16:11:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self._update_volume_stats() Apr 17 16:11:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 726, in _update_volume_stats Apr 17 16:11:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server pool_stats = self.ra.get_pool_stats() Apr 17 16:11:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 897, in get_pool_stats Apr 17 16:11:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] == 200: Apr 17 16:11:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 16:11:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 16:11:13 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 40.251(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:11:13 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:11:14 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:UPDATE=1,cinder:SELECT=2 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:11:16 user cinder-volume[84696]: DEBUG cinder.utils [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 138.745(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:11:16 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:11:16 user cinder-volume[84696]: INFO cinder.message.api [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Creating message record for request_id = req-40e48e44-d46b-4e9d-82c5-3b795728a90d Apr 17 16:11:16 user cinder-volume[84696]: DEBUG cinder.utils [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 40.150(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:11:16 user cinder-volume[84696]: DEBUG cinder.utils [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_65a642c3-b1e1-4b80-9bd2-f00151b6304a/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:11:16 user cinder-volume[84696]: DEBUG cinder.utils [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.982(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:11:16 user cinder-volume[84696]: DEBUG cinder.utils [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:11:16 user cinder-volume[84696]: DEBUG cinder.utils [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 72.241(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:11:16 user cinder-volume[84696]: DEBUG cinder.utils [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_4d960aac-48dd-470e-831f-eaafe64c9fdf/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:11:16 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 16:11:16 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (346283e3-48d0-4d43-87e5-6196db961c87) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 16:11:16 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 16:11:16 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:08:58Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesTransfersV355Test-Volume-1916167074',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=49effcda-e787-4983-92b2-f8261052732d,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='9796cc8ccc334a5b86ff098b10a0c413',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:08:58Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:08:58Z,use_quota=True,user_id='de05a598447a4e9fb3ff488ec2d84071',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 16:11:16 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:08:58Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesTransfersV355Test-Volume-1916167074',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=49effcda-e787-4983-92b2-f8261052732d,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='9796cc8ccc334a5b86ff098b10a0c413',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:08:58Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:08:58Z,use_quota=True,user_id='de05a598447a4e9fb3ff488ec2d84071',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(49effcda-e787-4983-92b2-f8261052732d),volume_id=49effcda-e787-4983-92b2-f8261052732d,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': '49effcda-e787-4983-92b2-f8261052732d', 'volume_name': 'volume-49effcda-e787-4983-92b2-f8261052732d', 'volume_size': 1}} Apr 17 16:11:16 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'retry': {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']}, 'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': '49effcda-e787-4983-92b2-f8261052732d', 'volume': {'id': '49effcda-e787-4983-92b2-f8261052732d', '_name_id': None, 'ec2_id': None, 'user_id': 'de05a598447a4e9fb3ff488ec2d84071', 'project_id': '9796cc8ccc334a5b86ff098b10a0c413', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-VolumesTransfersV355Test-Volume-1916167074', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T16:08:57.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-49effcda-e787-4983-92b2-f8261052732d', 'name_id': '49effcda-e787-4983-92b2-f8261052732d', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumesTransfersV355Test-Volume-1916167074', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '9796cc8ccc334a5b86ff098b10a0c413', 'qos_specs': None, 'reservations': ['18f38527-7ebd-41b2-8296-efa6d1a6acd7', '6a781a35-53f4-4739-a5cc-a9986cdaa2fe', 'd632f456-4c05-4c56-84e4-431db2216088', '398f088d-a47c-4ff9-aee7-12eb5bef8d35'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'de05a598447a4e9fb3ff488ec2d84071', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumesTransfersV355Test-Volume-1916167074', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '9796cc8ccc334a5b86ff098b10a0c413', 'qos_specs': None, 'reservations': ['18f38527-7ebd-41b2-8296-efa6d1a6acd7', '6a781a35-53f4-4739-a5cc-a9986cdaa2fe', 'd632f456-4c05-4c56-84e4-431db2216088', '398f088d-a47c-4ff9-aee7-12eb5bef8d35'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'de05a598447a4e9fb3ff488ec2d84071', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': 'de05a598447a4e9fb3ff488ec2d84071', 'metadata': {}, 'qos_specs': None}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(49effcda-e787-4983-92b2-f8261052732d),volume_id=49effcda-e787-4983-92b2-f8261052732d,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:08:58Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesTransfersV355Test-Volume-1916167074',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=49effcda-e787-4983-92b2-f8261052732d,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='9796cc8ccc334a5b86ff098b10a0c413',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:08:58Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:08:58Z,use_quota=True,user_id='de05a598447a4e9fb3ff488ec2d84071',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 16:11:16 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:08:58Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesTransfersV355Test-Volume-1916167074',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=49effcda-e787-4983-92b2-f8261052732d,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='9796cc8ccc334a5b86ff098b10a0c413',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:08:58Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:08:58Z,use_quota=True,user_id='de05a598447a4e9fb3ff488ec2d84071',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:08:58Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesTransfersV355Test-Volume-1916167074',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=49effcda-e787-4983-92b2-f8261052732d,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='9796cc8ccc334a5b86ff098b10a0c413',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:08:58Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:08:58Z,use_quota=True,user_id='de05a598447a4e9fb3ff488ec2d84071',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 16:11:16 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 16:11:16 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 16:11:16 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:11:16 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 16:11:16 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:11:16 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 16:11:16 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:11:16 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 16:11:16 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:11:16 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 16:11:16 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:11:16 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 16:11:16 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:11:16 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 16:11:16 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:11:16 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 16:11:16 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:11:16 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:11:16 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 16:11:16 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 16:11:16 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (346283e3-48d0-4d43-87e5-6196db961c87) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:11:16 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (346283e3-48d0-4d43-87e5-6196db961c87) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:11:16 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (7c6d64c8-6145-4ee2-ba3b-04a2ed0734d5) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:11:16 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (7c6d64c8-6145-4ee2-ba3b-04a2ed0734d5) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:11:16 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (331f78e6-704c-4c85-9cf4-7b1b63e706b8) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:11:16 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (331f78e6-704c-4c85-9cf4-7b1b63e706b8) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:11:16 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (85e8bd62-0b6e-4718-b1b6-4e9d77ee4eeb) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:11:16 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Updating volume 49effcda-e787-4983-92b2-f8261052732d with {'scheduled_at': datetime.datetime(2023, 4, 17, 16, 11, 16, 702149), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 16:11:16 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Volume 49effcda-e787-4983-92b2-f8261052732d: re-scheduling SchedulerAPI.create_volume attempt 1 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 16:11:16 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Volume 49effcda-e787-4983-92b2-f8261052732d: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 16:11:16 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] deleating volume v_49effcda-e787-4983-92b2-f8261052732d {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 16:11:16 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] get snapshots for volume v_49effcda-e787-4983-92b2-f8261052732d {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 16:11:16 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_49effcda-e787-4983-92b2-f8261052732d/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:11:16 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_49effcda-e787-4983-92b2-f8261052732d/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:11:16 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Volume reschedule parameters: True retry: {'num_attempts': 2, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 16:11:16 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Flow 'volume_create_manager' (8d475cc0-3d90-45f1-aca7-4f5582599063) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 16:11:16 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (5d43cfbc-acb0-4d8b-80f6-2b361bd00590) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:11:16 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (5d43cfbc-acb0-4d8b-80f6-2b361bd00590) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:08:58Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesTransfersV355Test-Volume-1916167074',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=49effcda-e787-4983-92b2-f8261052732d,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='9796cc8ccc334a5b86ff098b10a0c413',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:11:17Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:11:17Z,use_quota=True,user_id='de05a598447a4e9fb3ff488ec2d84071',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:11:16 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (3d2ff43b-f7cd-4323-a7ca-708272500c69) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:11:16 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (3d2ff43b-f7cd-4323-a7ca-708272500c69) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:11:16 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (734bf18c-4925-40b1-aadc-d990a830512a) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:11:16 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (734bf18c-4925-40b1-aadc-d990a830512a) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': '49effcda-e787-4983-92b2-f8261052732d', 'volume_name': 'volume-49effcda-e787-4983-92b2-f8261052732d', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:11:16 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (efc22af4-edd7-4f33-9025-66abbe62a0ed) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:11:16 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (efc22af4-edd7-4f33-9025-66abbe62a0ed) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:11:16 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (a1c24603-84f5-4df0-9bcf-c5809e299c36) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:11:16 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Volume 49effcda-e787-4983-92b2-f8261052732d: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-49effcda-e787-4983-92b2-f8261052732d', 'volume_size': 1} Apr 17 16:11:16 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] creating volume v_49effcda-e787-4983-92b2-f8261052732d. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 16:11:16 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] create volume {'name': 'v_49effcda-e787-4983-92b2-f8261052732d', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 16:11:16 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:11:16 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:11:18 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Volume reschedule parameters: True retry: {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 16:11:18 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Flow 'volume_create_manager' (d86e39d2-4925-436d-a8a4-c8d45174658c) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 16:11:18 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (e6ebcd93-56f8-4a20-84fa-a6d4959579b7) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:11:18 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (e6ebcd93-56f8-4a20-84fa-a6d4959579b7) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:11:18Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesGetTestJSON-Volume-1455502637',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=0a7d4209-92e0-4900-8e98-36a62e68c247,launched_at=None,metadata={Type='work'},migration_status=None,multiattach=False,previous_status=None,project_id='e330412972fa44b9bf60cd9afe4399cd',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:11:18Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:11:18Z,use_quota=True,user_id='e14688d19f1b44928fa8aceb2da0ac82',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:11:18 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (c14008a8-e359-4437-b365-01be3b46be87) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:11:18 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (c14008a8-e359-4437-b365-01be3b46be87) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:11:18 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (6dbd5de6-65ed-4d90-a87c-0913b47bbfc5) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:11:18 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (6dbd5de6-65ed-4d90-a87c-0913b47bbfc5) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': '0a7d4209-92e0-4900-8e98-36a62e68c247', 'volume_name': 'volume-0a7d4209-92e0-4900-8e98-36a62e68c247', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:11:18 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (720939a1-4205-4e33-893c-eeeaf74c03ee) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:11:18 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (720939a1-4205-4e33-893c-eeeaf74c03ee) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:11:18 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (82fdf2da-94ae-4259-9a5e-f30f2727b449) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:11:18 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Volume 0a7d4209-92e0-4900-8e98-36a62e68c247: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-0a7d4209-92e0-4900-8e98-36a62e68c247', 'volume_size': 1} Apr 17 16:11:18 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] creating volume v_0a7d4209-92e0-4900-8e98-36a62e68c247. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 16:11:18 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] create volume {'name': 'v_0a7d4209-92e0-4900-8e98-36a62e68c247', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 16:11:18 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:11:18 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:11:19 user cinder-volume[84696]: DEBUG cinder.utils [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.978(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:11:19 user cinder-volume[84696]: DEBUG cinder.utils [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_49effcda-e787-4983-92b2-f8261052732d/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:11:19 user cinder-volume[84696]: DEBUG cinder.utils [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 1.400(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:11:19 user cinder-volume[84696]: DEBUG cinder.utils [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:11:19 user cinder-volume[84696]: DEBUG cinder.utils [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.786(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:11:19 user cinder-volume[84696]: DEBUG cinder.utils [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:11:22 user cinder-volume[84696]: DEBUG cinder.utils [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 4.469(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:11:22 user cinder-volume[84696]: DEBUG cinder.utils [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:11:22 user cinder-volume[84696]: DEBUG cinder.utils [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.052(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:11:22 user cinder-volume[84696]: DEBUG cinder.utils [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_49effcda-e787-4983-92b2-f8261052732d/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:11:22 user cinder-volume[84696]: DEBUG cinder.utils [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.856(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:11:22 user cinder-volume[84696]: DEBUG cinder.utils [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:11:24 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-bb0706f3-0b2c-473a-acc5-93c2ac742a5f req-c78e09d4-3679-4ee2-ab6c-f8158e1aaa1d tempest-ServerStableDeviceRescueTest-869290889 None] terminate connection for 32819ec1-2236-465f-a1c7-ee31aa4a0911 {{(pid=84696) terminate_connection /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:1032}} Apr 17 16:11:24 user cinder-volume[84696]: INFO cinder.volume.manager [req-bb0706f3-0b2c-473a-acc5-93c2ac742a5f req-c78e09d4-3679-4ee2-ab6c-f8158e1aaa1d tempest-ServerStableDeviceRescueTest-869290889 None] Terminate volume connection completed successfully. Apr 17 16:11:24 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-bb0706f3-0b2c-473a-acc5-93c2ac742a5f req-c78e09d4-3679-4ee2-ab6c-f8158e1aaa1d tempest-ServerStableDeviceRescueTest-869290889 None] Deleting attachment fb44dce3-1af2-43cf-95fa-68ff027e0699. {{(pid=84696) attachment_delete /opt/stack/cinder/cinder/volume/manager.py:5006}} Apr 17 16:11:24 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-bb0706f3-0b2c-473a-acc5-93c2ac742a5f req-c78e09d4-3679-4ee2-ab6c-f8158e1aaa1d tempest-ServerStableDeviceRescueTest-869290889 None] remove_export for volume: 32819ec1-2236-465f-a1c7-ee31aa4a0911. {{(pid=84696) remove_export /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:718}} Apr 17 16:11:24 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-bb0706f3-0b2c-473a-acc5-93c2ac742a5f req-c78e09d4-3679-4ee2-ab6c-f8158e1aaa1d tempest-ServerStableDeviceRescueTest-869290889 None] remove export {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:912}} Apr 17 16:11:24 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-bb0706f3-0b2c-473a-acc5-93c2ac742a5f req-c78e09d4-3679-4ee2-ab6c-f8158e1aaa1d tempest-ServerStableDeviceRescueTest-869290889 None] detach volume:Volume(_name_id=None,admin_metadata=,attach_status='attached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:27:42Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-ServerStableDeviceRescueTest-volume-648918308',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=32819ec1-2236-465f-a1c7-ee31aa4a0911,launched_at=2023-04-17T15:27:46Z,metadata=,migration_status=None,multiattach=False,previous_status=None,project_id='6bdfb8642b9341f6ba289e920c33cf47',provider_auth='CHAP CmhLYVJy GWkyzUl8QJxnjY',provider_geometry=None,provider_id=None,provider_location='172.16.0.220:3260,1 iqn.2010-10.org.openstack:32819ec1-2236-465f-a1c7-ee31aa4a0911 0',replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:27:43Z,service_uuid='e628352e-c4cf-40eb-8389-cce40663026a',shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='in-use',terminated_at=None,updated_at=2023-04-17T15:27:57Z,use_quota=True,user_id='a972c04314724233bc8e33b64747fecf',volume_attachment=,volume_type=,volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c) from target:iqn.2010-10.org.openstack:32819ec1-2236-465f-a1c7-ee31aa4a0911. {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:913}} Apr 17 16:11:24 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [req-bb0706f3-0b2c-473a-acc5-93c2ac742a5f req-c78e09d4-3679-4ee2-ab6c-f8158e1aaa1d tempest-ServerStableDeviceRescueTest-869290889 None] detach volume v_32819ec1-2236-465f-a1c7-ee31aa4a0911 from target iqn.2010-10.org.openstack:32819ec1-2236-465f-a1c7-ee31aa4a0911 {{(pid=84696) detach_target_vol /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:606}} Apr 17 16:11:24 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-bb0706f3-0b2c-473a-acc5-93c2ac742a5f req-c78e09d4-3679-4ee2-ab6c-f8158e1aaa1d tempest-ServerStableDeviceRescueTest-869290889 None] Sending pool request DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:32819ec1-2236-465f-a1c7-ee31aa4a0911/luns/v_32819ec1-2236-465f-a1c7-ee31aa4a0911 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:11:24 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-bb0706f3-0b2c-473a-acc5-93c2ac742a5f req-c78e09d4-3679-4ee2-ab6c-f8158e1aaa1d tempest-ServerStableDeviceRescueTest-869290889 None] Sending DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:32819ec1-2236-465f-a1c7-ee31aa4a0911/luns/v_32819ec1-2236-465f-a1c7-ee31aa4a0911 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:11:25 user cinder-volume[84696]: DEBUG cinder.utils [req-bb0706f3-0b2c-473a-acc5-93c2ac742a5f req-c78e09d4-3679-4ee2-ab6c-f8158e1aaa1d tempest-ServerStableDeviceRescueTest-869290889 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 1.603(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:11:25 user cinder-volume[84696]: DEBUG cinder.utils [req-bb0706f3-0b2c-473a-acc5-93c2ac742a5f req-c78e09d4-3679-4ee2-ab6c-f8158e1aaa1d tempest-ServerStableDeviceRescueTest-869290889 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:32819ec1-2236-465f-a1c7-ee31aa4a0911/luns/v_32819ec1-2236-465f-a1c7-ee31aa4a0911 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:11:28 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Volume reschedule parameters: True retry: {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 16:11:28 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Flow 'volume_create_manager' (2c5d7b4e-4249-4979-b15e-0da0ce7eb111) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 16:11:28 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (9dc28f78-d7a1-4c45-9145-3b3ea2bffdb5) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:11:28 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (9dc28f78-d7a1-4c45-9145-3b3ea2bffdb5) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:11:28Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesSnapshotsTestJSON-volume-126016572',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=85fcb006-cd96-4018-b0e3-04872ff08267,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='34ab34dfa2e14c3785276a62a076f8e8',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:11:28Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:11:28Z,use_quota=True,user_id='c98520ee1d2c4eabaf6c186bd3ae7fee',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:11:28 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (eb867ed7-2620-4fdf-9e2c-6a76b6a3779a) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:11:28 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (eb867ed7-2620-4fdf-9e2c-6a76b6a3779a) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:11:28 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (b049a73b-c509-4dd8-a72c-6e48206f5c59) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:11:28 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (b049a73b-c509-4dd8-a72c-6e48206f5c59) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': '85fcb006-cd96-4018-b0e3-04872ff08267', 'volume_name': 'volume-85fcb006-cd96-4018-b0e3-04872ff08267', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:11:28 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (ea60f5fe-e36e-419a-897b-8806e39051b6) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:11:28 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (ea60f5fe-e36e-419a-897b-8806e39051b6) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:11:28 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (d347957b-4463-462d-ac29-65eaf5d8a189) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:11:28 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Volume 85fcb006-cd96-4018-b0e3-04872ff08267: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-85fcb006-cd96-4018-b0e3-04872ff08267', 'volume_size': 1} Apr 17 16:11:28 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] creating volume v_85fcb006-cd96-4018-b0e3-04872ff08267. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 16:11:28 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] create volume {'name': 'v_85fcb006-cd96-4018-b0e3-04872ff08267', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 16:11:28 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:11:28 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:11:28 user cinder-volume[84696]: DEBUG cinder.utils [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.194(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:11:28 user cinder-volume[84696]: DEBUG cinder.utils [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_49effcda-e787-4983-92b2-f8261052732d/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:11:28 user cinder-volume[84696]: DEBUG cinder.utils [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 10.614(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:11:28 user cinder-volume[84696]: DEBUG cinder.utils [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:11:28 user cinder-volume[84696]: DEBUG cinder.utils [req-bb0706f3-0b2c-473a-acc5-93c2ac742a5f req-c78e09d4-3679-4ee2-ab6c-f8158e1aaa1d tempest-ServerStableDeviceRescueTest-869290889 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 4.677(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:11:28 user cinder-volume[84696]: DEBUG cinder.utils [req-bb0706f3-0b2c-473a-acc5-93c2ac742a5f req-c78e09d4-3679-4ee2-ab6c-f8158e1aaa1d tempest-ServerStableDeviceRescueTest-869290889 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:32819ec1-2236-465f-a1c7-ee31aa4a0911/luns/v_32819ec1-2236-465f-a1c7-ee31aa4a0911 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:11:28 user cinder-volume[84696]: DEBUG cinder.utils [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.000(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:11:28 user cinder-volume[84696]: DEBUG cinder.utils [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:11:28 user cinder-volume[84696]: DEBUG cinder.utils [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 0.338(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:11:28 user cinder-volume[84696]: DEBUG cinder.utils [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:11:31 user cinder-volume[84696]: DEBUG cinder.utils [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 3.406(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:11:31 user cinder-volume[84696]: DEBUG cinder.utils [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:11:34 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=19,cinder:UPDATE=4,cinder:INSERT=1 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:11:35 user cinder-volume[84696]: DEBUG cinder.utils [req-bb0706f3-0b2c-473a-acc5-93c2ac742a5f req-c78e09d4-3679-4ee2-ab6c-f8158e1aaa1d tempest-ServerStableDeviceRescueTest-869290889 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 10.819(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:11:35 user cinder-volume[84696]: DEBUG cinder.utils [req-bb0706f3-0b2c-473a-acc5-93c2ac742a5f req-c78e09d4-3679-4ee2-ab6c-f8158e1aaa1d tempest-ServerStableDeviceRescueTest-869290889 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:32819ec1-2236-465f-a1c7-ee31aa4a0911/luns/v_32819ec1-2236-465f-a1c7-ee31aa4a0911 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:11:38 user cinder-volume[84696]: DEBUG cinder.utils [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 9.551(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:11:38 user cinder-volume[84696]: DEBUG cinder.utils [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:11:38 user cinder-volume[84696]: DEBUG cinder.utils [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.411(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:11:38 user cinder-volume[84696]: DEBUG cinder.utils [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_49effcda-e787-4983-92b2-f8261052732d/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:11:38 user cinder-volume[84696]: DEBUG cinder.utils [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.216(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:11:38 user cinder-volume[84696]: DEBUG cinder.utils [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:11:38 user cinder-volume[84696]: DEBUG cinder.utils [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 19.833(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:11:38 user cinder-volume[84696]: DEBUG cinder.utils [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:11:44 user cinder-volume[84696]: DEBUG cinder.utils [req-bb0706f3-0b2c-473a-acc5-93c2ac742a5f req-c78e09d4-3679-4ee2-ab6c-f8158e1aaa1d tempest-ServerStableDeviceRescueTest-869290889 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 20.035(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:11:44 user cinder-volume[84696]: DEBUG cinder.utils [req-bb0706f3-0b2c-473a-acc5-93c2ac742a5f req-c78e09d4-3679-4ee2-ab6c-f8158e1aaa1d tempest-ServerStableDeviceRescueTest-869290889 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:32819ec1-2236-465f-a1c7-ee31aa4a0911/luns/v_32819ec1-2236-465f-a1c7-ee31aa4a0911 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:11:44 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:INSERT=1,cinder:UPDATE=4,cinder:SELECT=13 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:11:47 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 74.043(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:11:47 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:11:47 user cinder-volume[84696]: DEBUG cinder.utils [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 18.769(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:11:47 user cinder-volume[84696]: DEBUG cinder.utils [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:11:50 user cinder-volume[84696]: DEBUG cinder.utils [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 73.773(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:11:50 user cinder-volume[84696]: DEBUG cinder.utils [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:11:50 user cinder-volume[84696]: DEBUG cinder.utils [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 73.943(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:11:50 user cinder-volume[84696]: DEBUG cinder.utils [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_65a642c3-b1e1-4b80-9bd2-f00151b6304a/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:11:54 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=3,cinder:UPDATE=1 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:11:56 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-008ec7fb-052b-43ae-952b-3c2d087ab270 req-625dc9c1-5166-463d-b990-0b1f19b50106 tempest-ServerActionsTestJSON-645547919 None] terminate connection for 1072d05f-2ebb-4398-902c-902df25464e9 {{(pid=84696) terminate_connection /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:1032}} Apr 17 16:11:56 user cinder-volume[84696]: INFO cinder.volume.manager [req-008ec7fb-052b-43ae-952b-3c2d087ab270 req-625dc9c1-5166-463d-b990-0b1f19b50106 tempest-ServerActionsTestJSON-645547919 None] Terminate volume connection completed successfully. Apr 17 16:11:56 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-008ec7fb-052b-43ae-952b-3c2d087ab270 req-625dc9c1-5166-463d-b990-0b1f19b50106 tempest-ServerActionsTestJSON-645547919 None] Deleting attachment a68f5fc4-dea3-48a4-bdb0-1d89db356c29. {{(pid=84696) attachment_delete /opt/stack/cinder/cinder/volume/manager.py:5006}} Apr 17 16:11:56 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-008ec7fb-052b-43ae-952b-3c2d087ab270 req-625dc9c1-5166-463d-b990-0b1f19b50106 tempest-ServerActionsTestJSON-645547919 None] remove_export for volume: 1072d05f-2ebb-4398-902c-902df25464e9. {{(pid=84696) remove_export /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:718}} Apr 17 16:11:56 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-008ec7fb-052b-43ae-952b-3c2d087ab270 req-625dc9c1-5166-463d-b990-0b1f19b50106 tempest-ServerActionsTestJSON-645547919 None] remove export {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:912}} Apr 17 16:11:56 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-008ec7fb-052b-43ae-952b-3c2d087ab270 req-625dc9c1-5166-463d-b990-0b1f19b50106 tempest-ServerActionsTestJSON-645547919 None] detach volume:Volume(_name_id=None,admin_metadata=,attach_status='attached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:27:39Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-ServerActionsTestJSON-volume-1520907984',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=1072d05f-2ebb-4398-902c-902df25464e9,launched_at=2023-04-17T15:27:40Z,metadata=,migration_status=None,multiattach=False,previous_status=None,project_id='cb5f6b5ed2fc43b69c47e1fc18646545',provider_auth='CHAP NKISaVyG nHFlRwNr8b5KS2',provider_geometry=None,provider_id=None,provider_location='172.16.0.220:3260,1 iqn.2010-10.org.openstack:1072d05f-2ebb-4398-902c-902df25464e9 0',replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:27:39Z,service_uuid='e628352e-c4cf-40eb-8389-cce40663026a',shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='in-use',terminated_at=None,updated_at=2023-04-17T15:32:47Z,use_quota=True,user_id='638a4f535e9740fc9d810639a5eb24d7',volume_attachment=,volume_type=,volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c) from target:iqn.2010-10.org.openstack:1072d05f-2ebb-4398-902c-902df25464e9. {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:913}} Apr 17 16:11:56 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [req-008ec7fb-052b-43ae-952b-3c2d087ab270 req-625dc9c1-5166-463d-b990-0b1f19b50106 tempest-ServerActionsTestJSON-645547919 None] detach volume v_1072d05f-2ebb-4398-902c-902df25464e9 from target iqn.2010-10.org.openstack:1072d05f-2ebb-4398-902c-902df25464e9 {{(pid=84696) detach_target_vol /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:606}} Apr 17 16:11:56 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-008ec7fb-052b-43ae-952b-3c2d087ab270 req-625dc9c1-5166-463d-b990-0b1f19b50106 tempest-ServerActionsTestJSON-645547919 None] Sending pool request DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:1072d05f-2ebb-4398-902c-902df25464e9/luns/v_1072d05f-2ebb-4398-902c-902df25464e9 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:11:56 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-008ec7fb-052b-43ae-952b-3c2d087ab270 req-625dc9c1-5166-463d-b990-0b1f19b50106 tempest-ServerActionsTestJSON-645547919 None] Sending DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:1072d05f-2ebb-4398-902c-902df25464e9/luns/v_1072d05f-2ebb-4398-902c-902df25464e9 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:11:56 user cinder-volume[84696]: DEBUG cinder.utils [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.645(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:11:56 user cinder-volume[84696]: DEBUG cinder.utils [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:11:56 user cinder-volume[84696]: DEBUG cinder.utils [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.843(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:11:56 user cinder-volume[84696]: DEBUG cinder.utils [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_49effcda-e787-4983-92b2-f8261052732d/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:11:56 user cinder-volume[84696]: DEBUG cinder.utils [req-008ec7fb-052b-43ae-952b-3c2d087ab270 req-625dc9c1-5166-463d-b990-0b1f19b50106 tempest-ServerActionsTestJSON-645547919 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 0.019(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:11:56 user cinder-volume[84696]: DEBUG cinder.utils [req-008ec7fb-052b-43ae-952b-3c2d087ab270 req-625dc9c1-5166-463d-b990-0b1f19b50106 tempest-ServerActionsTestJSON-645547919 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:1072d05f-2ebb-4398-902c-902df25464e9/luns/v_1072d05f-2ebb-4398-902c-902df25464e9 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:11:56 user cinder-volume[84696]: DEBUG cinder.utils [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 38.264(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:11:56 user cinder-volume[84696]: DEBUG cinder.utils [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:11:57 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Volume reschedule parameters: True retry: {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 16:11:57 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Flow 'volume_create_manager' (4168c4f1-1544-4b91-aba3-a1021cbaea61) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 16:11:57 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (5967b5df-f73b-4f24-8f7c-ecba2ea1601d) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:11:57 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (5967b5df-f73b-4f24-8f7c-ecba2ea1601d) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:11:57Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-AttachVolumeTestJSON-volume-1944488700',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=bf4c0b13-49c6-4b1d-b790-7090eb82d4c4,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='2eb998c84b5b41e599af9636b929295d',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:11:57Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:11:57Z,use_quota=True,user_id='b9d4a317838b47d6a2a2e7eab7a453dd',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:11:57 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (796c9884-6828-42aa-af51-26e3f4a7894b) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:11:57 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (796c9884-6828-42aa-af51-26e3f4a7894b) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:11:57 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (302605de-c8ab-4ef2-8494-5b3df8c7e043) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:11:57 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (302605de-c8ab-4ef2-8494-5b3df8c7e043) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': 'bf4c0b13-49c6-4b1d-b790-7090eb82d4c4', 'volume_name': 'volume-bf4c0b13-49c6-4b1d-b790-7090eb82d4c4', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:11:57 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (d2970805-692b-4145-9883-29442b556549) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:11:57 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (d2970805-692b-4145-9883-29442b556549) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:11:57 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (10ad4d26-3665-4d72-aa82-e8ed3c66581a) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:11:57 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Volume bf4c0b13-49c6-4b1d-b790-7090eb82d4c4: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-bf4c0b13-49c6-4b1d-b790-7090eb82d4c4', 'volume_size': 1} Apr 17 16:11:57 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] creating volume v_bf4c0b13-49c6-4b1d-b790-7090eb82d4c4. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 16:11:57 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] create volume {'name': 'v_bf4c0b13-49c6-4b1d-b790-7090eb82d4c4', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 16:11:57 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:11:57 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:11:59 user cinder-volume[84696]: DEBUG cinder.utils [req-008ec7fb-052b-43ae-952b-3c2d087ab270 req-625dc9c1-5166-463d-b990-0b1f19b50106 tempest-ServerActionsTestJSON-645547919 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 3.089(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:11:59 user cinder-volume[84696]: DEBUG cinder.utils [req-008ec7fb-052b-43ae-952b-3c2d087ab270 req-625dc9c1-5166-463d-b990-0b1f19b50106 tempest-ServerActionsTestJSON-645547919 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:1072d05f-2ebb-4398-902c-902df25464e9/luns/v_1072d05f-2ebb-4398-902c-902df25464e9 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:11:59 user cinder-volume[84696]: DEBUG cinder.utils [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.145(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:11:59 user cinder-volume[84696]: DEBUG cinder.utils [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:12:02 user cinder-volume[84696]: DEBUG cinder.utils [req-bb0706f3-0b2c-473a-acc5-93c2ac742a5f req-c78e09d4-3679-4ee2-ab6c-f8158e1aaa1d tempest-ServerStableDeviceRescueTest-869290889 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 38.467(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:12:02 user cinder-volume[84696]: DEBUG cinder.utils [req-bb0706f3-0b2c-473a-acc5-93c2ac742a5f req-c78e09d4-3679-4ee2-ab6c-f8158e1aaa1d tempest-ServerStableDeviceRescueTest-869290889 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:32819ec1-2236-465f-a1c7-ee31aa4a0911/luns/v_32819ec1-2236-465f-a1c7-ee31aa4a0911 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:12:02 user cinder-volume[84696]: DEBUG cinder.utils [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.216(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:12:02 user cinder-volume[84696]: DEBUG cinder.utils [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:12:05 user cinder-volume[84696]: DEBUG cinder.utils [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 37.198(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:12:05 user cinder-volume[84696]: DEBUG cinder.utils [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:12:05 user cinder-volume[84696]: DEBUG cinder.utils [req-008ec7fb-052b-43ae-952b-3c2d087ab270 req-625dc9c1-5166-463d-b990-0b1f19b50106 tempest-ServerActionsTestJSON-645547919 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 9.234(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:12:05 user cinder-volume[84696]: DEBUG cinder.utils [req-008ec7fb-052b-43ae-952b-3c2d087ab270 req-625dc9c1-5166-463d-b990-0b1f19b50106 tempest-ServerActionsTestJSON-645547919 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:1072d05f-2ebb-4398-902c-902df25464e9/luns/v_1072d05f-2ebb-4398-902c-902df25464e9 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:12:08 user cinder-volume[84696]: DEBUG cinder.utils [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 11.359(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:12:08 user cinder-volume[84696]: DEBUG cinder.utils [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:12:11 user cinder-volume[84696]: DEBUG cinder.utils [req-008ec7fb-052b-43ae-952b-3c2d087ab270 req-15d5486a-4efe-4884-8b8c-05a078b8fa5f tempest-ServerActionsTestJSON-645547919 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 135.413(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:12:11 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-008ec7fb-052b-43ae-952b-3c2d087ab270 req-15d5486a-4efe-4884-8b8c-05a078b8fa5f tempest-ServerActionsTestJSON-645547919 None] Geting None from DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:1072d05f-2ebb-4398-902c-902df25464e9/luns/v_1072d05f-2ebb-4398-902c-902df25464e9 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:12:11 user cinder-volume[84696]: WARNING cinder.volume.manager [req-008ec7fb-052b-43ae-952b-3c2d087ab270 req-15d5486a-4efe-4884-8b8c-05a078b8fa5f tempest-ServerActionsTestJSON-645547919 None] Failed to detach volume on the backend, ignoring failure 'NoneType' object is not subscriptable Apr 17 16:12:14 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Volume reschedule parameters: True retry: {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 16:12:14 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Flow 'volume_create_manager' (b2538276-308a-4b31-bb9e-95fe32632134) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 16:12:14 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (12dd2903-492e-4ff3-a822-b7b3de012387) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:12:14 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (12dd2903-492e-4ff3-a822-b7b3de012387) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:12:14Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesTransfersV355Test-Volume-94958465',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=125cc0cd-98dc-41a8-a252-3ae03797fb6f,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='9796cc8ccc334a5b86ff098b10a0c413',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:12:14Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:12:14Z,use_quota=True,user_id='de05a598447a4e9fb3ff488ec2d84071',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:12:14 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (b576dc36-ce9c-49c5-adf5-5b1f237b943a) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:12:14 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (b576dc36-ce9c-49c5-adf5-5b1f237b943a) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:12:14 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (6ff0458b-6371-467e-bf79-3c4e4a6cfded) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:12:14 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (6ff0458b-6371-467e-bf79-3c4e4a6cfded) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': '125cc0cd-98dc-41a8-a252-3ae03797fb6f', 'volume_name': 'volume-125cc0cd-98dc-41a8-a252-3ae03797fb6f', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:12:14 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (0f45186b-cd3b-42e6-9c64-16d8f0bfbf38) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:12:14 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (0f45186b-cd3b-42e6-9c64-16d8f0bfbf38) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:12:14 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (a9526e95-913f-468c-ad38-84c3b8a1adea) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:12:14 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Volume 125cc0cd-98dc-41a8-a252-3ae03797fb6f: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-125cc0cd-98dc-41a8-a252-3ae03797fb6f', 'volume_size': 1} Apr 17 16:12:14 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] creating volume v_125cc0cd-98dc-41a8-a252-3ae03797fb6f. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 16:12:14 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] create volume {'name': 'v_125cc0cd-98dc-41a8-a252-3ae03797fb6f', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 16:12:14 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:12:14 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:12:15 user cinder-volume[84696]: DEBUG cinder.utils [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 0.917(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:12:15 user cinder-volume[84696]: DEBUG cinder.utils [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:12:15 user cinder-volume[84696]: DEBUG cinder.utils [req-008ec7fb-052b-43ae-952b-3c2d087ab270 req-625dc9c1-5166-463d-b990-0b1f19b50106 tempest-ServerActionsTestJSON-645547919 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 18.451(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:12:15 user cinder-volume[84696]: DEBUG cinder.utils [req-008ec7fb-052b-43ae-952b-3c2d087ab270 req-625dc9c1-5166-463d-b990-0b1f19b50106 tempest-ServerActionsTestJSON-645547919 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:1072d05f-2ebb-4398-902c-902df25464e9/luns/v_1072d05f-2ebb-4398-902c-902df25464e9 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:12:18 user cinder-volume[84696]: DEBUG cinder.utils [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 20.575(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:12:18 user cinder-volume[84696]: DEBUG cinder.utils [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:12:18 user cinder-volume[84696]: DEBUG cinder.utils [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 3.990(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:12:18 user cinder-volume[84696]: DEBUG cinder.utils [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:12:21 user cinder-volume[84696]: DEBUG cinder.utils [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 136.748(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:12:21 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_4d960aac-48dd-470e-831f-eaafe64c9fdf/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:12:21 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (0c9c6108-3ef2-48ba-8fbd-83c6bc609fe4) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 16:12:21 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (0212d309-5066-4b3c-aee3-0e008e1bcfa3) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:12:21 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (0212d309-5066-4b3c-aee3-0e008e1bcfa3) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:12:21 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Flow 'volume_create_manager' (4836f35e-faae-495a-9e9f-3b924cbf44b9) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 16:12:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-2f261810-f79c-48b0-b0ef-2727c4b52668 tempest-VolumesListTestJSON-659728137 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 16:12:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 16:12:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 16:12:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 16:12:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 16:12:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 16:12:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 16:12:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 16:12:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 16:12:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 16:12:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 16:12:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 16:12:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 16:12:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 16:12:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 16:12:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 16:12:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 16:12:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 16:12:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 16:12:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 16:12:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 16:12:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 16:12:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 16:12:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 16:12:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:12:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 16:12:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:12:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 16:12:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:12:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 16:12:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:12:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 16:12:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:12:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 16:12:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:12:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 16:12:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:12:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 16:12:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:12:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:12:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 16:12:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 16:12:24 user cinder-volume[84696]: DEBUG cinder.utils [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 10.325(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:12:24 user cinder-volume[84696]: DEBUG cinder.utils [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:12:24 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=14,cinder:UPDATE=3 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:12:24 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=11,cinder:UPDATE=2 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:12:31 user cinder-volume[84696]: DEBUG cinder.utils [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 73.109(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:12:31 user cinder-volume[84696]: DEBUG cinder.utils [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:12:31 user cinder-volume[84696]: DEBUG cinder.utils [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 74.695(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:12:31 user cinder-volume[84696]: DEBUG cinder.utils [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_49effcda-e787-4983-92b2-f8261052732d/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:12:31 user cinder-volume[84696]: DEBUG cinder.utils [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 74.500(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:12:31 user cinder-volume[84696]: DEBUG cinder.utils [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:12:31 user cinder-volume[84696]: DEBUG cinder.utils [req-008ec7fb-052b-43ae-952b-3c2d087ab270 req-625dc9c1-5166-463d-b990-0b1f19b50106 tempest-ServerActionsTestJSON-645547919 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 34.873(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:12:31 user cinder-volume[84696]: DEBUG cinder.utils [req-008ec7fb-052b-43ae-952b-3c2d087ab270 req-625dc9c1-5166-463d-b990-0b1f19b50106 tempest-ServerActionsTestJSON-645547919 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:1072d05f-2ebb-4398-902c-902df25464e9/luns/v_1072d05f-2ebb-4398-902c-902df25464e9 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:12:34 user cinder-volume[84696]: DEBUG cinder.utils [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 36.991(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:12:34 user cinder-volume[84696]: DEBUG cinder.utils [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:12:34 user cinder-volume[84696]: DEBUG cinder.utils [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 20.406(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:12:34 user cinder-volume[84696]: DEBUG cinder.utils [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:12:37 user cinder-volume[84696]: DEBUG cinder.utils [req-bb0706f3-0b2c-473a-acc5-93c2ac742a5f req-c78e09d4-3679-4ee2-ab6c-f8158e1aaa1d tempest-ServerStableDeviceRescueTest-869290889 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 73.315(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:12:37 user cinder-volume[84696]: DEBUG cinder.utils [req-bb0706f3-0b2c-473a-acc5-93c2ac742a5f req-c78e09d4-3679-4ee2-ab6c-f8158e1aaa1d tempest-ServerStableDeviceRescueTest-869290889 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:32819ec1-2236-465f-a1c7-ee31aa4a0911/luns/v_32819ec1-2236-465f-a1c7-ee31aa4a0911 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:12:40 user cinder-volume[84696]: DEBUG cinder.utils [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 72.046(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:12:40 user cinder-volume[84696]: DEBUG cinder.utils [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:12:44 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=2,cinder:UPDATE=2 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:12:53 user cinder-volume[84696]: DEBUG cinder.utils [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.414(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:12:53 user cinder-volume[84696]: DEBUG cinder.utils [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:12:53 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 140.191(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:12:53 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:12:53 user cinder-volume[84696]: ERROR oslo_service.periodic_task [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Error during VolumeManager.publish_service_capabilities: TypeError: 'NoneType' object is not subscriptable Apr 17 16:12:53 user cinder-volume[84696]: ERROR oslo_service.periodic_task Traceback (most recent call last): Apr 17 16:12:53 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/usr/local/lib/python3.10/dist-packages/oslo_service/periodic_task.py", line 216, in run_periodic_tasks Apr 17 16:12:53 user cinder-volume[84696]: ERROR oslo_service.periodic_task task(self, context) Apr 17 16:12:53 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/manager.py", line 2876, in publish_service_capabilities Apr 17 16:12:53 user cinder-volume[84696]: ERROR oslo_service.periodic_task self._report_driver_status(context) Apr 17 16:12:53 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/manager.py", line 2755, in _report_driver_status Apr 17 16:12:53 user cinder-volume[84696]: ERROR oslo_service.periodic_task volume_stats = get_stats() Apr 17 16:12:53 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/usr/local/lib/python3.10/dist-packages/oslo_utils/timeutils.py", line 319, in wrapper Apr 17 16:12:53 user cinder-volume[84696]: ERROR oslo_service.periodic_task result = func(*args, **kwargs) Apr 17 16:12:53 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/manager.py", line 2753, in get_stats Apr 17 16:12:53 user cinder-volume[84696]: ERROR oslo_service.periodic_task return self.driver.get_volume_stats(refresh=True) Apr 17 16:12:53 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/driver.py", line 666, in get_volume_stats Apr 17 16:12:53 user cinder-volume[84696]: ERROR oslo_service.periodic_task self._update_volume_stats() Apr 17 16:12:53 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 726, in _update_volume_stats Apr 17 16:12:53 user cinder-volume[84696]: ERROR oslo_service.periodic_task pool_stats = self.ra.get_pool_stats() Apr 17 16:12:53 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 897, in get_pool_stats Apr 17 16:12:53 user cinder-volume[84696]: ERROR oslo_service.periodic_task if not resp["error"] and resp["code"] == 200: Apr 17 16:12:53 user cinder-volume[84696]: ERROR oslo_service.periodic_task TypeError: 'NoneType' object is not subscriptable Apr 17 16:12:53 user cinder-volume[84696]: ERROR oslo_service.periodic_task Apr 17 16:12:53 user cinder-volume[84696]: WARNING oslo.service.loopingcall [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Function 'cinder.service.Service.periodic_tasks' run outlasted interval by 80.20 sec Apr 17 16:12:53 user cinder-volume[84696]: DEBUG oslo_service.periodic_task [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Running periodic task VolumeManager.publish_service_capabilities {{(pid=84696) run_periodic_tasks /usr/local/lib/python3.10/dist-packages/oslo_service/periodic_task.py:210}} Apr 17 16:12:53 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Updating volume stats {{(pid=84696) _update_volume_stats /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:724}} Apr 17 16:12:53 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Get pool Pool-0 fsprops {{(pid=84696) get_pool_stats /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:894}} Apr 17 16:12:53 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:12:53 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:12:54 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=5,cinder:UPDATE=1 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:12:56 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 3.060(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:12:56 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:12:56 user cinder-volume[84696]: DEBUG cinder.utils [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 139.917(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:12:56 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:12:56 user cinder-volume[84696]: INFO cinder.message.api [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Creating message record for request_id = req-251ec93b-05ae-4736-ab3c-338a5d9efb63 Apr 17 16:12:56 user cinder-volume[84696]: DEBUG cinder.utils [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 140.094(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:12:56 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_65a642c3-b1e1-4b80-9bd2-f00151b6304a/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:12:56 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (8e18a790-1ba9-497b-a9bd-8edd30a199fa) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 16:12:56 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (8257a320-a94c-48aa-88f3-d98a64bd5017) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:12:56 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (8257a320-a94c-48aa-88f3-d98a64bd5017) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:12:56 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Flow 'volume_create_manager' (c7250c61-21e3-4f2f-87b2-a4e7a77f94cf) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 16:12:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 16:12:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 16:12:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 16:12:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 16:12:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 16:12:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 16:12:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 16:12:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 16:12:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 16:12:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 16:12:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 16:12:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 16:12:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 16:12:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 16:12:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 16:12:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 16:12:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 16:12:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 16:12:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 16:12:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 16:12:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 16:12:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 16:12:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 16:12:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 16:12:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:12:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 16:12:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:12:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 16:12:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:12:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 16:12:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:12:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 16:12:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:12:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 16:12:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:12:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 16:12:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:12:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 16:12:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:12:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:12:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 16:12:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 16:12:56 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 16:12:56 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (97d3fe9c-7f5d-49bb-b63c-18f72dd43d16) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 16:12:56 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 16:12:56 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:05:55Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-AttachVolumeTestJSON-volume-1580982693',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=65a642c3-b1e1-4b80-9bd2-f00151b6304a,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='2eb998c84b5b41e599af9636b929295d',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:10:37Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:10:37Z,use_quota=True,user_id='b9d4a317838b47d6a2a2e7eab7a453dd',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 16:12:56 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:05:55Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-AttachVolumeTestJSON-volume-1580982693',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=65a642c3-b1e1-4b80-9bd2-f00151b6304a,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='2eb998c84b5b41e599af9636b929295d',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:10:37Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:10:37Z,use_quota=True,user_id='b9d4a317838b47d6a2a2e7eab7a453dd',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(65a642c3-b1e1-4b80-9bd2-f00151b6304a),volume_id=65a642c3-b1e1-4b80-9bd2-f00151b6304a,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': '65a642c3-b1e1-4b80-9bd2-f00151b6304a', 'volume_name': 'volume-65a642c3-b1e1-4b80-9bd2-f00151b6304a', 'volume_size': 1}} Apr 17 16:12:56 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': '65a642c3-b1e1-4b80-9bd2-f00151b6304a', 'volume': {'id': '65a642c3-b1e1-4b80-9bd2-f00151b6304a', '_name_id': None, 'ec2_id': None, 'user_id': 'b9d4a317838b47d6a2a2e7eab7a453dd', 'project_id': '2eb998c84b5b41e599af9636b929295d', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-AttachVolumeTestJSON-volume-1580982693', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'admin_metadata': {}, 'glance_metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T16:05:55.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-65a642c3-b1e1-4b80-9bd2-f00151b6304a', 'name_id': '65a642c3-b1e1-4b80-9bd2-f00151b6304a', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-AttachVolumeTestJSON-volume-1580982693', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '2eb998c84b5b41e599af9636b929295d', 'qos_specs': None, 'reservations': ['1201d95e-040b-4401-9092-4910db00f9ec', '61d42360-766f-47d3-8707-d968887b405f', '54625198-5e49-42b5-a0f1-c764a5a3aaeb', '917c67ca-7481-4181-951b-961fc40b1138'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'b9d4a317838b47d6a2a2e7eab7a453dd', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-AttachVolumeTestJSON-volume-1580982693', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '2eb998c84b5b41e599af9636b929295d', 'qos_specs': None, 'reservations': ['1201d95e-040b-4401-9092-4910db00f9ec', '61d42360-766f-47d3-8707-d968887b405f', '54625198-5e49-42b5-a0f1-c764a5a3aaeb', '917c67ca-7481-4181-951b-961fc40b1138'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'b9d4a317838b47d6a2a2e7eab7a453dd', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': 'b9d4a317838b47d6a2a2e7eab7a453dd', 'metadata': {}, 'qos_specs': None, 'retry': {'num_attempts': 3, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']}}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(65a642c3-b1e1-4b80-9bd2-f00151b6304a),volume_id=65a642c3-b1e1-4b80-9bd2-f00151b6304a,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:05:55Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-AttachVolumeTestJSON-volume-1580982693',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=65a642c3-b1e1-4b80-9bd2-f00151b6304a,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='2eb998c84b5b41e599af9636b929295d',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:10:37Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:10:37Z,use_quota=True,user_id='b9d4a317838b47d6a2a2e7eab7a453dd',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 16:12:56 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:05:55Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-AttachVolumeTestJSON-volume-1580982693',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=65a642c3-b1e1-4b80-9bd2-f00151b6304a,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='2eb998c84b5b41e599af9636b929295d',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:10:37Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:10:37Z,use_quota=True,user_id='b9d4a317838b47d6a2a2e7eab7a453dd',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:05:55Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-AttachVolumeTestJSON-volume-1580982693',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=65a642c3-b1e1-4b80-9bd2-f00151b6304a,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='2eb998c84b5b41e599af9636b929295d',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:10:37Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:10:37Z,use_quota=True,user_id='b9d4a317838b47d6a2a2e7eab7a453dd',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 16:12:56 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 16:12:56 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 16:12:56 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:12:56 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 16:12:56 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:12:56 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 16:12:56 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:12:56 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 16:12:56 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:12:56 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 16:12:56 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:12:56 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 16:12:56 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:12:56 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 16:12:56 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:12:56 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 16:12:56 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:12:56 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:12:56 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 16:12:56 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 16:12:56 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (97d3fe9c-7f5d-49bb-b63c-18f72dd43d16) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:12:56 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (97d3fe9c-7f5d-49bb-b63c-18f72dd43d16) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:12:56 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (1631cbb1-e31d-4ba1-9964-bc84509c09a7) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:12:56 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (1631cbb1-e31d-4ba1-9964-bc84509c09a7) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:12:56 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (00e215bb-de2b-4c68-9321-ea88043a137f) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:12:56 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (00e215bb-de2b-4c68-9321-ea88043a137f) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:12:56 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (23d34cdc-882e-4b5d-abc8-2a3d8ec2a97f) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:12:56 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Updating volume 65a642c3-b1e1-4b80-9bd2-f00151b6304a with {'scheduled_at': datetime.datetime(2023, 4, 17, 16, 12, 56, 676904), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 16:12:56 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Volume 65a642c3-b1e1-4b80-9bd2-f00151b6304a: re-scheduling SchedulerAPI.create_volume attempt 3 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 16:12:56 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Volume 65a642c3-b1e1-4b80-9bd2-f00151b6304a: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 16:12:56 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] deleating volume v_65a642c3-b1e1-4b80-9bd2-f00151b6304a {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 16:12:56 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] get snapshots for volume v_65a642c3-b1e1-4b80-9bd2-f00151b6304a {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 16:12:56 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_65a642c3-b1e1-4b80-9bd2-f00151b6304a/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:12:56 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_65a642c3-b1e1-4b80-9bd2-f00151b6304a/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:12:59 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.132(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:12:59 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:12:59 user cinder-volume[84696]: DEBUG cinder.utils [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.939(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:12:59 user cinder-volume[84696]: DEBUG cinder.utils [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_65a642c3-b1e1-4b80-9bd2-f00151b6304a/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:13:02 user cinder-volume[84696]: DEBUG cinder.utils [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.013(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:13:02 user cinder-volume[84696]: DEBUG cinder.utils [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_65a642c3-b1e1-4b80-9bd2-f00151b6304a/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:13:05 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.276(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:13:05 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:13:05 user cinder-volume[84696]: DEBUG cinder.utils [req-008ec7fb-052b-43ae-952b-3c2d087ab270 req-625dc9c1-5166-463d-b990-0b1f19b50106 tempest-ServerActionsTestJSON-645547919 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 69.234(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:13:05 user cinder-volume[84696]: DEBUG cinder.utils [req-008ec7fb-052b-43ae-952b-3c2d087ab270 req-625dc9c1-5166-463d-b990-0b1f19b50106 tempest-ServerActionsTestJSON-645547919 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:1072d05f-2ebb-4398-902c-902df25464e9/luns/v_1072d05f-2ebb-4398-902c-902df25464e9 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:13:08 user cinder-volume[84696]: DEBUG cinder.utils [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 71.363(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:13:08 user cinder-volume[84696]: DEBUG cinder.utils [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:13:08 user cinder-volume[84696]: DEBUG cinder.utils [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.160(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:13:08 user cinder-volume[84696]: DEBUG cinder.utils [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_65a642c3-b1e1-4b80-9bd2-f00151b6304a/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:13:14 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=6,cinder:UPDATE=2,cinder:INSERT=1 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:13:15 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.940(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:13:15 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:13:18 user cinder-volume[84696]: DEBUG cinder.utils [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.818(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:13:18 user cinder-volume[84696]: DEBUG cinder.utils [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_65a642c3-b1e1-4b80-9bd2-f00151b6304a/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:13:24 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=7,cinder:INSERT=1,cinder:UPDATE=3 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:13:24 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-bb0706f3-0b2c-473a-acc5-93c2ac742a5f req-f2b51132-0b2f-4350-b59e-e65a5d094af4 tempest-ServerStableDeviceRescueTest-869290889 None] terminate connection for 32819ec1-2236-465f-a1c7-ee31aa4a0911 {{(pid=84696) terminate_connection /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:1032}} Apr 17 16:13:24 user cinder-volume[84696]: INFO cinder.volume.manager [req-bb0706f3-0b2c-473a-acc5-93c2ac742a5f req-f2b51132-0b2f-4350-b59e-e65a5d094af4 tempest-ServerStableDeviceRescueTest-869290889 None] Terminate volume connection completed successfully. Apr 17 16:13:24 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-bb0706f3-0b2c-473a-acc5-93c2ac742a5f req-f2b51132-0b2f-4350-b59e-e65a5d094af4 tempest-ServerStableDeviceRescueTest-869290889 None] Deleting attachment fb44dce3-1af2-43cf-95fa-68ff027e0699. {{(pid=84696) attachment_delete /opt/stack/cinder/cinder/volume/manager.py:5006}} Apr 17 16:13:24 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-bb0706f3-0b2c-473a-acc5-93c2ac742a5f req-f2b51132-0b2f-4350-b59e-e65a5d094af4 tempest-ServerStableDeviceRescueTest-869290889 None] remove_export for volume: 32819ec1-2236-465f-a1c7-ee31aa4a0911. {{(pid=84696) remove_export /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:718}} Apr 17 16:13:24 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-bb0706f3-0b2c-473a-acc5-93c2ac742a5f req-f2b51132-0b2f-4350-b59e-e65a5d094af4 tempest-ServerStableDeviceRescueTest-869290889 None] remove export {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:912}} Apr 17 16:13:24 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-bb0706f3-0b2c-473a-acc5-93c2ac742a5f req-f2b51132-0b2f-4350-b59e-e65a5d094af4 tempest-ServerStableDeviceRescueTest-869290889 None] detach volume:Volume(_name_id=None,admin_metadata=,attach_status='attached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:27:42Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-ServerStableDeviceRescueTest-volume-648918308',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=32819ec1-2236-465f-a1c7-ee31aa4a0911,launched_at=2023-04-17T15:27:46Z,metadata=,migration_status=None,multiattach=False,previous_status=None,project_id='6bdfb8642b9341f6ba289e920c33cf47',provider_auth='CHAP CmhLYVJy GWkyzUl8QJxnjY',provider_geometry=None,provider_id=None,provider_location='172.16.0.220:3260,1 iqn.2010-10.org.openstack:32819ec1-2236-465f-a1c7-ee31aa4a0911 0',replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:27:43Z,service_uuid='e628352e-c4cf-40eb-8389-cce40663026a',shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='in-use',terminated_at=None,updated_at=2023-04-17T15:27:57Z,use_quota=True,user_id='a972c04314724233bc8e33b64747fecf',volume_attachment=,volume_type=,volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c) from target:iqn.2010-10.org.openstack:32819ec1-2236-465f-a1c7-ee31aa4a0911. {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:913}} Apr 17 16:13:24 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [req-bb0706f3-0b2c-473a-acc5-93c2ac742a5f req-f2b51132-0b2f-4350-b59e-e65a5d094af4 tempest-ServerStableDeviceRescueTest-869290889 None] detach volume v_32819ec1-2236-465f-a1c7-ee31aa4a0911 from target iqn.2010-10.org.openstack:32819ec1-2236-465f-a1c7-ee31aa4a0911 {{(pid=84696) detach_target_vol /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:606}} Apr 17 16:13:24 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-bb0706f3-0b2c-473a-acc5-93c2ac742a5f req-f2b51132-0b2f-4350-b59e-e65a5d094af4 tempest-ServerStableDeviceRescueTest-869290889 None] Sending pool request DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:32819ec1-2236-465f-a1c7-ee31aa4a0911/luns/v_32819ec1-2236-465f-a1c7-ee31aa4a0911 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:13:24 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-bb0706f3-0b2c-473a-acc5-93c2ac742a5f req-f2b51132-0b2f-4350-b59e-e65a5d094af4 tempest-ServerStableDeviceRescueTest-869290889 None] Sending DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:32819ec1-2236-465f-a1c7-ee31aa4a0911/luns/v_32819ec1-2236-465f-a1c7-ee31aa4a0911 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:13:24 user cinder-volume[84696]: DEBUG cinder.utils [req-bb0706f3-0b2c-473a-acc5-93c2ac742a5f req-f2b51132-0b2f-4350-b59e-e65a5d094af4 tempest-ServerStableDeviceRescueTest-869290889 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 0.029(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:13:24 user cinder-volume[84696]: DEBUG cinder.utils [req-bb0706f3-0b2c-473a-acc5-93c2ac742a5f req-f2b51132-0b2f-4350-b59e-e65a5d094af4 tempest-ServerStableDeviceRescueTest-869290889 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:32819ec1-2236-465f-a1c7-ee31aa4a0911/luns/v_32819ec1-2236-465f-a1c7-ee31aa4a0911 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:13:28 user cinder-volume[84696]: DEBUG cinder.utils [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 74.361(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:13:28 user cinder-volume[84696]: DEBUG cinder.utils [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:13:28 user cinder-volume[84696]: DEBUG cinder.utils [req-bb0706f3-0b2c-473a-acc5-93c2ac742a5f req-f2b51132-0b2f-4350-b59e-e65a5d094af4 tempest-ServerStableDeviceRescueTest-869290889 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 3.811(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:13:28 user cinder-volume[84696]: DEBUG cinder.utils [req-bb0706f3-0b2c-473a-acc5-93c2ac742a5f req-f2b51132-0b2f-4350-b59e-e65a5d094af4 tempest-ServerStableDeviceRescueTest-869290889 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:32819ec1-2236-465f-a1c7-ee31aa4a0911/luns/v_32819ec1-2236-465f-a1c7-ee31aa4a0911 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:13:31 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 38.004(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:13:31 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:13:35 user cinder-volume[84696]: DEBUG cinder.utils [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 137.141(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:13:35 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:13:35 user cinder-volume[84696]: INFO cinder.message.api [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Creating message record for request_id = req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 Apr 17 16:13:35 user cinder-volume[84696]: DEBUG cinder.utils [req-bb0706f3-0b2c-473a-acc5-93c2ac742a5f req-f2b51132-0b2f-4350-b59e-e65a5d094af4 tempest-ServerStableDeviceRescueTest-869290889 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 10.820(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:13:35 user cinder-volume[84696]: DEBUG cinder.utils [req-bb0706f3-0b2c-473a-acc5-93c2ac742a5f req-f2b51132-0b2f-4350-b59e-e65a5d094af4 tempest-ServerStableDeviceRescueTest-869290889 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:32819ec1-2236-465f-a1c7-ee31aa4a0911/luns/v_32819ec1-2236-465f-a1c7-ee31aa4a0911 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:13:35 user cinder-volume[84696]: DEBUG cinder.utils [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 138.729(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:13:35 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_49effcda-e787-4983-92b2-f8261052732d/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:13:35 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (85e8bd62-0b6e-4718-b1b6-4e9d77ee4eeb) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 16:13:35 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (78aba441-05df-4a59-bfb0-daad5876261a) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:13:35 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (78aba441-05df-4a59-bfb0-daad5876261a) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:13:35 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Flow 'volume_create_manager' (2bedb8fd-4b3a-49a4-aa70-0a68a6e5f424) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 16:13:35 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 16:13:35 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 16:13:35 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 16:13:35 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 16:13:35 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 16:13:35 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 16:13:35 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 16:13:35 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 16:13:35 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 16:13:35 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 16:13:35 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 16:13:35 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 16:13:35 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 16:13:35 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 16:13:35 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 16:13:35 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 16:13:35 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 16:13:35 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 16:13:35 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 16:13:35 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 16:13:35 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 16:13:35 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 16:13:35 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 16:13:35 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 16:13:35 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:13:35 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 16:13:35 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:13:35 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 16:13:35 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:13:35 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 16:13:35 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:13:35 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 16:13:35 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:13:35 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 16:13:35 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:13:35 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 16:13:35 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:13:35 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 16:13:35 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:13:35 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:13:35 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 16:13:35 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 16:13:35 user cinder-volume[84696]: DEBUG cinder.utils [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 38.771(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:13:35 user cinder-volume[84696]: DEBUG cinder.utils [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_65a642c3-b1e1-4b80-9bd2-f00151b6304a/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:13:35 user cinder-volume[84696]: DEBUG cinder.utils [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 138.551(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:13:35 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:13:35 user cinder-volume[84696]: INFO cinder.message.api [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Creating message record for request_id = req-40e48e44-d46b-4e9d-82c5-3b795728a90d Apr 17 16:13:35 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 16:13:35 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 16:13:35 user cinder-volume[84696]: WARNING cinder.volume.manager [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (82fdf2da-94ae-4259-9a5e-f30f2727b449) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 16:13:35 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 16:13:35 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:11:18Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesGetTestJSON-Volume-1455502637',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=0a7d4209-92e0-4900-8e98-36a62e68c247,launched_at=None,metadata={Type='work'},migration_status=None,multiattach=False,previous_status=None,project_id='e330412972fa44b9bf60cd9afe4399cd',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:11:18Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:11:18Z,use_quota=True,user_id='e14688d19f1b44928fa8aceb2da0ac82',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 16:13:35 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:11:18Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesGetTestJSON-Volume-1455502637',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=0a7d4209-92e0-4900-8e98-36a62e68c247,launched_at=None,metadata={Type='work'},migration_status=None,multiattach=False,previous_status=None,project_id='e330412972fa44b9bf60cd9afe4399cd',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:11:18Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:11:18Z,use_quota=True,user_id='e14688d19f1b44928fa8aceb2da0ac82',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(0a7d4209-92e0-4900-8e98-36a62e68c247),volume_id=0a7d4209-92e0-4900-8e98-36a62e68c247,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': '0a7d4209-92e0-4900-8e98-36a62e68c247', 'volume_name': 'volume-0a7d4209-92e0-4900-8e98-36a62e68c247', 'volume_size': 1}} Apr 17 16:13:35 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'retry': {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']}, 'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': '0a7d4209-92e0-4900-8e98-36a62e68c247', 'volume': {'id': '0a7d4209-92e0-4900-8e98-36a62e68c247', '_name_id': None, 'ec2_id': None, 'user_id': 'e14688d19f1b44928fa8aceb2da0ac82', 'project_id': 'e330412972fa44b9bf60cd9afe4399cd', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-VolumesGetTestJSON-Volume-1455502637', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {'Type': 'work'}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T16:11:17.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-0a7d4209-92e0-4900-8e98-36a62e68c247', 'name_id': '0a7d4209-92e0-4900-8e98-36a62e68c247', 'volume_metadata': [{'key': 'Type', 'value': 'work'}], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumesGetTestJSON-Volume-1455502637', 'encryption_key_id': None, 'metadata': {'Type': 'work'}, 'multiattach': False, 'project_id': 'e330412972fa44b9bf60cd9afe4399cd', 'qos_specs': None, 'reservations': ['a1d7a4e6-78b5-4539-8c86-495dc07de36d', '0ea4500b-e7dd-4705-a54f-22c9b06155b5', 'b2cf1ff4-8bad-445f-8b8b-6ebee58654f1', '9aeb8f74-fd45-412b-b0b4-1970c607402d'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'e14688d19f1b44928fa8aceb2da0ac82', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumesGetTestJSON-Volume-1455502637', 'encryption_key_id': None, 'metadata': {'Type': 'work'}, 'multiattach': False, 'project_id': 'e330412972fa44b9bf60cd9afe4399cd', 'qos_specs': None, 'reservations': ['a1d7a4e6-78b5-4539-8c86-495dc07de36d', '0ea4500b-e7dd-4705-a54f-22c9b06155b5', 'b2cf1ff4-8bad-445f-8b8b-6ebee58654f1', '9aeb8f74-fd45-412b-b0b4-1970c607402d'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'e14688d19f1b44928fa8aceb2da0ac82', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': 'e14688d19f1b44928fa8aceb2da0ac82', 'metadata': {'Type': 'work'}, 'qos_specs': None}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(0a7d4209-92e0-4900-8e98-36a62e68c247),volume_id=0a7d4209-92e0-4900-8e98-36a62e68c247,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:11:18Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesGetTestJSON-Volume-1455502637',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=0a7d4209-92e0-4900-8e98-36a62e68c247,launched_at=None,metadata={Type='work'},migration_status=None,multiattach=False,previous_status=None,project_id='e330412972fa44b9bf60cd9afe4399cd',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:11:18Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:11:18Z,use_quota=True,user_id='e14688d19f1b44928fa8aceb2da0ac82',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 16:13:35 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:11:18Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesGetTestJSON-Volume-1455502637',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=0a7d4209-92e0-4900-8e98-36a62e68c247,launched_at=None,metadata={Type='work'},migration_status=None,multiattach=False,previous_status=None,project_id='e330412972fa44b9bf60cd9afe4399cd',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:11:18Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:11:18Z,use_quota=True,user_id='e14688d19f1b44928fa8aceb2da0ac82',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:11:18Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesGetTestJSON-Volume-1455502637',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=0a7d4209-92e0-4900-8e98-36a62e68c247,launched_at=None,metadata={Type='work'},migration_status=None,multiattach=False,previous_status=None,project_id='e330412972fa44b9bf60cd9afe4399cd',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:11:18Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:11:18Z,use_quota=True,user_id='e14688d19f1b44928fa8aceb2da0ac82',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 16:13:35 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 16:13:35 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 16:13:35 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:13:35 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 16:13:35 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:13:35 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 16:13:35 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:13:35 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 16:13:35 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:13:35 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 16:13:35 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:13:35 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 16:13:35 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:13:35 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 16:13:35 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:13:35 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 16:13:35 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:13:35 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:13:35 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 16:13:35 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 16:13:35 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (82fdf2da-94ae-4259-9a5e-f30f2727b449) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:13:35 user cinder-volume[84696]: WARNING cinder.volume.manager [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (82fdf2da-94ae-4259-9a5e-f30f2727b449) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:13:35 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (720939a1-4205-4e33-893c-eeeaf74c03ee) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:13:35 user cinder-volume[84696]: WARNING cinder.volume.manager [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (720939a1-4205-4e33-893c-eeeaf74c03ee) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:13:35 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (6dbd5de6-65ed-4d90-a87c-0913b47bbfc5) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:13:35 user cinder-volume[84696]: WARNING cinder.volume.manager [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (6dbd5de6-65ed-4d90-a87c-0913b47bbfc5) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:13:35 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (c14008a8-e359-4437-b365-01be3b46be87) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:13:35 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Updating volume 0a7d4209-92e0-4900-8e98-36a62e68c247 with {'scheduled_at': datetime.datetime(2023, 4, 17, 16, 13, 35, 539747), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 16:13:35 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (a1c24603-84f5-4df0-9bcf-c5809e299c36) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 16:13:35 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 16:13:35 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:08:58Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesTransfersV355Test-Volume-1916167074',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=49effcda-e787-4983-92b2-f8261052732d,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='9796cc8ccc334a5b86ff098b10a0c413',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:11:17Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:11:17Z,use_quota=True,user_id='de05a598447a4e9fb3ff488ec2d84071',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 16:13:35 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:08:58Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesTransfersV355Test-Volume-1916167074',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=49effcda-e787-4983-92b2-f8261052732d,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='9796cc8ccc334a5b86ff098b10a0c413',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:11:17Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:11:17Z,use_quota=True,user_id='de05a598447a4e9fb3ff488ec2d84071',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(49effcda-e787-4983-92b2-f8261052732d),volume_id=49effcda-e787-4983-92b2-f8261052732d,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': '49effcda-e787-4983-92b2-f8261052732d', 'volume_name': 'volume-49effcda-e787-4983-92b2-f8261052732d', 'volume_size': 1}} Apr 17 16:13:35 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': '49effcda-e787-4983-92b2-f8261052732d', 'volume': {'id': '49effcda-e787-4983-92b2-f8261052732d', '_name_id': None, 'ec2_id': None, 'user_id': 'de05a598447a4e9fb3ff488ec2d84071', 'project_id': '9796cc8ccc334a5b86ff098b10a0c413', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-VolumesTransfersV355Test-Volume-1916167074', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'admin_metadata': {}, 'glance_metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T16:08:57.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-49effcda-e787-4983-92b2-f8261052732d', 'name_id': '49effcda-e787-4983-92b2-f8261052732d', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumesTransfersV355Test-Volume-1916167074', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '9796cc8ccc334a5b86ff098b10a0c413', 'qos_specs': None, 'reservations': ['18f38527-7ebd-41b2-8296-efa6d1a6acd7', '6a781a35-53f4-4739-a5cc-a9986cdaa2fe', 'd632f456-4c05-4c56-84e4-431db2216088', '398f088d-a47c-4ff9-aee7-12eb5bef8d35'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'de05a598447a4e9fb3ff488ec2d84071', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumesTransfersV355Test-Volume-1916167074', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '9796cc8ccc334a5b86ff098b10a0c413', 'qos_specs': None, 'reservations': ['18f38527-7ebd-41b2-8296-efa6d1a6acd7', '6a781a35-53f4-4739-a5cc-a9986cdaa2fe', 'd632f456-4c05-4c56-84e4-431db2216088', '398f088d-a47c-4ff9-aee7-12eb5bef8d35'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'de05a598447a4e9fb3ff488ec2d84071', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': 'de05a598447a4e9fb3ff488ec2d84071', 'metadata': {}, 'qos_specs': None, 'retry': {'num_attempts': 2, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']}}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(49effcda-e787-4983-92b2-f8261052732d),volume_id=49effcda-e787-4983-92b2-f8261052732d,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:08:58Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesTransfersV355Test-Volume-1916167074',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=49effcda-e787-4983-92b2-f8261052732d,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='9796cc8ccc334a5b86ff098b10a0c413',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:11:17Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:11:17Z,use_quota=True,user_id='de05a598447a4e9fb3ff488ec2d84071',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 16:13:35 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:08:58Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesTransfersV355Test-Volume-1916167074',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=49effcda-e787-4983-92b2-f8261052732d,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='9796cc8ccc334a5b86ff098b10a0c413',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:11:17Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:11:17Z,use_quota=True,user_id='de05a598447a4e9fb3ff488ec2d84071',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:08:58Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesTransfersV355Test-Volume-1916167074',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=49effcda-e787-4983-92b2-f8261052732d,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='9796cc8ccc334a5b86ff098b10a0c413',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:11:17Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:11:17Z,use_quota=True,user_id='de05a598447a4e9fb3ff488ec2d84071',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 16:13:35 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 16:13:35 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 16:13:35 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:13:35 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 16:13:35 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:13:35 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 16:13:35 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:13:35 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 16:13:35 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:13:35 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 16:13:35 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:13:35 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 16:13:35 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:13:35 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 16:13:35 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:13:35 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 16:13:35 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:13:35 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:13:35 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 16:13:35 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 16:13:35 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (a1c24603-84f5-4df0-9bcf-c5809e299c36) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:13:35 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (a1c24603-84f5-4df0-9bcf-c5809e299c36) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:13:35 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (efc22af4-edd7-4f33-9025-66abbe62a0ed) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:13:35 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (efc22af4-edd7-4f33-9025-66abbe62a0ed) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:13:35 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (734bf18c-4925-40b1-aadc-d990a830512a) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:13:35 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (734bf18c-4925-40b1-aadc-d990a830512a) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:13:35 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (3d2ff43b-f7cd-4323-a7ca-708272500c69) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:13:35 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Updating volume 49effcda-e787-4983-92b2-f8261052732d with {'scheduled_at': datetime.datetime(2023, 4, 17, 16, 13, 35, 587780), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 16:13:35 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Volume 0a7d4209-92e0-4900-8e98-36a62e68c247: re-scheduling SchedulerAPI.create_volume attempt 1 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 16:13:35 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Volume 49effcda-e787-4983-92b2-f8261052732d: re-scheduling SchedulerAPI.create_volume attempt 2 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 16:13:35 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Volume 0a7d4209-92e0-4900-8e98-36a62e68c247: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 16:13:35 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] deleating volume v_0a7d4209-92e0-4900-8e98-36a62e68c247 {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 16:13:35 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] get snapshots for volume v_0a7d4209-92e0-4900-8e98-36a62e68c247 {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 16:13:35 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_0a7d4209-92e0-4900-8e98-36a62e68c247/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:13:35 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_0a7d4209-92e0-4900-8e98-36a62e68c247/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:13:35 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Volume 49effcda-e787-4983-92b2-f8261052732d: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 16:13:35 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] deleating volume v_49effcda-e787-4983-92b2-f8261052732d {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 16:13:35 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] get snapshots for volume v_49effcda-e787-4983-92b2-f8261052732d {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 16:13:35 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_49effcda-e787-4983-92b2-f8261052732d/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:13:35 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_49effcda-e787-4983-92b2-f8261052732d/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:13:35 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Volume reschedule parameters: True retry: {'num_attempts': 3, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 16:13:35 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Flow 'volume_create_manager' (dafcc11f-88ba-4457-a8aa-fb38e1e0bf03) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 16:13:35 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (220f3894-090d-4a96-875a-84d6149caebb) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:13:35 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Volume reschedule parameters: True retry: {'num_attempts': 2, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 16:13:35 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Flow 'volume_create_manager' (82856028-7c93-4c88-8de4-2c59742b7b78) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 16:13:35 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (9cf5485e-90e4-43ca-b95f-e75d7cad9b4b) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:13:35 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (220f3894-090d-4a96-875a-84d6149caebb) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:08:58Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesTransfersV355Test-Volume-1916167074',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=49effcda-e787-4983-92b2-f8261052732d,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='9796cc8ccc334a5b86ff098b10a0c413',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:13:36Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:13:36Z,use_quota=True,user_id='de05a598447a4e9fb3ff488ec2d84071',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:13:35 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (1c72d6cd-9242-492c-a056-e0f57367779c) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:13:35 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (1c72d6cd-9242-492c-a056-e0f57367779c) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:13:35 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (efd8d66c-639a-40e1-a8ec-3bb893b2f072) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:13:35 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (efd8d66c-639a-40e1-a8ec-3bb893b2f072) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': '49effcda-e787-4983-92b2-f8261052732d', 'volume_name': 'volume-49effcda-e787-4983-92b2-f8261052732d', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:13:35 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (abf9cb80-caa0-49b4-8eaa-eb33f8d35921) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:13:35 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (abf9cb80-caa0-49b4-8eaa-eb33f8d35921) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:13:35 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (ef69ec07-4798-4508-b77d-01a0b3ee438a) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:13:35 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Volume 49effcda-e787-4983-92b2-f8261052732d: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-49effcda-e787-4983-92b2-f8261052732d', 'volume_size': 1} Apr 17 16:13:35 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] creating volume v_49effcda-e787-4983-92b2-f8261052732d. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 16:13:35 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] create volume {'name': 'v_49effcda-e787-4983-92b2-f8261052732d', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 16:13:35 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:13:35 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:13:35 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (9cf5485e-90e4-43ca-b95f-e75d7cad9b4b) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:11:18Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesGetTestJSON-Volume-1455502637',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=0a7d4209-92e0-4900-8e98-36a62e68c247,launched_at=None,metadata={Type='work'},migration_status=None,multiattach=False,previous_status=None,project_id='e330412972fa44b9bf60cd9afe4399cd',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:13:36Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:13:36Z,use_quota=True,user_id='e14688d19f1b44928fa8aceb2da0ac82',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:13:35 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (19afd6eb-c20b-49ac-9713-0aa108e20290) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:13:35 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (19afd6eb-c20b-49ac-9713-0aa108e20290) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:13:35 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (c898e4fb-b6e1-4616-adde-a84aa62ee9bc) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:13:35 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (c898e4fb-b6e1-4616-adde-a84aa62ee9bc) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': '0a7d4209-92e0-4900-8e98-36a62e68c247', 'volume_name': 'volume-0a7d4209-92e0-4900-8e98-36a62e68c247', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:13:35 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (cea38384-7ce8-4fb9-9afc-f90b5758584d) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:13:35 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (cea38384-7ce8-4fb9-9afc-f90b5758584d) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:13:35 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (fbe14b45-ed1a-4d2c-b25b-03f9c9df58ae) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:13:35 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Volume 0a7d4209-92e0-4900-8e98-36a62e68c247: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-0a7d4209-92e0-4900-8e98-36a62e68c247', 'volume_size': 1} Apr 17 16:13:35 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] creating volume v_0a7d4209-92e0-4900-8e98-36a62e68c247. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 16:13:35 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] create volume {'name': 'v_0a7d4209-92e0-4900-8e98-36a62e68c247', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 16:13:35 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:13:35 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:13:38 user cinder-volume[84696]: DEBUG cinder.utils [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.906(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:13:38 user cinder-volume[84696]: DEBUG cinder.utils [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_0a7d4209-92e0-4900-8e98-36a62e68c247/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:13:38 user cinder-volume[84696]: DEBUG cinder.utils [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.646(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:13:38 user cinder-volume[84696]: DEBUG cinder.utils [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:13:38 user cinder-volume[84696]: DEBUG cinder.utils [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.909(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:13:38 user cinder-volume[84696]: DEBUG cinder.utils [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_49effcda-e787-4983-92b2-f8261052732d/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:13:38 user cinder-volume[84696]: DEBUG cinder.utils [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.566(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:13:38 user cinder-volume[84696]: DEBUG cinder.utils [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:13:41 user cinder-volume[84696]: DEBUG cinder.utils [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.982(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:13:41 user cinder-volume[84696]: DEBUG cinder.utils [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_0a7d4209-92e0-4900-8e98-36a62e68c247/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:13:41 user cinder-volume[84696]: DEBUG cinder.utils [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.720(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:13:41 user cinder-volume[84696]: DEBUG cinder.utils [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:13:41 user cinder-volume[84696]: DEBUG cinder.utils [req-bb0706f3-0b2c-473a-acc5-93c2ac742a5f req-c78e09d4-3679-4ee2-ab6c-f8158e1aaa1d tempest-ServerStableDeviceRescueTest-869290889 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 137.355(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:13:41 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-bb0706f3-0b2c-473a-acc5-93c2ac742a5f req-c78e09d4-3679-4ee2-ab6c-f8158e1aaa1d tempest-ServerStableDeviceRescueTest-869290889 None] Geting None from DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:32819ec1-2236-465f-a1c7-ee31aa4a0911/luns/v_32819ec1-2236-465f-a1c7-ee31aa4a0911 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:13:41 user cinder-volume[84696]: WARNING cinder.volume.manager [req-bb0706f3-0b2c-473a-acc5-93c2ac742a5f req-c78e09d4-3679-4ee2-ab6c-f8158e1aaa1d tempest-ServerStableDeviceRescueTest-869290889 None] Failed to detach volume on the backend, ignoring failure 'NoneType' object is not subscriptable Apr 17 16:13:41 user cinder-volume[84696]: DEBUG cinder.utils [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.641(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:13:41 user cinder-volume[84696]: DEBUG cinder.utils [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:13:41 user cinder-volume[84696]: DEBUG cinder.utils [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.986(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:13:41 user cinder-volume[84696]: DEBUG cinder.utils [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_49effcda-e787-4983-92b2-f8261052732d/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:13:44 user cinder-volume[84696]: DEBUG cinder.utils [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 136.078(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:13:44 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:13:44 user cinder-volume[84696]: INFO cinder.message.api [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Creating message record for request_id = req-726a1377-a691-4c8a-a56d-8b87aed706f7 Apr 17 16:13:44 user cinder-volume[84696]: DEBUG cinder.utils [req-bb0706f3-0b2c-473a-acc5-93c2ac742a5f req-f2b51132-0b2f-4350-b59e-e65a5d094af4 tempest-ServerStableDeviceRescueTest-869290889 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 20.033(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:13:44 user cinder-volume[84696]: DEBUG cinder.utils [req-bb0706f3-0b2c-473a-acc5-93c2ac742a5f req-f2b51132-0b2f-4350-b59e-e65a5d094af4 tempest-ServerStableDeviceRescueTest-869290889 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:32819ec1-2236-465f-a1c7-ee31aa4a0911/luns/v_32819ec1-2236-465f-a1c7-ee31aa4a0911 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:13:44 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 16:13:44 user cinder-volume[84696]: WARNING cinder.volume.manager [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (d347957b-4463-462d-ac29-65eaf5d8a189) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 16:13:44 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 16:13:44 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:11:28Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesSnapshotsTestJSON-volume-126016572',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=85fcb006-cd96-4018-b0e3-04872ff08267,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='34ab34dfa2e14c3785276a62a076f8e8',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:11:28Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:11:28Z,use_quota=True,user_id='c98520ee1d2c4eabaf6c186bd3ae7fee',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 16:13:44 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:11:28Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesSnapshotsTestJSON-volume-126016572',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=85fcb006-cd96-4018-b0e3-04872ff08267,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='34ab34dfa2e14c3785276a62a076f8e8',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:11:28Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:11:28Z,use_quota=True,user_id='c98520ee1d2c4eabaf6c186bd3ae7fee',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(85fcb006-cd96-4018-b0e3-04872ff08267),volume_id=85fcb006-cd96-4018-b0e3-04872ff08267,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': '85fcb006-cd96-4018-b0e3-04872ff08267', 'volume_name': 'volume-85fcb006-cd96-4018-b0e3-04872ff08267', 'volume_size': 1}} Apr 17 16:13:44 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'retry': {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']}, 'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': '85fcb006-cd96-4018-b0e3-04872ff08267', 'volume': {'id': '85fcb006-cd96-4018-b0e3-04872ff08267', '_name_id': None, 'ec2_id': None, 'user_id': 'c98520ee1d2c4eabaf6c186bd3ae7fee', 'project_id': '34ab34dfa2e14c3785276a62a076f8e8', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-VolumesSnapshotsTestJSON-volume-126016572', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T16:11:28.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-85fcb006-cd96-4018-b0e3-04872ff08267', 'name_id': '85fcb006-cd96-4018-b0e3-04872ff08267', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumesSnapshotsTestJSON-volume-126016572', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '34ab34dfa2e14c3785276a62a076f8e8', 'qos_specs': None, 'reservations': ['68a86b1f-7431-4aa0-8593-4b26c93f38d5', 'e3c2cd32-910b-402d-b512-12abf2966403', '3f9aa9b2-3b03-45b0-ab25-91483149a902', 'f37d20df-0298-46bc-8806-f2f882d834ac'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'c98520ee1d2c4eabaf6c186bd3ae7fee', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumesSnapshotsTestJSON-volume-126016572', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '34ab34dfa2e14c3785276a62a076f8e8', 'qos_specs': None, 'reservations': ['68a86b1f-7431-4aa0-8593-4b26c93f38d5', 'e3c2cd32-910b-402d-b512-12abf2966403', '3f9aa9b2-3b03-45b0-ab25-91483149a902', 'f37d20df-0298-46bc-8806-f2f882d834ac'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'c98520ee1d2c4eabaf6c186bd3ae7fee', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': 'c98520ee1d2c4eabaf6c186bd3ae7fee', 'metadata': {}, 'qos_specs': None}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(85fcb006-cd96-4018-b0e3-04872ff08267),volume_id=85fcb006-cd96-4018-b0e3-04872ff08267,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:11:28Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesSnapshotsTestJSON-volume-126016572',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=85fcb006-cd96-4018-b0e3-04872ff08267,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='34ab34dfa2e14c3785276a62a076f8e8',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:11:28Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:11:28Z,use_quota=True,user_id='c98520ee1d2c4eabaf6c186bd3ae7fee',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 16:13:44 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:11:28Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesSnapshotsTestJSON-volume-126016572',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=85fcb006-cd96-4018-b0e3-04872ff08267,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='34ab34dfa2e14c3785276a62a076f8e8',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:11:28Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:11:28Z,use_quota=True,user_id='c98520ee1d2c4eabaf6c186bd3ae7fee',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:11:28Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesSnapshotsTestJSON-volume-126016572',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=85fcb006-cd96-4018-b0e3-04872ff08267,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='34ab34dfa2e14c3785276a62a076f8e8',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:11:28Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:11:28Z,use_quota=True,user_id='c98520ee1d2c4eabaf6c186bd3ae7fee',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 16:13:44 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 16:13:44 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 16:13:44 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:13:44 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 16:13:44 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:13:44 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 16:13:44 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:13:44 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 16:13:44 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:13:44 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 16:13:44 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:13:44 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 16:13:44 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:13:44 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 16:13:44 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:13:44 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 16:13:44 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:13:44 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:13:44 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 16:13:44 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 16:13:44 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (d347957b-4463-462d-ac29-65eaf5d8a189) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:13:44 user cinder-volume[84696]: WARNING cinder.volume.manager [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (d347957b-4463-462d-ac29-65eaf5d8a189) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:13:44 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (ea60f5fe-e36e-419a-897b-8806e39051b6) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:13:44 user cinder-volume[84696]: WARNING cinder.volume.manager [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (ea60f5fe-e36e-419a-897b-8806e39051b6) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:13:44 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (b049a73b-c509-4dd8-a72c-6e48206f5c59) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:13:44 user cinder-volume[84696]: WARNING cinder.volume.manager [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (b049a73b-c509-4dd8-a72c-6e48206f5c59) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:13:44 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (eb867ed7-2620-4fdf-9e2c-6a76b6a3779a) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:13:44 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Updating volume 85fcb006-cd96-4018-b0e3-04872ff08267 with {'scheduled_at': datetime.datetime(2023, 4, 17, 16, 13, 44, 730063), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 16:13:44 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Volume 85fcb006-cd96-4018-b0e3-04872ff08267: re-scheduling SchedulerAPI.create_volume attempt 1 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 16:13:44 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Volume 85fcb006-cd96-4018-b0e3-04872ff08267: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 16:13:44 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] deleating volume v_85fcb006-cd96-4018-b0e3-04872ff08267 {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 16:13:44 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] get snapshots for volume v_85fcb006-cd96-4018-b0e3-04872ff08267 {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 16:13:44 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_85fcb006-cd96-4018-b0e3-04872ff08267/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:13:44 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_85fcb006-cd96-4018-b0e3-04872ff08267/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:13:44 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Volume reschedule parameters: True retry: {'num_attempts': 2, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 16:13:44 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Flow 'volume_create_manager' (abab98a1-a3a3-491d-be7e-914cf2cd02b4) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 16:13:44 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (a4e7d08a-d2ca-4ed8-a966-205fb3381725) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:13:44 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (a4e7d08a-d2ca-4ed8-a966-205fb3381725) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:11:28Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesSnapshotsTestJSON-volume-126016572',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=85fcb006-cd96-4018-b0e3-04872ff08267,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='34ab34dfa2e14c3785276a62a076f8e8',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:13:45Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:13:45Z,use_quota=True,user_id='c98520ee1d2c4eabaf6c186bd3ae7fee',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:13:44 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (8f41e083-88b2-4f7e-8f3f-e49bca1072d7) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:13:44 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (8f41e083-88b2-4f7e-8f3f-e49bca1072d7) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:13:44 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (a625ba15-a535-4855-8603-aab98456a989) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:13:44 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (a625ba15-a535-4855-8603-aab98456a989) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': '85fcb006-cd96-4018-b0e3-04872ff08267', 'volume_name': 'volume-85fcb006-cd96-4018-b0e3-04872ff08267', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:13:44 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (31b3d485-41c0-46f0-a5de-2a8f06153e0f) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:13:44 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (31b3d485-41c0-46f0-a5de-2a8f06153e0f) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:13:44 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (9bb5f497-fdef-4e47-9975-64fad5b22577) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:13:44 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Volume 85fcb006-cd96-4018-b0e3-04872ff08267: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-85fcb006-cd96-4018-b0e3-04872ff08267', 'volume_size': 1} Apr 17 16:13:44 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] creating volume v_85fcb006-cd96-4018-b0e3-04872ff08267. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 16:13:44 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] create volume {'name': 'v_85fcb006-cd96-4018-b0e3-04872ff08267', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 16:13:44 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:13:44 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:13:47 user cinder-volume[84696]: DEBUG cinder.utils [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.819(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:13:47 user cinder-volume[84696]: DEBUG cinder.utils [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:13:47 user cinder-volume[84696]: DEBUG cinder.utils [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.987(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:13:47 user cinder-volume[84696]: DEBUG cinder.utils [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_85fcb006-cd96-4018-b0e3-04872ff08267/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:13:47 user cinder-volume[84696]: DEBUG cinder.utils [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.124(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:13:47 user cinder-volume[84696]: DEBUG cinder.utils [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_0a7d4209-92e0-4900-8e98-36a62e68c247/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:13:47 user cinder-volume[84696]: DEBUG cinder.utils [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 11.862(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:13:47 user cinder-volume[84696]: DEBUG cinder.utils [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:13:47 user cinder-volume[84696]: DEBUG cinder.utils [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.123(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:13:47 user cinder-volume[84696]: DEBUG cinder.utils [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_49effcda-e787-4983-92b2-f8261052732d/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:13:47 user cinder-volume[84696]: DEBUG cinder.utils [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 11.780(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:13:47 user cinder-volume[84696]: DEBUG cinder.utils [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:13:50 user cinder-volume[84696]: DEBUG cinder.utils [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.891(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:13:50 user cinder-volume[84696]: DEBUG cinder.utils [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:13:50 user cinder-volume[84696]: DEBUG cinder.utils [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.060(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:13:50 user cinder-volume[84696]: DEBUG cinder.utils [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_85fcb006-cd96-4018-b0e3-04872ff08267/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:13:54 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=25,cinder:INSERT=3,cinder:UPDATE=5 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:13:56 user cinder-volume[84696]: DEBUG cinder.utils [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.035(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:13:56 user cinder-volume[84696]: DEBUG cinder.utils [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:13:56 user cinder-volume[84696]: DEBUG cinder.utils [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 20.996(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:13:56 user cinder-volume[84696]: DEBUG cinder.utils [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:13:56 user cinder-volume[84696]: DEBUG cinder.utils [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.081(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:13:56 user cinder-volume[84696]: DEBUG cinder.utils [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:13:56 user cinder-volume[84696]: DEBUG cinder.utils [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.346(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:13:56 user cinder-volume[84696]: DEBUG cinder.utils [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_0a7d4209-92e0-4900-8e98-36a62e68c247/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:13:56 user cinder-volume[84696]: DEBUG cinder.utils [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.344(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:13:56 user cinder-volume[84696]: DEBUG cinder.utils [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_49effcda-e787-4983-92b2-f8261052732d/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:13:56 user cinder-volume[84696]: DEBUG cinder.utils [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.212(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:13:56 user cinder-volume[84696]: DEBUG cinder.utils [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_85fcb006-cd96-4018-b0e3-04872ff08267/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:14:03 user cinder-volume[84696]: DEBUG cinder.utils [req-bb0706f3-0b2c-473a-acc5-93c2ac742a5f req-f2b51132-0b2f-4350-b59e-e65a5d094af4 tempest-ServerStableDeviceRescueTest-869290889 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 38.462(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:14:03 user cinder-volume[84696]: DEBUG cinder.utils [req-bb0706f3-0b2c-473a-acc5-93c2ac742a5f req-f2b51132-0b2f-4350-b59e-e65a5d094af4 tempest-ServerStableDeviceRescueTest-869290889 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:32819ec1-2236-465f-a1c7-ee31aa4a0911/luns/v_32819ec1-2236-465f-a1c7-ee31aa4a0911 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:14:04 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=19,cinder:UPDATE=8,cinder:INSERT=3 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:14:06 user cinder-volume[84696]: DEBUG cinder.utils [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.418(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:14:06 user cinder-volume[84696]: DEBUG cinder.utils [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_85fcb006-cd96-4018-b0e3-04872ff08267/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:14:06 user cinder-volume[84696]: DEBUG cinder.utils [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.253(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:14:06 user cinder-volume[84696]: DEBUG cinder.utils [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:14:06 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 72.663(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:14:06 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:14:09 user cinder-volume[84696]: DEBUG cinder.utils [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 72.537(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:14:09 user cinder-volume[84696]: DEBUG cinder.utils [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_65a642c3-b1e1-4b80-9bd2-f00151b6304a/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:14:12 user cinder-volume[84696]: DEBUG cinder.utils [req-008ec7fb-052b-43ae-952b-3c2d087ab270 req-625dc9c1-5166-463d-b990-0b1f19b50106 tempest-ServerActionsTestJSON-645547919 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 135.761(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:14:12 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-008ec7fb-052b-43ae-952b-3c2d087ab270 req-625dc9c1-5166-463d-b990-0b1f19b50106 tempest-ServerActionsTestJSON-645547919 None] Geting None from DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:1072d05f-2ebb-4398-902c-902df25464e9/luns/v_1072d05f-2ebb-4398-902c-902df25464e9 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:14:12 user cinder-volume[84696]: WARNING cinder.volume.manager [req-008ec7fb-052b-43ae-952b-3c2d087ab270 req-625dc9c1-5166-463d-b990-0b1f19b50106 tempest-ServerActionsTestJSON-645547919 None] Failed to detach volume on the backend, ignoring failure 'NoneType' object is not subscriptable Apr 17 16:14:15 user cinder-volume[84696]: DEBUG cinder.utils [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.422(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:14:15 user cinder-volume[84696]: DEBUG cinder.utils [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:14:15 user cinder-volume[84696]: DEBUG cinder.utils [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 137.888(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:14:15 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:14:15 user cinder-volume[84696]: INFO cinder.message.api [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Creating message record for request_id = req-e76412e0-459f-4be5-bc08-e01e70270c82 Apr 17 16:14:15 user cinder-volume[84696]: DEBUG cinder.utils [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.513(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:14:15 user cinder-volume[84696]: DEBUG cinder.utils [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:14:15 user cinder-volume[84696]: DEBUG cinder.utils [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.774(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:14:15 user cinder-volume[84696]: DEBUG cinder.utils [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_49effcda-e787-4983-92b2-f8261052732d/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:14:15 user cinder-volume[84696]: DEBUG cinder.utils [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.778(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:14:15 user cinder-volume[84696]: DEBUG cinder.utils [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_0a7d4209-92e0-4900-8e98-36a62e68c247/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:14:15 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 16:14:15 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (10ad4d26-3665-4d72-aa82-e8ed3c66581a) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 16:14:15 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 16:14:15 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:11:57Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-AttachVolumeTestJSON-volume-1944488700',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=bf4c0b13-49c6-4b1d-b790-7090eb82d4c4,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='2eb998c84b5b41e599af9636b929295d',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:11:57Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:11:57Z,use_quota=True,user_id='b9d4a317838b47d6a2a2e7eab7a453dd',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 16:14:15 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:11:57Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-AttachVolumeTestJSON-volume-1944488700',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=bf4c0b13-49c6-4b1d-b790-7090eb82d4c4,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='2eb998c84b5b41e599af9636b929295d',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:11:57Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:11:57Z,use_quota=True,user_id='b9d4a317838b47d6a2a2e7eab7a453dd',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(bf4c0b13-49c6-4b1d-b790-7090eb82d4c4),volume_id=bf4c0b13-49c6-4b1d-b790-7090eb82d4c4,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': 'bf4c0b13-49c6-4b1d-b790-7090eb82d4c4', 'volume_name': 'volume-bf4c0b13-49c6-4b1d-b790-7090eb82d4c4', 'volume_size': 1}} Apr 17 16:14:15 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'retry': {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']}, 'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': 'bf4c0b13-49c6-4b1d-b790-7090eb82d4c4', 'volume': {'id': 'bf4c0b13-49c6-4b1d-b790-7090eb82d4c4', '_name_id': None, 'ec2_id': None, 'user_id': 'b9d4a317838b47d6a2a2e7eab7a453dd', 'project_id': '2eb998c84b5b41e599af9636b929295d', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-AttachVolumeTestJSON-volume-1944488700', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T16:11:57.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-bf4c0b13-49c6-4b1d-b790-7090eb82d4c4', 'name_id': 'bf4c0b13-49c6-4b1d-b790-7090eb82d4c4', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-AttachVolumeTestJSON-volume-1944488700', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '2eb998c84b5b41e599af9636b929295d', 'qos_specs': None, 'reservations': ['b181be02-433e-4478-a0ce-e1f2f6eeae64', 'fe95550b-8f6c-452e-b791-feeedb2022c3', '19f55cd7-af23-4f74-9f0b-5af38cf186a9', '2ba006a2-24af-4c11-bba8-0fe8459bb371'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'b9d4a317838b47d6a2a2e7eab7a453dd', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-AttachVolumeTestJSON-volume-1944488700', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '2eb998c84b5b41e599af9636b929295d', 'qos_specs': None, 'reservations': ['b181be02-433e-4478-a0ce-e1f2f6eeae64', 'fe95550b-8f6c-452e-b791-feeedb2022c3', '19f55cd7-af23-4f74-9f0b-5af38cf186a9', '2ba006a2-24af-4c11-bba8-0fe8459bb371'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'b9d4a317838b47d6a2a2e7eab7a453dd', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': 'b9d4a317838b47d6a2a2e7eab7a453dd', 'metadata': {}, 'qos_specs': None}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(bf4c0b13-49c6-4b1d-b790-7090eb82d4c4),volume_id=bf4c0b13-49c6-4b1d-b790-7090eb82d4c4,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:11:57Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-AttachVolumeTestJSON-volume-1944488700',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=bf4c0b13-49c6-4b1d-b790-7090eb82d4c4,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='2eb998c84b5b41e599af9636b929295d',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:11:57Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:11:57Z,use_quota=True,user_id='b9d4a317838b47d6a2a2e7eab7a453dd',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 16:14:15 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:11:57Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-AttachVolumeTestJSON-volume-1944488700',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=bf4c0b13-49c6-4b1d-b790-7090eb82d4c4,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='2eb998c84b5b41e599af9636b929295d',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:11:57Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:11:57Z,use_quota=True,user_id='b9d4a317838b47d6a2a2e7eab7a453dd',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:11:57Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-AttachVolumeTestJSON-volume-1944488700',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=bf4c0b13-49c6-4b1d-b790-7090eb82d4c4,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='2eb998c84b5b41e599af9636b929295d',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:11:57Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:11:57Z,use_quota=True,user_id='b9d4a317838b47d6a2a2e7eab7a453dd',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 16:14:15 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 16:14:15 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 16:14:15 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:14:15 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 16:14:15 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:14:15 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 16:14:15 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:14:15 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 16:14:15 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:14:15 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 16:14:15 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:14:15 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 16:14:15 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:14:15 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 16:14:15 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:14:15 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 16:14:15 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:14:15 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:14:15 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 16:14:15 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 16:14:15 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (10ad4d26-3665-4d72-aa82-e8ed3c66581a) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:14:15 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (10ad4d26-3665-4d72-aa82-e8ed3c66581a) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:14:15 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (d2970805-692b-4145-9883-29442b556549) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:14:15 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (d2970805-692b-4145-9883-29442b556549) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:14:15 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (302605de-c8ab-4ef2-8494-5b3df8c7e043) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:14:15 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (302605de-c8ab-4ef2-8494-5b3df8c7e043) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:14:15 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (796c9884-6828-42aa-af51-26e3f4a7894b) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:14:15 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Updating volume bf4c0b13-49c6-4b1d-b790-7090eb82d4c4 with {'scheduled_at': datetime.datetime(2023, 4, 17, 16, 14, 15, 462416), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 16:14:15 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Volume bf4c0b13-49c6-4b1d-b790-7090eb82d4c4: re-scheduling SchedulerAPI.create_volume attempt 1 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 16:14:15 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Volume bf4c0b13-49c6-4b1d-b790-7090eb82d4c4: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 16:14:15 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] deleating volume v_bf4c0b13-49c6-4b1d-b790-7090eb82d4c4 {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 16:14:15 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] get snapshots for volume v_bf4c0b13-49c6-4b1d-b790-7090eb82d4c4 {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 16:14:15 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_bf4c0b13-49c6-4b1d-b790-7090eb82d4c4/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:14:15 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_bf4c0b13-49c6-4b1d-b790-7090eb82d4c4/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:14:15 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Volume reschedule parameters: True retry: {'num_attempts': 2, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 16:14:15 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Flow 'volume_create_manager' (86ea0934-ade0-4b6a-b927-9976b9811491) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 16:14:15 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (dabb6e4d-e4a6-4a65-9999-f9d8abceef0c) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:14:15 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (dabb6e4d-e4a6-4a65-9999-f9d8abceef0c) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:11:57Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-AttachVolumeTestJSON-volume-1944488700',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=bf4c0b13-49c6-4b1d-b790-7090eb82d4c4,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='2eb998c84b5b41e599af9636b929295d',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:14:16Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:14:16Z,use_quota=True,user_id='b9d4a317838b47d6a2a2e7eab7a453dd',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:14:15 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (814a3795-e586-4bcc-bc41-e322e51eb5a8) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:14:15 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (814a3795-e586-4bcc-bc41-e322e51eb5a8) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:14:15 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (431d0e7d-b66f-4827-bf8d-40d053555488) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:14:15 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (431d0e7d-b66f-4827-bf8d-40d053555488) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': 'bf4c0b13-49c6-4b1d-b790-7090eb82d4c4', 'volume_name': 'volume-bf4c0b13-49c6-4b1d-b790-7090eb82d4c4', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:14:15 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (465369c0-d203-478d-a929-0eb4d44505a5) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:14:15 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (465369c0-d203-478d-a929-0eb4d44505a5) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:14:15 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (fac9db02-eeb4-4c3c-8b2b-52719d146993) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:14:15 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Volume bf4c0b13-49c6-4b1d-b790-7090eb82d4c4: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-bf4c0b13-49c6-4b1d-b790-7090eb82d4c4', 'volume_size': 1} Apr 17 16:14:15 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] creating volume v_bf4c0b13-49c6-4b1d-b790-7090eb82d4c4. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 16:14:15 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] create volume {'name': 'v_bf4c0b13-49c6-4b1d-b790-7090eb82d4c4', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 16:14:15 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:14:15 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:14:18 user cinder-volume[84696]: DEBUG cinder.utils [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.795(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:14:18 user cinder-volume[84696]: DEBUG cinder.utils [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:14:18 user cinder-volume[84696]: DEBUG cinder.utils [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.966(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:14:18 user cinder-volume[84696]: DEBUG cinder.utils [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_bf4c0b13-49c6-4b1d-b790-7090eb82d4c4/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:14:21 user cinder-volume[84696]: DEBUG cinder.utils [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.036(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:14:21 user cinder-volume[84696]: DEBUG cinder.utils [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_bf4c0b13-49c6-4b1d-b790-7090eb82d4c4/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:14:21 user cinder-volume[84696]: DEBUG cinder.utils [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.870(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:14:21 user cinder-volume[84696]: DEBUG cinder.utils [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:14:24 user cinder-volume[84696]: DEBUG cinder.utils [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.849(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:14:24 user cinder-volume[84696]: DEBUG cinder.utils [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_85fcb006-cd96-4018-b0e3-04872ff08267/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:14:24 user cinder-volume[84696]: DEBUG cinder.utils [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.684(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:14:24 user cinder-volume[84696]: DEBUG cinder.utils [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:14:27 user cinder-volume[84696]: DEBUG cinder.utils [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.181(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:14:27 user cinder-volume[84696]: DEBUG cinder.utils [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_bf4c0b13-49c6-4b1d-b790-7090eb82d4c4/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:14:27 user cinder-volume[84696]: DEBUG cinder.utils [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.013(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:14:27 user cinder-volume[84696]: DEBUG cinder.utils [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:14:33 user cinder-volume[84696]: DEBUG cinder.utils [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 139.733(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:14:33 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:14:33 user cinder-volume[84696]: INFO cinder.message.api [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Creating message record for request_id = req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc Apr 17 16:14:33 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 16:14:33 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=9,cinder:UPDATE=2,cinder:INSERT=2 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:14:33 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (a9526e95-913f-468c-ad38-84c3b8a1adea) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 16:14:33 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 16:14:33 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:12:14Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesTransfersV355Test-Volume-94958465',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=125cc0cd-98dc-41a8-a252-3ae03797fb6f,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='9796cc8ccc334a5b86ff098b10a0c413',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:12:14Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:12:14Z,use_quota=True,user_id='de05a598447a4e9fb3ff488ec2d84071',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 16:14:33 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:12:14Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesTransfersV355Test-Volume-94958465',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=125cc0cd-98dc-41a8-a252-3ae03797fb6f,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='9796cc8ccc334a5b86ff098b10a0c413',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:12:14Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:12:14Z,use_quota=True,user_id='de05a598447a4e9fb3ff488ec2d84071',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(125cc0cd-98dc-41a8-a252-3ae03797fb6f),volume_id=125cc0cd-98dc-41a8-a252-3ae03797fb6f,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': '125cc0cd-98dc-41a8-a252-3ae03797fb6f', 'volume_name': 'volume-125cc0cd-98dc-41a8-a252-3ae03797fb6f', 'volume_size': 1}} Apr 17 16:14:33 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'retry': {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']}, 'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': '125cc0cd-98dc-41a8-a252-3ae03797fb6f', 'volume': {'id': '125cc0cd-98dc-41a8-a252-3ae03797fb6f', '_name_id': None, 'ec2_id': None, 'user_id': 'de05a598447a4e9fb3ff488ec2d84071', 'project_id': '9796cc8ccc334a5b86ff098b10a0c413', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-VolumesTransfersV355Test-Volume-94958465', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T16:12:13.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-125cc0cd-98dc-41a8-a252-3ae03797fb6f', 'name_id': '125cc0cd-98dc-41a8-a252-3ae03797fb6f', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumesTransfersV355Test-Volume-94958465', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '9796cc8ccc334a5b86ff098b10a0c413', 'qos_specs': None, 'reservations': ['a45788dc-5f7c-49c5-baac-3ecbd8da3ffc', 'ede61d0b-a2d4-4097-88bb-c2d0d901d083', '70e9f0f6-daa6-48d0-a906-1bcc00ee96d7', 'b04070c8-a6ec-4747-a48e-c741818cad1d'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'de05a598447a4e9fb3ff488ec2d84071', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumesTransfersV355Test-Volume-94958465', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '9796cc8ccc334a5b86ff098b10a0c413', 'qos_specs': None, 'reservations': ['a45788dc-5f7c-49c5-baac-3ecbd8da3ffc', 'ede61d0b-a2d4-4097-88bb-c2d0d901d083', '70e9f0f6-daa6-48d0-a906-1bcc00ee96d7', 'b04070c8-a6ec-4747-a48e-c741818cad1d'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'de05a598447a4e9fb3ff488ec2d84071', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': 'de05a598447a4e9fb3ff488ec2d84071', 'metadata': {}, 'qos_specs': None}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(125cc0cd-98dc-41a8-a252-3ae03797fb6f),volume_id=125cc0cd-98dc-41a8-a252-3ae03797fb6f,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:12:14Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesTransfersV355Test-Volume-94958465',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=125cc0cd-98dc-41a8-a252-3ae03797fb6f,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='9796cc8ccc334a5b86ff098b10a0c413',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:12:14Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:12:14Z,use_quota=True,user_id='de05a598447a4e9fb3ff488ec2d84071',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 16:14:33 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:12:14Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesTransfersV355Test-Volume-94958465',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=125cc0cd-98dc-41a8-a252-3ae03797fb6f,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='9796cc8ccc334a5b86ff098b10a0c413',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:12:14Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:12:14Z,use_quota=True,user_id='de05a598447a4e9fb3ff488ec2d84071',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:12:14Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesTransfersV355Test-Volume-94958465',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=125cc0cd-98dc-41a8-a252-3ae03797fb6f,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='9796cc8ccc334a5b86ff098b10a0c413',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:12:14Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:12:14Z,use_quota=True,user_id='de05a598447a4e9fb3ff488ec2d84071',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 16:14:33 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 16:14:33 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 16:14:33 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:14:33 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 16:14:33 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:14:33 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 16:14:33 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:14:33 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 16:14:33 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:14:33 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 16:14:33 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:14:33 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 16:14:33 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:14:33 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 16:14:33 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:14:33 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 16:14:33 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:14:33 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:14:33 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 16:14:33 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 16:14:33 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (a9526e95-913f-468c-ad38-84c3b8a1adea) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:14:33 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (a9526e95-913f-468c-ad38-84c3b8a1adea) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:14:33 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (0f45186b-cd3b-42e6-9c64-16d8f0bfbf38) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:14:33 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (0f45186b-cd3b-42e6-9c64-16d8f0bfbf38) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:14:33 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (6ff0458b-6371-467e-bf79-3c4e4a6cfded) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:14:33 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (6ff0458b-6371-467e-bf79-3c4e4a6cfded) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:14:33 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (b576dc36-ce9c-49c5-adf5-5b1f237b943a) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:14:33 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Updating volume 125cc0cd-98dc-41a8-a252-3ae03797fb6f with {'scheduled_at': datetime.datetime(2023, 4, 17, 16, 14, 33, 886567), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 16:14:33 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Volume 125cc0cd-98dc-41a8-a252-3ae03797fb6f: re-scheduling SchedulerAPI.create_volume attempt 1 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 16:14:33 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Volume 125cc0cd-98dc-41a8-a252-3ae03797fb6f: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 16:14:33 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] deleating volume v_125cc0cd-98dc-41a8-a252-3ae03797fb6f {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 16:14:33 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] get snapshots for volume v_125cc0cd-98dc-41a8-a252-3ae03797fb6f {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 16:14:33 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_125cc0cd-98dc-41a8-a252-3ae03797fb6f/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:14:33 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_125cc0cd-98dc-41a8-a252-3ae03797fb6f/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:14:34 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Volume reschedule parameters: True retry: {'num_attempts': 2, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 16:14:34 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Flow 'volume_create_manager' (20d31dd8-69af-4174-ae46-47214cef6a04) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 16:14:34 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (ef15c68a-a058-42d1-b8ed-a854e1ecb105) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:14:34 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (ef15c68a-a058-42d1-b8ed-a854e1ecb105) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:12:14Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesTransfersV355Test-Volume-94958465',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=125cc0cd-98dc-41a8-a252-3ae03797fb6f,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='9796cc8ccc334a5b86ff098b10a0c413',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:14:34Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:14:34Z,use_quota=True,user_id='de05a598447a4e9fb3ff488ec2d84071',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:14:34 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (b41ac25a-a402-4ab8-92fa-6f3376c8c478) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:14:34 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (b41ac25a-a402-4ab8-92fa-6f3376c8c478) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:14:34 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (e0c30a52-1cbf-438a-9623-5050c5607675) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:14:34 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (e0c30a52-1cbf-438a-9623-5050c5607675) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': '125cc0cd-98dc-41a8-a252-3ae03797fb6f', 'volume_name': 'volume-125cc0cd-98dc-41a8-a252-3ae03797fb6f', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:14:34 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (a605305a-8a58-477a-8eca-0d40f1f96fcc) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:14:34 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (a605305a-8a58-477a-8eca-0d40f1f96fcc) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:14:34 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (6f7e4a34-1280-466d-ac2c-ccdf977b421e) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:14:34 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Volume 125cc0cd-98dc-41a8-a252-3ae03797fb6f: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-125cc0cd-98dc-41a8-a252-3ae03797fb6f', 'volume_size': 1} Apr 17 16:14:34 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] creating volume v_125cc0cd-98dc-41a8-a252-3ae03797fb6f. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 16:14:34 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] create volume {'name': 'v_125cc0cd-98dc-41a8-a252-3ae03797fb6f', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 16:14:34 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:14:34 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:14:36 user cinder-volume[84696]: DEBUG cinder.utils [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.822(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:14:36 user cinder-volume[84696]: DEBUG cinder.utils [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:14:36 user cinder-volume[84696]: DEBUG cinder.utils [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.984(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:14:36 user cinder-volume[84696]: DEBUG cinder.utils [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_125cc0cd-98dc-41a8-a252-3ae03797fb6f/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:14:36 user cinder-volume[84696]: DEBUG cinder.utils [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.399(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:14:36 user cinder-volume[84696]: DEBUG cinder.utils [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_bf4c0b13-49c6-4b1d-b790-7090eb82d4c4/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:14:36 user cinder-volume[84696]: DEBUG cinder.utils [req-bb0706f3-0b2c-473a-acc5-93c2ac742a5f req-f2b51132-0b2f-4350-b59e-e65a5d094af4 tempest-ServerStableDeviceRescueTest-869290889 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 72.258(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:14:36 user cinder-volume[84696]: DEBUG cinder.utils [req-bb0706f3-0b2c-473a-acc5-93c2ac742a5f req-f2b51132-0b2f-4350-b59e-e65a5d094af4 tempest-ServerStableDeviceRescueTest-869290889 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:32819ec1-2236-465f-a1c7-ee31aa4a0911/luns/v_32819ec1-2236-465f-a1c7-ee31aa4a0911 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:14:36 user cinder-volume[84696]: DEBUG cinder.utils [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.232(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:14:36 user cinder-volume[84696]: DEBUG cinder.utils [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:14:39 user cinder-volume[84696]: DEBUG cinder.utils [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.054(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:14:39 user cinder-volume[84696]: DEBUG cinder.utils [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_125cc0cd-98dc-41a8-a252-3ae03797fb6f/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:14:39 user cinder-volume[84696]: DEBUG cinder.utils [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.896(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:14:39 user cinder-volume[84696]: DEBUG cinder.utils [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:14:44 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=13,cinder:UPDATE=6,cinder:INSERT=2 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:14:46 user cinder-volume[84696]: DEBUG cinder.utils [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.198(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:14:46 user cinder-volume[84696]: DEBUG cinder.utils [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_125cc0cd-98dc-41a8-a252-3ae03797fb6f/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:14:46 user cinder-volume[84696]: DEBUG cinder.utils [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.040(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:14:46 user cinder-volume[84696]: DEBUG cinder.utils [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:14:46 user cinder-volume[84696]: INFO cinder.volume.manager [None req-8a4be074-8e3d-42a9-aae8-4a9dcdb6498b tempest-GroupsTest-789764105 None] Group tempest-Group1-581669039: creating Apr 17 16:14:46 user cinder-volume[84696]: INFO cinder.volume.manager [None req-8a4be074-8e3d-42a9-aae8-4a9dcdb6498b tempest-GroupsTest-789764105 None] Group tempest-Group1-581669039: created successfully Apr 17 16:14:46 user cinder-volume[84696]: INFO cinder.volume.manager [None req-8a4be074-8e3d-42a9-aae8-4a9dcdb6498b tempest-GroupsTest-789764105 None] Create group completed successfully. Apr 17 16:14:47 user cinder-volume[84696]: INFO cinder.volume.manager [None req-75d1bffa-cc99-439c-98ef-55fbfbaee58f tempest-GroupsTest-789764105 None] Group tempest-Group2-1666988956: creating Apr 17 16:14:47 user cinder-volume[84696]: INFO cinder.volume.manager [None req-75d1bffa-cc99-439c-98ef-55fbfbaee58f tempest-GroupsTest-789764105 None] Group tempest-Group2-1666988956: created successfully Apr 17 16:14:47 user cinder-volume[84696]: INFO cinder.volume.manager [None req-75d1bffa-cc99-439c-98ef-55fbfbaee58f tempest-GroupsTest-789764105 None] Create group completed successfully. Apr 17 16:14:49 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Volume reschedule parameters: True retry: {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 16:14:49 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Flow 'volume_create_manager' (6edae92f-f435-48a3-b055-93905afcbb72) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 16:14:49 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (81b9e09d-c5f4-4b6e-b7a8-368a1c23729f) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:14:49 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (81b9e09d-c5f4-4b6e-b7a8-368a1c23729f) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:14:49Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-volume-992255480',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=07829a92-7f87-41a0-ad0d-04c53bb24b30,host='user@joviandss-2#joviandss-2',id=ae1eb1a7-8e56-418f-9032-a464645060fe,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='e10e34b8774b42d8a4cb1f78d72b411d',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:14:49Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:14:49Z,use_quota=True,user_id='e7392ed432ad41279967273f16b4a4c8',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(39cee13b-7a8b-43bd-ae52-5d071608f5ab),volume_type_id=39cee13b-7a8b-43bd-ae52-5d071608f5ab)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:14:49 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (15190e54-bc51-46cd-ac25-f5f4b36278fe) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:14:49 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (15190e54-bc51-46cd-ac25-f5f4b36278fe) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:14:49 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (db334af0-fd19-46f0-88ca-1fcdd66c739c) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:14:49 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (db334af0-fd19-46f0-88ca-1fcdd66c739c) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': 'ae1eb1a7-8e56-418f-9032-a464645060fe', 'volume_name': 'volume-ae1eb1a7-8e56-418f-9032-a464645060fe', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:14:49 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (d4dfa3a6-6ff3-4def-a3f8-a804a73fef86) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:14:49 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (d4dfa3a6-6ff3-4def-a3f8-a804a73fef86) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:14:49 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (d99e52ba-d0d9-4f61-be83-6ede164e256d) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:14:49 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Volume ae1eb1a7-8e56-418f-9032-a464645060fe: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-ae1eb1a7-8e56-418f-9032-a464645060fe', 'volume_size': 1} Apr 17 16:14:49 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] creating volume v_ae1eb1a7-8e56-418f-9032-a464645060fe. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 16:14:49 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] create volume {'name': 'v_ae1eb1a7-8e56-418f-9032-a464645060fe', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 16:14:49 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:14:49 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:14:49 user cinder-volume[84696]: DEBUG cinder.utils [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 73.559(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:14:49 user cinder-volume[84696]: DEBUG cinder.utils [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_49effcda-e787-4983-92b2-f8261052732d/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:14:49 user cinder-volume[84696]: DEBUG cinder.utils [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 73.300(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:14:49 user cinder-volume[84696]: DEBUG cinder.utils [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:14:49 user cinder-volume[84696]: DEBUG cinder.utils [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 73.218(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:14:49 user cinder-volume[84696]: DEBUG cinder.utils [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:14:49 user cinder-volume[84696]: DEBUG cinder.utils [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 73.567(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:14:49 user cinder-volume[84696]: DEBUG cinder.utils [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_0a7d4209-92e0-4900-8e98-36a62e68c247/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:14:49 user cinder-volume[84696]: DEBUG cinder.utils [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 0.079(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:14:49 user cinder-volume[84696]: DEBUG cinder.utils [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:14:52 user cinder-volume[84696]: DEBUG cinder.utils [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 3.145(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:14:52 user cinder-volume[84696]: DEBUG cinder.utils [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:14:55 user cinder-volume[84696]: DEBUG cinder.utils [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.828(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:14:55 user cinder-volume[84696]: DEBUG cinder.utils [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_bf4c0b13-49c6-4b1d-b790-7090eb82d4c4/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:14:55 user cinder-volume[84696]: DEBUG cinder.utils [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.660(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:14:55 user cinder-volume[84696]: DEBUG cinder.utils [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:14:55 user cinder-volume[84696]: DEBUG cinder.utils [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.257(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:14:55 user cinder-volume[84696]: DEBUG cinder.utils [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:14:55 user cinder-volume[84696]: DEBUG cinder.utils [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.417(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:14:55 user cinder-volume[84696]: DEBUG cinder.utils [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_125cc0cd-98dc-41a8-a252-3ae03797fb6f/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:14:58 user cinder-volume[84696]: DEBUG cinder.utils [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 73.479(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:14:58 user cinder-volume[84696]: DEBUG cinder.utils [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:14:58 user cinder-volume[84696]: DEBUG cinder.utils [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 73.647(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:14:58 user cinder-volume[84696]: DEBUG cinder.utils [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_85fcb006-cd96-4018-b0e3-04872ff08267/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:14:58 user cinder-volume[84696]: DEBUG cinder.utils [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 9.295(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:14:58 user cinder-volume[84696]: DEBUG cinder.utils [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:15:04 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=23,cinder:UPDATE=8 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:15:08 user cinder-volume[84696]: DEBUG cinder.utils [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 19.433(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:15:08 user cinder-volume[84696]: DEBUG cinder.utils [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:15:11 user cinder-volume[84696]: DEBUG cinder.utils [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 37.702(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:15:11 user cinder-volume[84696]: DEBUG cinder.utils [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_125cc0cd-98dc-41a8-a252-3ae03797fb6f/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:15:11 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 138.101(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:15:11 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:15:11 user cinder-volume[84696]: ERROR oslo_service.periodic_task [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Error during VolumeManager.publish_service_capabilities: TypeError: 'NoneType' object is not subscriptable Apr 17 16:15:11 user cinder-volume[84696]: ERROR oslo_service.periodic_task Traceback (most recent call last): Apr 17 16:15:11 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/usr/local/lib/python3.10/dist-packages/oslo_service/periodic_task.py", line 216, in run_periodic_tasks Apr 17 16:15:11 user cinder-volume[84696]: ERROR oslo_service.periodic_task task(self, context) Apr 17 16:15:11 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/manager.py", line 2876, in publish_service_capabilities Apr 17 16:15:11 user cinder-volume[84696]: ERROR oslo_service.periodic_task self._report_driver_status(context) Apr 17 16:15:11 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/manager.py", line 2755, in _report_driver_status Apr 17 16:15:11 user cinder-volume[84696]: ERROR oslo_service.periodic_task volume_stats = get_stats() Apr 17 16:15:11 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/usr/local/lib/python3.10/dist-packages/oslo_utils/timeutils.py", line 319, in wrapper Apr 17 16:15:11 user cinder-volume[84696]: ERROR oslo_service.periodic_task result = func(*args, **kwargs) Apr 17 16:15:11 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/manager.py", line 2753, in get_stats Apr 17 16:15:11 user cinder-volume[84696]: ERROR oslo_service.periodic_task return self.driver.get_volume_stats(refresh=True) Apr 17 16:15:11 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/driver.py", line 666, in get_volume_stats Apr 17 16:15:11 user cinder-volume[84696]: ERROR oslo_service.periodic_task self._update_volume_stats() Apr 17 16:15:11 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 726, in _update_volume_stats Apr 17 16:15:11 user cinder-volume[84696]: ERROR oslo_service.periodic_task pool_stats = self.ra.get_pool_stats() Apr 17 16:15:11 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 897, in get_pool_stats Apr 17 16:15:11 user cinder-volume[84696]: ERROR oslo_service.periodic_task if not resp["error"] and resp["code"] == 200: Apr 17 16:15:11 user cinder-volume[84696]: ERROR oslo_service.periodic_task TypeError: 'NoneType' object is not subscriptable Apr 17 16:15:11 user cinder-volume[84696]: ERROR oslo_service.periodic_task Apr 17 16:15:11 user cinder-volume[84696]: DEBUG cinder.utils [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 37.545(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:15:11 user cinder-volume[84696]: DEBUG cinder.utils [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:15:11 user cinder-volume[84696]: WARNING oslo.service.loopingcall [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Function 'cinder.service.Service.periodic_tasks' run outlasted interval by 78.11 sec Apr 17 16:15:11 user cinder-volume[84696]: DEBUG oslo_service.periodic_task [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Running periodic task VolumeManager.publish_service_capabilities {{(pid=84696) run_periodic_tasks /usr/local/lib/python3.10/dist-packages/oslo_service/periodic_task.py:210}} Apr 17 16:15:11 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Updating volume stats {{(pid=84696) _update_volume_stats /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:724}} Apr 17 16:15:11 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Get pool Pool-0 fsprops {{(pid=84696) get_pool_stats /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:894}} Apr 17 16:15:11 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:15:11 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:15:14 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 3.066(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:15:14 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:15:14 user cinder-volume[84696]: DEBUG cinder.utils [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 137.979(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:15:14 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_65a642c3-b1e1-4b80-9bd2-f00151b6304a/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:15:14 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (23d34cdc-882e-4b5d-abc8-2a3d8ec2a97f) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 16:15:14 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (7efe3b15-9aa2-436c-8283-68e86b986737) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:15:14 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (7efe3b15-9aa2-436c-8283-68e86b986737) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:15:14 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Flow 'volume_create_manager' (3f4b9fa0-2892-458f-906b-35570f6450e1) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 16:15:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-251ec93b-05ae-4736-ab3c-338a5d9efb63 tempest-AttachVolumeTestJSON-1193954556 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 16:15:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 16:15:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 16:15:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 16:15:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 16:15:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 16:15:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 16:15:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 16:15:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 16:15:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 16:15:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 16:15:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 16:15:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 16:15:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 16:15:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 16:15:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 16:15:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 16:15:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 16:15:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 16:15:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 16:15:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 16:15:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 16:15:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 16:15:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 16:15:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:15:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 16:15:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:15:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 16:15:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:15:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 16:15:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:15:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 16:15:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:15:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 16:15:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:15:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 16:15:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:15:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 16:15:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:15:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:15:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 16:15:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 16:15:17 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.138(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:15:17 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:15:23 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.286(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:15:23 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:15:24 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=17,cinder:UPDATE=2 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:15:24 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-bb0706f3-0b2c-473a-acc5-93c2ac742a5f req-0a3ae49c-837d-4b32-a241-6c27a7c860b1 tempest-ServerStableDeviceRescueTest-869290889 None] terminate connection for 32819ec1-2236-465f-a1c7-ee31aa4a0911 {{(pid=84696) terminate_connection /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:1032}} Apr 17 16:15:24 user cinder-volume[84696]: INFO cinder.volume.manager [req-bb0706f3-0b2c-473a-acc5-93c2ac742a5f req-0a3ae49c-837d-4b32-a241-6c27a7c860b1 tempest-ServerStableDeviceRescueTest-869290889 None] Terminate volume connection completed successfully. Apr 17 16:15:24 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-bb0706f3-0b2c-473a-acc5-93c2ac742a5f req-0a3ae49c-837d-4b32-a241-6c27a7c860b1 tempest-ServerStableDeviceRescueTest-869290889 None] Deleting attachment fb44dce3-1af2-43cf-95fa-68ff027e0699. {{(pid=84696) attachment_delete /opt/stack/cinder/cinder/volume/manager.py:5006}} Apr 17 16:15:24 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-bb0706f3-0b2c-473a-acc5-93c2ac742a5f req-0a3ae49c-837d-4b32-a241-6c27a7c860b1 tempest-ServerStableDeviceRescueTest-869290889 None] remove_export for volume: 32819ec1-2236-465f-a1c7-ee31aa4a0911. {{(pid=84696) remove_export /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:718}} Apr 17 16:15:24 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-bb0706f3-0b2c-473a-acc5-93c2ac742a5f req-0a3ae49c-837d-4b32-a241-6c27a7c860b1 tempest-ServerStableDeviceRescueTest-869290889 None] remove export {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:912}} Apr 17 16:15:24 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-bb0706f3-0b2c-473a-acc5-93c2ac742a5f req-0a3ae49c-837d-4b32-a241-6c27a7c860b1 tempest-ServerStableDeviceRescueTest-869290889 None] detach volume:Volume(_name_id=None,admin_metadata=,attach_status='attached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:27:42Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-ServerStableDeviceRescueTest-volume-648918308',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=32819ec1-2236-465f-a1c7-ee31aa4a0911,launched_at=2023-04-17T15:27:46Z,metadata=,migration_status=None,multiattach=False,previous_status=None,project_id='6bdfb8642b9341f6ba289e920c33cf47',provider_auth='CHAP CmhLYVJy GWkyzUl8QJxnjY',provider_geometry=None,provider_id=None,provider_location='172.16.0.220:3260,1 iqn.2010-10.org.openstack:32819ec1-2236-465f-a1c7-ee31aa4a0911 0',replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:27:43Z,service_uuid='e628352e-c4cf-40eb-8389-cce40663026a',shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='in-use',terminated_at=None,updated_at=2023-04-17T15:27:57Z,use_quota=True,user_id='a972c04314724233bc8e33b64747fecf',volume_attachment=,volume_type=,volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c) from target:iqn.2010-10.org.openstack:32819ec1-2236-465f-a1c7-ee31aa4a0911. {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:913}} Apr 17 16:15:24 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [req-bb0706f3-0b2c-473a-acc5-93c2ac742a5f req-0a3ae49c-837d-4b32-a241-6c27a7c860b1 tempest-ServerStableDeviceRescueTest-869290889 None] detach volume v_32819ec1-2236-465f-a1c7-ee31aa4a0911 from target iqn.2010-10.org.openstack:32819ec1-2236-465f-a1c7-ee31aa4a0911 {{(pid=84696) detach_target_vol /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:606}} Apr 17 16:15:24 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-bb0706f3-0b2c-473a-acc5-93c2ac742a5f req-0a3ae49c-837d-4b32-a241-6c27a7c860b1 tempest-ServerStableDeviceRescueTest-869290889 None] Sending pool request DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:32819ec1-2236-465f-a1c7-ee31aa4a0911/luns/v_32819ec1-2236-465f-a1c7-ee31aa4a0911 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:15:24 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-bb0706f3-0b2c-473a-acc5-93c2ac742a5f req-0a3ae49c-837d-4b32-a241-6c27a7c860b1 tempest-ServerStableDeviceRescueTest-869290889 None] Sending DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:32819ec1-2236-465f-a1c7-ee31aa4a0911/luns/v_32819ec1-2236-465f-a1c7-ee31aa4a0911 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:15:26 user cinder-volume[84696]: DEBUG cinder.utils [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 37.865(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:15:26 user cinder-volume[84696]: DEBUG cinder.utils [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:15:26 user cinder-volume[84696]: DEBUG cinder.utils [req-bb0706f3-0b2c-473a-acc5-93c2ac742a5f req-0a3ae49c-837d-4b32-a241-6c27a7c860b1 tempest-ServerStableDeviceRescueTest-869290889 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.336(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:15:26 user cinder-volume[84696]: DEBUG cinder.utils [req-bb0706f3-0b2c-473a-acc5-93c2ac742a5f req-0a3ae49c-837d-4b32-a241-6c27a7c860b1 tempest-ServerStableDeviceRescueTest-869290889 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:32819ec1-2236-465f-a1c7-ee31aa4a0911/luns/v_32819ec1-2236-465f-a1c7-ee31aa4a0911 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:15:30 user cinder-volume[84696]: DEBUG cinder.utils [req-bb0706f3-0b2c-473a-acc5-93c2ac742a5f req-0a3ae49c-837d-4b32-a241-6c27a7c860b1 tempest-ServerStableDeviceRescueTest-869290889 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.407(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:15:30 user cinder-volume[84696]: DEBUG cinder.utils [req-bb0706f3-0b2c-473a-acc5-93c2ac742a5f req-0a3ae49c-837d-4b32-a241-6c27a7c860b1 tempest-ServerStableDeviceRescueTest-869290889 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:32819ec1-2236-465f-a1c7-ee31aa4a0911/luns/v_32819ec1-2236-465f-a1c7-ee31aa4a0911 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:15:30 user cinder-volume[84696]: DEBUG cinder.utils [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 74.380(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:15:30 user cinder-volume[84696]: DEBUG cinder.utils [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:15:30 user cinder-volume[84696]: DEBUG cinder.utils [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 74.550(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:15:30 user cinder-volume[84696]: DEBUG cinder.utils [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_bf4c0b13-49c6-4b1d-b790-7090eb82d4c4/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:15:33 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.502(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:15:33 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:15:34 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Volume reschedule parameters: True retry: {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 16:15:34 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Flow 'volume_create_manager' (45002bb1-e3ac-48b3-afe5-acf3c897c114) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 16:15:34 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (252e12a0-0371-401d-9f23-c1e3320e16e6) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:15:34 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (252e12a0-0371-401d-9f23-c1e3320e16e6) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:15:34Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesTransfersV357Test-Volume-788739626',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=4725fbeb-e869-48db-a487-33998b955f83,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='1db59dbe212347b8a44e27ebed9e22fb',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:15:35Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:15:35Z,use_quota=True,user_id='ac9545120d3a48bbb11e4cd5e706b53b',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:15:34 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (136d9366-1462-4c4a-a656-b45f2e911887) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:15:34 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (136d9366-1462-4c4a-a656-b45f2e911887) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:15:34 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (27b4ae4a-d21f-45cb-96cf-4dd3cdc3cae8) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:15:34 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (27b4ae4a-d21f-45cb-96cf-4dd3cdc3cae8) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': '4725fbeb-e869-48db-a487-33998b955f83', 'volume_name': 'volume-4725fbeb-e869-48db-a487-33998b955f83', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:15:34 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (32e7f0be-7eb2-4352-888d-75f5b25bf57f) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:15:34 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (32e7f0be-7eb2-4352-888d-75f5b25bf57f) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:15:34 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (38a5767d-d01e-4076-b859-6d3bbf0f9eac) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:15:34 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Volume 4725fbeb-e869-48db-a487-33998b955f83: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-4725fbeb-e869-48db-a487-33998b955f83', 'volume_size': 1} Apr 17 16:15:34 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] creating volume v_4725fbeb-e869-48db-a487-33998b955f83. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 16:15:34 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] create volume {'name': 'v_4725fbeb-e869-48db-a487-33998b955f83', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 16:15:34 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:15:34 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:15:34 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=10,cinder:UPDATE=1 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:15:36 user cinder-volume[84696]: DEBUG cinder.utils [req-bb0706f3-0b2c-473a-acc5-93c2ac742a5f req-0a3ae49c-837d-4b32-a241-6c27a7c860b1 tempest-ServerStableDeviceRescueTest-869290889 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 11.551(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:15:36 user cinder-volume[84696]: DEBUG cinder.utils [req-bb0706f3-0b2c-473a-acc5-93c2ac742a5f req-0a3ae49c-837d-4b32-a241-6c27a7c860b1 tempest-ServerStableDeviceRescueTest-869290889 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:32819ec1-2236-465f-a1c7-ee31aa4a0911/luns/v_32819ec1-2236-465f-a1c7-ee31aa4a0911 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:15:36 user cinder-volume[84696]: DEBUG cinder.utils [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 1.550(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:15:36 user cinder-volume[84696]: DEBUG cinder.utils [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:15:39 user cinder-volume[84696]: DEBUG cinder.utils [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 4.620(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:15:39 user cinder-volume[84696]: DEBUG cinder.utils [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:15:42 user cinder-volume[84696]: DEBUG cinder.utils [req-bb0706f3-0b2c-473a-acc5-93c2ac742a5f req-f2b51132-0b2f-4350-b59e-e65a5d094af4 tempest-ServerStableDeviceRescueTest-869290889 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 137.694(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:15:42 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-bb0706f3-0b2c-473a-acc5-93c2ac742a5f req-f2b51132-0b2f-4350-b59e-e65a5d094af4 tempest-ServerStableDeviceRescueTest-869290889 None] Geting None from DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:32819ec1-2236-465f-a1c7-ee31aa4a0911/luns/v_32819ec1-2236-465f-a1c7-ee31aa4a0911 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:15:42 user cinder-volume[84696]: WARNING cinder.volume.manager [req-bb0706f3-0b2c-473a-acc5-93c2ac742a5f req-f2b51132-0b2f-4350-b59e-e65a5d094af4 tempest-ServerStableDeviceRescueTest-869290889 None] Failed to detach volume on the backend, ignoring failure 'NoneType' object is not subscriptable Apr 17 16:15:44 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=5,cinder:UPDATE=3 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:15:45 user cinder-volume[84696]: DEBUG cinder.utils [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 71.498(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:15:45 user cinder-volume[84696]: DEBUG cinder.utils [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_125cc0cd-98dc-41a8-a252-3ae03797fb6f/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:15:45 user cinder-volume[84696]: DEBUG cinder.utils [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 10.769(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:15:45 user cinder-volume[84696]: DEBUG cinder.utils [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:15:45 user cinder-volume[84696]: DEBUG cinder.utils [req-bb0706f3-0b2c-473a-acc5-93c2ac742a5f req-0a3ae49c-837d-4b32-a241-6c27a7c860b1 tempest-ServerStableDeviceRescueTest-869290889 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 20.773(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:15:45 user cinder-volume[84696]: DEBUG cinder.utils [req-bb0706f3-0b2c-473a-acc5-93c2ac742a5f req-0a3ae49c-837d-4b32-a241-6c27a7c860b1 tempest-ServerStableDeviceRescueTest-869290889 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:32819ec1-2236-465f-a1c7-ee31aa4a0911/luns/v_32819ec1-2236-465f-a1c7-ee31aa4a0911 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:15:45 user cinder-volume[84696]: DEBUG cinder.utils [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 71.342(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:15:45 user cinder-volume[84696]: DEBUG cinder.utils [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:15:51 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.930(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:15:51 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:15:54 user cinder-volume[84696]: DEBUG cinder.utils [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 139.003(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:15:54 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_49effcda-e787-4983-92b2-f8261052732d/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:15:54 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (3d2ff43b-f7cd-4323-a7ca-708272500c69) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 16:15:54 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (5d43cfbc-acb0-4d8b-80f6-2b361bd00590) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:15:54 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (5d43cfbc-acb0-4d8b-80f6-2b361bd00590) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:15:54 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Flow 'volume_create_manager' (8d475cc0-3d90-45f1-aca7-4f5582599063) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 16:15:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 16:15:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 16:15:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 16:15:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 16:15:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 16:15:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 16:15:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 16:15:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 16:15:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 16:15:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 16:15:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 16:15:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 16:15:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 16:15:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 16:15:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 16:15:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 16:15:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 16:15:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 16:15:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 16:15:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 16:15:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 16:15:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 16:15:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 16:15:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 16:15:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:15:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 16:15:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:15:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 16:15:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:15:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 16:15:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:15:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 16:15:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:15:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 16:15:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:15:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 16:15:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:15:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 16:15:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:15:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:15:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 16:15:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 16:15:54 user cinder-volume[84696]: DEBUG cinder.utils [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 138.768(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:15:54 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:15:54 user cinder-volume[84696]: INFO cinder.message.api [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Creating message record for request_id = req-40e48e44-d46b-4e9d-82c5-3b795728a90d Apr 17 16:15:54 user cinder-volume[84696]: DEBUG cinder.utils [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 139.036(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:15:54 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_0a7d4209-92e0-4900-8e98-36a62e68c247/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:15:54 user cinder-volume[84696]: WARNING cinder.volume.manager [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (c14008a8-e359-4437-b365-01be3b46be87) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 16:15:54 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (e6ebcd93-56f8-4a20-84fa-a6d4959579b7) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:15:54 user cinder-volume[84696]: WARNING cinder.volume.manager [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (e6ebcd93-56f8-4a20-84fa-a6d4959579b7) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:15:54 user cinder-volume[84696]: WARNING cinder.volume.manager [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Flow 'volume_create_manager' (d86e39d2-4925-436d-a8a4-c8d45174658c) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 16:15:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 16:15:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 16:15:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 16:15:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 16:15:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 16:15:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 16:15:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 16:15:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 16:15:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 16:15:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 16:15:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 16:15:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 16:15:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 16:15:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 16:15:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 16:15:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 16:15:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 16:15:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 16:15:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 16:15:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 16:15:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 16:15:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 16:15:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 16:15:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 16:15:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:15:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 16:15:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:15:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 16:15:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:15:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 16:15:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:15:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 16:15:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:15:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 16:15:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:15:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 16:15:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:15:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 16:15:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:15:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:15:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 16:15:54 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 16:15:54 user cinder-volume[84696]: DEBUG cinder.utils [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 20.033(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:15:54 user cinder-volume[84696]: DEBUG cinder.utils [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:15:54 user cinder-volume[84696]: DEBUG cinder.utils [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 138.708(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:15:54 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:15:54 user cinder-volume[84696]: INFO cinder.message.api [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Creating message record for request_id = req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 Apr 17 16:15:54 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=3,cinder:UPDATE=1 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:15:54 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 16:15:54 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-8f5be14a-310d-428b-a218-6628609e68ca None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 16:15:54 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (ef69ec07-4798-4508-b77d-01a0b3ee438a) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 16:15:54 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 16:15:54 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:08:58Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesTransfersV355Test-Volume-1916167074',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=49effcda-e787-4983-92b2-f8261052732d,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='9796cc8ccc334a5b86ff098b10a0c413',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:13:36Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:13:36Z,use_quota=True,user_id='de05a598447a4e9fb3ff488ec2d84071',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 16:15:54 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:08:58Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesTransfersV355Test-Volume-1916167074',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=49effcda-e787-4983-92b2-f8261052732d,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='9796cc8ccc334a5b86ff098b10a0c413',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:13:36Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:13:36Z,use_quota=True,user_id='de05a598447a4e9fb3ff488ec2d84071',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(49effcda-e787-4983-92b2-f8261052732d),volume_id=49effcda-e787-4983-92b2-f8261052732d,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': '49effcda-e787-4983-92b2-f8261052732d', 'volume_name': 'volume-49effcda-e787-4983-92b2-f8261052732d', 'volume_size': 1}} Apr 17 16:15:54 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': '49effcda-e787-4983-92b2-f8261052732d', 'volume': {'id': '49effcda-e787-4983-92b2-f8261052732d', '_name_id': None, 'ec2_id': None, 'user_id': 'de05a598447a4e9fb3ff488ec2d84071', 'project_id': '9796cc8ccc334a5b86ff098b10a0c413', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-VolumesTransfersV355Test-Volume-1916167074', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'admin_metadata': {}, 'glance_metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T16:08:57.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-49effcda-e787-4983-92b2-f8261052732d', 'name_id': '49effcda-e787-4983-92b2-f8261052732d', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumesTransfersV355Test-Volume-1916167074', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '9796cc8ccc334a5b86ff098b10a0c413', 'qos_specs': None, 'reservations': ['18f38527-7ebd-41b2-8296-efa6d1a6acd7', '6a781a35-53f4-4739-a5cc-a9986cdaa2fe', 'd632f456-4c05-4c56-84e4-431db2216088', '398f088d-a47c-4ff9-aee7-12eb5bef8d35'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'de05a598447a4e9fb3ff488ec2d84071', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumesTransfersV355Test-Volume-1916167074', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '9796cc8ccc334a5b86ff098b10a0c413', 'qos_specs': None, 'reservations': ['18f38527-7ebd-41b2-8296-efa6d1a6acd7', '6a781a35-53f4-4739-a5cc-a9986cdaa2fe', 'd632f456-4c05-4c56-84e4-431db2216088', '398f088d-a47c-4ff9-aee7-12eb5bef8d35'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'de05a598447a4e9fb3ff488ec2d84071', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': 'de05a598447a4e9fb3ff488ec2d84071', 'metadata': {}, 'qos_specs': None, 'retry': {'num_attempts': 3, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']}}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(49effcda-e787-4983-92b2-f8261052732d),volume_id=49effcda-e787-4983-92b2-f8261052732d,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:08:58Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesTransfersV355Test-Volume-1916167074',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=49effcda-e787-4983-92b2-f8261052732d,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='9796cc8ccc334a5b86ff098b10a0c413',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:13:36Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:13:36Z,use_quota=True,user_id='de05a598447a4e9fb3ff488ec2d84071',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 16:15:54 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:08:58Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesTransfersV355Test-Volume-1916167074',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=49effcda-e787-4983-92b2-f8261052732d,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='9796cc8ccc334a5b86ff098b10a0c413',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:13:36Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:13:36Z,use_quota=True,user_id='de05a598447a4e9fb3ff488ec2d84071',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:08:58Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesTransfersV355Test-Volume-1916167074',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=49effcda-e787-4983-92b2-f8261052732d,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='9796cc8ccc334a5b86ff098b10a0c413',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:13:36Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:13:36Z,use_quota=True,user_id='de05a598447a4e9fb3ff488ec2d84071',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 16:15:54 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 16:15:54 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 16:15:54 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:15:54 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 16:15:54 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:15:54 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 16:15:54 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:15:54 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 16:15:54 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:15:54 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 16:15:54 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:15:54 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 16:15:54 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:15:54 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 16:15:54 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:15:54 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 16:15:54 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:15:54 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:15:54 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 16:15:54 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 16:15:54 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (ef69ec07-4798-4508-b77d-01a0b3ee438a) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:15:54 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (ef69ec07-4798-4508-b77d-01a0b3ee438a) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:15:54 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (abf9cb80-caa0-49b4-8eaa-eb33f8d35921) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:15:54 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (abf9cb80-caa0-49b4-8eaa-eb33f8d35921) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:15:54 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (efd8d66c-639a-40e1-a8ec-3bb893b2f072) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:15:54 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (efd8d66c-639a-40e1-a8ec-3bb893b2f072) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:15:54 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (1c72d6cd-9242-492c-a056-e0f57367779c) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:15:54 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Updating volume 49effcda-e787-4983-92b2-f8261052732d with {'scheduled_at': datetime.datetime(2023, 4, 17, 16, 15, 54, 747144), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 16:15:54 user cinder-volume[84696]: WARNING cinder.volume.manager [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (fbe14b45-ed1a-4d2c-b25b-03f9c9df58ae) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 16:15:54 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 16:15:54 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:11:18Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesGetTestJSON-Volume-1455502637',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=0a7d4209-92e0-4900-8e98-36a62e68c247,launched_at=None,metadata={Type='work'},migration_status=None,multiattach=False,previous_status=None,project_id='e330412972fa44b9bf60cd9afe4399cd',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:13:36Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:13:36Z,use_quota=True,user_id='e14688d19f1b44928fa8aceb2da0ac82',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 16:15:54 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:11:18Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesGetTestJSON-Volume-1455502637',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=0a7d4209-92e0-4900-8e98-36a62e68c247,launched_at=None,metadata={Type='work'},migration_status=None,multiattach=False,previous_status=None,project_id='e330412972fa44b9bf60cd9afe4399cd',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:13:36Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:13:36Z,use_quota=True,user_id='e14688d19f1b44928fa8aceb2da0ac82',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(0a7d4209-92e0-4900-8e98-36a62e68c247),volume_id=0a7d4209-92e0-4900-8e98-36a62e68c247,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': '0a7d4209-92e0-4900-8e98-36a62e68c247', 'volume_name': 'volume-0a7d4209-92e0-4900-8e98-36a62e68c247', 'volume_size': 1}} Apr 17 16:15:54 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': '0a7d4209-92e0-4900-8e98-36a62e68c247', 'volume': {'id': '0a7d4209-92e0-4900-8e98-36a62e68c247', '_name_id': None, 'ec2_id': None, 'user_id': 'e14688d19f1b44928fa8aceb2da0ac82', 'project_id': 'e330412972fa44b9bf60cd9afe4399cd', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-VolumesGetTestJSON-Volume-1455502637', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {'Type': 'work'}, 'admin_metadata': {}, 'glance_metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T16:11:17.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-0a7d4209-92e0-4900-8e98-36a62e68c247', 'name_id': '0a7d4209-92e0-4900-8e98-36a62e68c247', 'volume_metadata': [{'key': 'Type', 'value': 'work'}], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumesGetTestJSON-Volume-1455502637', 'encryption_key_id': None, 'metadata': {'Type': 'work'}, 'multiattach': False, 'project_id': 'e330412972fa44b9bf60cd9afe4399cd', 'qos_specs': None, 'reservations': ['a1d7a4e6-78b5-4539-8c86-495dc07de36d', '0ea4500b-e7dd-4705-a54f-22c9b06155b5', 'b2cf1ff4-8bad-445f-8b8b-6ebee58654f1', '9aeb8f74-fd45-412b-b0b4-1970c607402d'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'e14688d19f1b44928fa8aceb2da0ac82', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumesGetTestJSON-Volume-1455502637', 'encryption_key_id': None, 'metadata': {'Type': 'work'}, 'multiattach': False, 'project_id': 'e330412972fa44b9bf60cd9afe4399cd', 'qos_specs': None, 'reservations': ['a1d7a4e6-78b5-4539-8c86-495dc07de36d', '0ea4500b-e7dd-4705-a54f-22c9b06155b5', 'b2cf1ff4-8bad-445f-8b8b-6ebee58654f1', '9aeb8f74-fd45-412b-b0b4-1970c607402d'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'e14688d19f1b44928fa8aceb2da0ac82', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': 'e14688d19f1b44928fa8aceb2da0ac82', 'metadata': {'Type': 'work'}, 'qos_specs': None, 'retry': {'num_attempts': 2, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']}}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(0a7d4209-92e0-4900-8e98-36a62e68c247),volume_id=0a7d4209-92e0-4900-8e98-36a62e68c247,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:11:18Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesGetTestJSON-Volume-1455502637',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=0a7d4209-92e0-4900-8e98-36a62e68c247,launched_at=None,metadata={Type='work'},migration_status=None,multiattach=False,previous_status=None,project_id='e330412972fa44b9bf60cd9afe4399cd',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:13:36Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:13:36Z,use_quota=True,user_id='e14688d19f1b44928fa8aceb2da0ac82',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 16:15:54 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:11:18Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesGetTestJSON-Volume-1455502637',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=0a7d4209-92e0-4900-8e98-36a62e68c247,launched_at=None,metadata={Type='work'},migration_status=None,multiattach=False,previous_status=None,project_id='e330412972fa44b9bf60cd9afe4399cd',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:13:36Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:13:36Z,use_quota=True,user_id='e14688d19f1b44928fa8aceb2da0ac82',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:11:18Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesGetTestJSON-Volume-1455502637',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=0a7d4209-92e0-4900-8e98-36a62e68c247,launched_at=None,metadata={Type='work'},migration_status=None,multiattach=False,previous_status=None,project_id='e330412972fa44b9bf60cd9afe4399cd',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:13:36Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:13:36Z,use_quota=True,user_id='e14688d19f1b44928fa8aceb2da0ac82',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 16:15:54 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 16:15:54 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 16:15:54 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:15:54 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 16:15:54 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:15:54 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 16:15:54 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:15:54 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 16:15:54 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:15:54 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 16:15:54 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:15:54 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 16:15:54 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:15:54 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 16:15:54 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:15:54 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 16:15:54 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:15:54 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:15:54 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 16:15:54 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 16:15:54 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (fbe14b45-ed1a-4d2c-b25b-03f9c9df58ae) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:15:54 user cinder-volume[84696]: WARNING cinder.volume.manager [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (fbe14b45-ed1a-4d2c-b25b-03f9c9df58ae) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:15:54 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (cea38384-7ce8-4fb9-9afc-f90b5758584d) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:15:54 user cinder-volume[84696]: WARNING cinder.volume.manager [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (cea38384-7ce8-4fb9-9afc-f90b5758584d) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:15:54 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (c898e4fb-b6e1-4616-adde-a84aa62ee9bc) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:15:54 user cinder-volume[84696]: WARNING cinder.volume.manager [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (c898e4fb-b6e1-4616-adde-a84aa62ee9bc) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:15:54 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (19afd6eb-c20b-49ac-9713-0aa108e20290) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:15:54 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Updating volume 0a7d4209-92e0-4900-8e98-36a62e68c247 with {'scheduled_at': datetime.datetime(2023, 4, 17, 16, 15, 54, 805390), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 16:15:54 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Volume 0a7d4209-92e0-4900-8e98-36a62e68c247: re-scheduling SchedulerAPI.create_volume attempt 2 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 16:15:54 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Volume 49effcda-e787-4983-92b2-f8261052732d: re-scheduling SchedulerAPI.create_volume attempt 3 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 16:15:54 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Volume 0a7d4209-92e0-4900-8e98-36a62e68c247: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 16:15:54 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] deleating volume v_0a7d4209-92e0-4900-8e98-36a62e68c247 {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 16:15:54 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] get snapshots for volume v_0a7d4209-92e0-4900-8e98-36a62e68c247 {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 16:15:54 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_0a7d4209-92e0-4900-8e98-36a62e68c247/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:15:54 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_0a7d4209-92e0-4900-8e98-36a62e68c247/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:15:54 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Volume 49effcda-e787-4983-92b2-f8261052732d: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 16:15:54 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] deleating volume v_49effcda-e787-4983-92b2-f8261052732d {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 16:15:54 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] get snapshots for volume v_49effcda-e787-4983-92b2-f8261052732d {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 16:15:54 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_49effcda-e787-4983-92b2-f8261052732d/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:15:54 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_49effcda-e787-4983-92b2-f8261052732d/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:15:54 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Volume reschedule parameters: True retry: {'num_attempts': 3, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 16:15:55 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Flow 'volume_create_manager' (2c280583-ddcd-4eb7-9be2-f7658b26a36f) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 16:15:55 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (3f23333b-0a38-4265-81bf-f9f7aa177600) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:15:55 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (3f23333b-0a38-4265-81bf-f9f7aa177600) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:11:18Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesGetTestJSON-Volume-1455502637',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=0a7d4209-92e0-4900-8e98-36a62e68c247,launched_at=None,metadata={Type='work'},migration_status=None,multiattach=False,previous_status=None,project_id='e330412972fa44b9bf60cd9afe4399cd',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:15:55Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:15:55Z,use_quota=True,user_id='e14688d19f1b44928fa8aceb2da0ac82',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:15:55 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (635d913b-fca8-4373-98cf-065284347dae) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:15:55 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (635d913b-fca8-4373-98cf-065284347dae) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:15:55 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (50176311-b437-463c-afe1-be6646108689) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:15:55 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (50176311-b437-463c-afe1-be6646108689) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': '0a7d4209-92e0-4900-8e98-36a62e68c247', 'volume_name': 'volume-0a7d4209-92e0-4900-8e98-36a62e68c247', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:15:55 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (d5a6989b-6ec3-4d84-b2d7-b5c3b7972f77) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:15:55 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (d5a6989b-6ec3-4d84-b2d7-b5c3b7972f77) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:15:55 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (d9b013a5-8ded-4da0-aceb-4e386c9783d5) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:15:55 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Volume 0a7d4209-92e0-4900-8e98-36a62e68c247: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-0a7d4209-92e0-4900-8e98-36a62e68c247', 'volume_size': 1} Apr 17 16:15:55 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] creating volume v_0a7d4209-92e0-4900-8e98-36a62e68c247. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 16:15:55 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] create volume {'name': 'v_0a7d4209-92e0-4900-8e98-36a62e68c247', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 16:15:55 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:15:55 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:15:57 user cinder-volume[84696]: DEBUG cinder.utils [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.848(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:15:57 user cinder-volume[84696]: DEBUG cinder.utils [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_0a7d4209-92e0-4900-8e98-36a62e68c247/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:15:57 user cinder-volume[84696]: DEBUG cinder.utils [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.615(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:15:57 user cinder-volume[84696]: DEBUG cinder.utils [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:15:57 user cinder-volume[84696]: DEBUG cinder.utils [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.849(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:15:57 user cinder-volume[84696]: DEBUG cinder.utils [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_49effcda-e787-4983-92b2-f8261052732d/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:16:00 user cinder-volume[84696]: DEBUG cinder.utils [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.916(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:16:00 user cinder-volume[84696]: DEBUG cinder.utils [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_49effcda-e787-4983-92b2-f8261052732d/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:16:00 user cinder-volume[84696]: DEBUG cinder.utils [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 71.659(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:16:00 user cinder-volume[84696]: DEBUG cinder.utils [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:16:00 user cinder-volume[84696]: DEBUG cinder.utils [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.924(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:16:00 user cinder-volume[84696]: DEBUG cinder.utils [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_0a7d4209-92e0-4900-8e98-36a62e68c247/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:16:00 user cinder-volume[84696]: DEBUG cinder.utils [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.689(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:16:00 user cinder-volume[84696]: DEBUG cinder.utils [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:16:03 user cinder-volume[84696]: DEBUG cinder.utils [req-bb0706f3-0b2c-473a-acc5-93c2ac742a5f req-0a3ae49c-837d-4b32-a241-6c27a7c860b1 tempest-ServerStableDeviceRescueTest-869290889 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.199(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:16:03 user cinder-volume[84696]: DEBUG cinder.utils [req-bb0706f3-0b2c-473a-acc5-93c2ac742a5f req-0a3ae49c-837d-4b32-a241-6c27a7c860b1 tempest-ServerStableDeviceRescueTest-869290889 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:32819ec1-2236-465f-a1c7-ee31aa4a0911/luns/v_32819ec1-2236-465f-a1c7-ee31aa4a0911 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:16:03 user cinder-volume[84696]: DEBUG cinder.utils [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 139.083(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:16:03 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_85fcb006-cd96-4018-b0e3-04872ff08267/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:16:03 user cinder-volume[84696]: WARNING cinder.volume.manager [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (eb867ed7-2620-4fdf-9e2c-6a76b6a3779a) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 16:16:03 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (9dc28f78-d7a1-4c45-9145-3b3ea2bffdb5) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:16:03 user cinder-volume[84696]: WARNING cinder.volume.manager [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (9dc28f78-d7a1-4c45-9145-3b3ea2bffdb5) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:16:03 user cinder-volume[84696]: WARNING cinder.volume.manager [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Flow 'volume_create_manager' (2c5d7b4e-4249-4979-b15e-0da0ce7eb111) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 16:16:03 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 16:16:03 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 16:16:03 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 16:16:03 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 16:16:03 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 16:16:03 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 16:16:03 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 16:16:03 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 16:16:03 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 16:16:03 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 16:16:03 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 16:16:03 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 16:16:03 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 16:16:03 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 16:16:03 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 16:16:03 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 16:16:03 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 16:16:03 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 16:16:03 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 16:16:03 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 16:16:03 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 16:16:03 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 16:16:03 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 16:16:03 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 16:16:03 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:16:03 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 16:16:03 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:16:03 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 16:16:03 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:16:03 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 16:16:03 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:16:03 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 16:16:03 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:16:03 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 16:16:03 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:16:03 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 16:16:03 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:16:03 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 16:16:03 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:16:03 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:16:03 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 16:16:03 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 16:16:03 user cinder-volume[84696]: DEBUG cinder.utils [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 138.935(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:16:03 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:16:03 user cinder-volume[84696]: INFO cinder.message.api [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Creating message record for request_id = req-726a1377-a691-4c8a-a56d-8b87aed706f7 Apr 17 16:16:03 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 16:16:03 user cinder-volume[84696]: WARNING cinder.volume.manager [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (9bb5f497-fdef-4e47-9975-64fad5b22577) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 16:16:03 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 16:16:03 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:11:28Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesSnapshotsTestJSON-volume-126016572',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=85fcb006-cd96-4018-b0e3-04872ff08267,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='34ab34dfa2e14c3785276a62a076f8e8',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:13:45Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:13:45Z,use_quota=True,user_id='c98520ee1d2c4eabaf6c186bd3ae7fee',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 16:16:03 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:11:28Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesSnapshotsTestJSON-volume-126016572',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=85fcb006-cd96-4018-b0e3-04872ff08267,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='34ab34dfa2e14c3785276a62a076f8e8',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:13:45Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:13:45Z,use_quota=True,user_id='c98520ee1d2c4eabaf6c186bd3ae7fee',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(85fcb006-cd96-4018-b0e3-04872ff08267),volume_id=85fcb006-cd96-4018-b0e3-04872ff08267,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': '85fcb006-cd96-4018-b0e3-04872ff08267', 'volume_name': 'volume-85fcb006-cd96-4018-b0e3-04872ff08267', 'volume_size': 1}} Apr 17 16:16:03 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': '85fcb006-cd96-4018-b0e3-04872ff08267', 'volume': {'id': '85fcb006-cd96-4018-b0e3-04872ff08267', '_name_id': None, 'ec2_id': None, 'user_id': 'c98520ee1d2c4eabaf6c186bd3ae7fee', 'project_id': '34ab34dfa2e14c3785276a62a076f8e8', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-VolumesSnapshotsTestJSON-volume-126016572', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'admin_metadata': {}, 'glance_metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T16:11:28.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-85fcb006-cd96-4018-b0e3-04872ff08267', 'name_id': '85fcb006-cd96-4018-b0e3-04872ff08267', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumesSnapshotsTestJSON-volume-126016572', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '34ab34dfa2e14c3785276a62a076f8e8', 'qos_specs': None, 'reservations': ['68a86b1f-7431-4aa0-8593-4b26c93f38d5', 'e3c2cd32-910b-402d-b512-12abf2966403', '3f9aa9b2-3b03-45b0-ab25-91483149a902', 'f37d20df-0298-46bc-8806-f2f882d834ac'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'c98520ee1d2c4eabaf6c186bd3ae7fee', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumesSnapshotsTestJSON-volume-126016572', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '34ab34dfa2e14c3785276a62a076f8e8', 'qos_specs': None, 'reservations': ['68a86b1f-7431-4aa0-8593-4b26c93f38d5', 'e3c2cd32-910b-402d-b512-12abf2966403', '3f9aa9b2-3b03-45b0-ab25-91483149a902', 'f37d20df-0298-46bc-8806-f2f882d834ac'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'c98520ee1d2c4eabaf6c186bd3ae7fee', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': 'c98520ee1d2c4eabaf6c186bd3ae7fee', 'metadata': {}, 'qos_specs': None, 'retry': {'num_attempts': 2, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']}}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(85fcb006-cd96-4018-b0e3-04872ff08267),volume_id=85fcb006-cd96-4018-b0e3-04872ff08267,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:11:28Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesSnapshotsTestJSON-volume-126016572',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=85fcb006-cd96-4018-b0e3-04872ff08267,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='34ab34dfa2e14c3785276a62a076f8e8',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:13:45Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:13:45Z,use_quota=True,user_id='c98520ee1d2c4eabaf6c186bd3ae7fee',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 16:16:03 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:11:28Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesSnapshotsTestJSON-volume-126016572',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=85fcb006-cd96-4018-b0e3-04872ff08267,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='34ab34dfa2e14c3785276a62a076f8e8',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:13:45Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:13:45Z,use_quota=True,user_id='c98520ee1d2c4eabaf6c186bd3ae7fee',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:11:28Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesSnapshotsTestJSON-volume-126016572',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=85fcb006-cd96-4018-b0e3-04872ff08267,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='34ab34dfa2e14c3785276a62a076f8e8',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:13:45Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:13:45Z,use_quota=True,user_id='c98520ee1d2c4eabaf6c186bd3ae7fee',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 16:16:03 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 16:16:03 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 16:16:03 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:16:03 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 16:16:03 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:16:03 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 16:16:03 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:16:03 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 16:16:03 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:16:03 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 16:16:03 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:16:03 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 16:16:03 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:16:03 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 16:16:03 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:16:03 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 16:16:03 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:16:03 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:16:03 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 16:16:03 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 16:16:03 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (9bb5f497-fdef-4e47-9975-64fad5b22577) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:16:03 user cinder-volume[84696]: WARNING cinder.volume.manager [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (9bb5f497-fdef-4e47-9975-64fad5b22577) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:16:03 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (31b3d485-41c0-46f0-a5de-2a8f06153e0f) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:16:03 user cinder-volume[84696]: WARNING cinder.volume.manager [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (31b3d485-41c0-46f0-a5de-2a8f06153e0f) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:16:03 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (a625ba15-a535-4855-8603-aab98456a989) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:16:03 user cinder-volume[84696]: WARNING cinder.volume.manager [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (a625ba15-a535-4855-8603-aab98456a989) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:16:03 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (8f41e083-88b2-4f7e-8f3f-e49bca1072d7) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:16:03 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Updating volume 85fcb006-cd96-4018-b0e3-04872ff08267 with {'scheduled_at': datetime.datetime(2023, 4, 17, 16, 16, 3, 913317), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 16:16:03 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Volume 85fcb006-cd96-4018-b0e3-04872ff08267: re-scheduling SchedulerAPI.create_volume attempt 2 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 16:16:03 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Volume 85fcb006-cd96-4018-b0e3-04872ff08267: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 16:16:03 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] deleating volume v_85fcb006-cd96-4018-b0e3-04872ff08267 {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 16:16:03 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] get snapshots for volume v_85fcb006-cd96-4018-b0e3-04872ff08267 {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 16:16:03 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_85fcb006-cd96-4018-b0e3-04872ff08267/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:16:03 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_85fcb006-cd96-4018-b0e3-04872ff08267/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:16:04 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Volume reschedule parameters: True retry: {'num_attempts': 3, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 16:16:04 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Flow 'volume_create_manager' (7554f208-702b-4fcd-840a-a9924da76383) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 16:16:04 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (8fcb7af7-c087-4742-a88f-580b970d9d51) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:16:04 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (8fcb7af7-c087-4742-a88f-580b970d9d51) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:11:28Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesSnapshotsTestJSON-volume-126016572',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=85fcb006-cd96-4018-b0e3-04872ff08267,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='34ab34dfa2e14c3785276a62a076f8e8',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:16:04Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:16:04Z,use_quota=True,user_id='c98520ee1d2c4eabaf6c186bd3ae7fee',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:16:04 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (b405f70a-8735-4c3f-9e23-c81972c36950) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:16:04 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (b405f70a-8735-4c3f-9e23-c81972c36950) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:16:04 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (0bc1b9f8-cfac-428e-98c1-ad90e610bf92) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:16:04 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (0bc1b9f8-cfac-428e-98c1-ad90e610bf92) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': '85fcb006-cd96-4018-b0e3-04872ff08267', 'volume_name': 'volume-85fcb006-cd96-4018-b0e3-04872ff08267', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:16:04 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (b3f1f466-baa8-4acf-b117-af167237b331) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:16:04 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (b3f1f466-baa8-4acf-b117-af167237b331) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:16:04 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (b1267986-f5a0-443c-b679-5606c00d9d41) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:16:04 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Volume 85fcb006-cd96-4018-b0e3-04872ff08267: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-85fcb006-cd96-4018-b0e3-04872ff08267', 'volume_size': 1} Apr 17 16:16:04 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] creating volume v_85fcb006-cd96-4018-b0e3-04872ff08267. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 16:16:04 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] create volume {'name': 'v_85fcb006-cd96-4018-b0e3-04872ff08267', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 16:16:04 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:16:04 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:16:06 user cinder-volume[84696]: DEBUG cinder.utils [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.971(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:16:06 user cinder-volume[84696]: DEBUG cinder.utils [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_85fcb006-cd96-4018-b0e3-04872ff08267/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:16:06 user cinder-volume[84696]: DEBUG cinder.utils [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.814(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:16:06 user cinder-volume[84696]: DEBUG cinder.utils [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:16:06 user cinder-volume[84696]: DEBUG cinder.utils [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.067(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:16:06 user cinder-volume[84696]: DEBUG cinder.utils [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_0a7d4209-92e0-4900-8e98-36a62e68c247/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:16:06 user cinder-volume[84696]: DEBUG cinder.utils [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 11.833(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:16:06 user cinder-volume[84696]: DEBUG cinder.utils [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:16:06 user cinder-volume[84696]: DEBUG cinder.utils [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.066(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:16:06 user cinder-volume[84696]: DEBUG cinder.utils [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_49effcda-e787-4983-92b2-f8261052732d/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:16:09 user cinder-volume[84696]: DEBUG cinder.utils [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.043(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:16:09 user cinder-volume[84696]: DEBUG cinder.utils [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_85fcb006-cd96-4018-b0e3-04872ff08267/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:16:09 user cinder-volume[84696]: DEBUG cinder.utils [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.885(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:16:09 user cinder-volume[84696]: DEBUG cinder.utils [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:16:13 user cinder-volume[84696]: DEBUG cinder.utils [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 38.416(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:16:13 user cinder-volume[84696]: DEBUG cinder.utils [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:16:16 user cinder-volume[84696]: DEBUG cinder.utils [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.045(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:16:16 user cinder-volume[84696]: DEBUG cinder.utils [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:16:16 user cinder-volume[84696]: DEBUG cinder.utils [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.282(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:16:16 user cinder-volume[84696]: DEBUG cinder.utils [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_0a7d4209-92e0-4900-8e98-36a62e68c247/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:16:16 user cinder-volume[84696]: DEBUG cinder.utils [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.031(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:16:16 user cinder-volume[84696]: DEBUG cinder.utils [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:16:16 user cinder-volume[84696]: DEBUG cinder.utils [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.280(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:16:16 user cinder-volume[84696]: DEBUG cinder.utils [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_49effcda-e787-4983-92b2-f8261052732d/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:16:16 user cinder-volume[84696]: DEBUG cinder.utils [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.193(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:16:16 user cinder-volume[84696]: DEBUG cinder.utils [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_85fcb006-cd96-4018-b0e3-04872ff08267/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:16:24 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=16,cinder:INSERT=4,cinder:UPDATE=5 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:16:24 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:INSERT=2,cinder:SELECT=16,cinder:UPDATE=6 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:16:25 user cinder-volume[84696]: DEBUG cinder.utils [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.727(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:16:25 user cinder-volume[84696]: DEBUG cinder.utils [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_85fcb006-cd96-4018-b0e3-04872ff08267/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:16:25 user cinder-volume[84696]: DEBUG cinder.utils [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.570(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:16:25 user cinder-volume[84696]: DEBUG cinder.utils [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:16:25 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 74.049(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:16:25 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:16:35 user cinder-volume[84696]: DEBUG cinder.utils [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 139.701(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:16:35 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_bf4c0b13-49c6-4b1d-b790-7090eb82d4c4/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:16:35 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (796c9884-6828-42aa-af51-26e3f4a7894b) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 16:16:35 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (5967b5df-f73b-4f24-8f7c-ecba2ea1601d) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:16:35 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (5967b5df-f73b-4f24-8f7c-ecba2ea1601d) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:16:35 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Flow 'volume_create_manager' (4168c4f1-1544-4b91-aba3-a1021cbaea61) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 16:16:35 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 16:16:35 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 16:16:35 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 16:16:35 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 16:16:35 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 16:16:35 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 16:16:35 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 16:16:35 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 16:16:35 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 16:16:35 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 16:16:35 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 16:16:35 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 16:16:35 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 16:16:35 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 16:16:35 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 16:16:35 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 16:16:35 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 16:16:35 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 16:16:35 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 16:16:35 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 16:16:35 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 16:16:35 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 16:16:35 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 16:16:35 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 16:16:35 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:16:35 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 16:16:35 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:16:35 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 16:16:35 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:16:35 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 16:16:35 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:16:35 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 16:16:35 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:16:35 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 16:16:35 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:16:35 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 16:16:35 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:16:35 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 16:16:35 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:16:35 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:16:35 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 16:16:35 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 16:16:35 user cinder-volume[84696]: DEBUG cinder.utils [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 40.137(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:16:35 user cinder-volume[84696]: DEBUG cinder.utils [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:16:35 user cinder-volume[84696]: DEBUG cinder.utils [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 40.370(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:16:35 user cinder-volume[84696]: DEBUG cinder.utils [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_49effcda-e787-4983-92b2-f8261052732d/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:16:35 user cinder-volume[84696]: DEBUG cinder.utils [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 40.376(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:16:35 user cinder-volume[84696]: DEBUG cinder.utils [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_0a7d4209-92e0-4900-8e98-36a62e68c247/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:16:35 user cinder-volume[84696]: DEBUG cinder.utils [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 139.556(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:16:35 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:16:35 user cinder-volume[84696]: INFO cinder.message.api [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Creating message record for request_id = req-e76412e0-459f-4be5-bc08-e01e70270c82 Apr 17 16:16:35 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 16:16:35 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (fac9db02-eeb4-4c3c-8b2b-52719d146993) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 16:16:35 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 16:16:35 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:11:57Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-AttachVolumeTestJSON-volume-1944488700',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=bf4c0b13-49c6-4b1d-b790-7090eb82d4c4,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='2eb998c84b5b41e599af9636b929295d',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:14:16Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:14:16Z,use_quota=True,user_id='b9d4a317838b47d6a2a2e7eab7a453dd',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 16:16:35 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:11:57Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-AttachVolumeTestJSON-volume-1944488700',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=bf4c0b13-49c6-4b1d-b790-7090eb82d4c4,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='2eb998c84b5b41e599af9636b929295d',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:14:16Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:14:16Z,use_quota=True,user_id='b9d4a317838b47d6a2a2e7eab7a453dd',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(bf4c0b13-49c6-4b1d-b790-7090eb82d4c4),volume_id=bf4c0b13-49c6-4b1d-b790-7090eb82d4c4,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': 'bf4c0b13-49c6-4b1d-b790-7090eb82d4c4', 'volume_name': 'volume-bf4c0b13-49c6-4b1d-b790-7090eb82d4c4', 'volume_size': 1}} Apr 17 16:16:35 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': 'bf4c0b13-49c6-4b1d-b790-7090eb82d4c4', 'volume': {'id': 'bf4c0b13-49c6-4b1d-b790-7090eb82d4c4', '_name_id': None, 'ec2_id': None, 'user_id': 'b9d4a317838b47d6a2a2e7eab7a453dd', 'project_id': '2eb998c84b5b41e599af9636b929295d', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-AttachVolumeTestJSON-volume-1944488700', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'admin_metadata': {}, 'glance_metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T16:11:57.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-bf4c0b13-49c6-4b1d-b790-7090eb82d4c4', 'name_id': 'bf4c0b13-49c6-4b1d-b790-7090eb82d4c4', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-AttachVolumeTestJSON-volume-1944488700', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '2eb998c84b5b41e599af9636b929295d', 'qos_specs': None, 'reservations': ['b181be02-433e-4478-a0ce-e1f2f6eeae64', 'fe95550b-8f6c-452e-b791-feeedb2022c3', '19f55cd7-af23-4f74-9f0b-5af38cf186a9', '2ba006a2-24af-4c11-bba8-0fe8459bb371'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'b9d4a317838b47d6a2a2e7eab7a453dd', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-AttachVolumeTestJSON-volume-1944488700', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '2eb998c84b5b41e599af9636b929295d', 'qos_specs': None, 'reservations': ['b181be02-433e-4478-a0ce-e1f2f6eeae64', 'fe95550b-8f6c-452e-b791-feeedb2022c3', '19f55cd7-af23-4f74-9f0b-5af38cf186a9', '2ba006a2-24af-4c11-bba8-0fe8459bb371'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'b9d4a317838b47d6a2a2e7eab7a453dd', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': 'b9d4a317838b47d6a2a2e7eab7a453dd', 'metadata': {}, 'qos_specs': None, 'retry': {'num_attempts': 2, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']}}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(bf4c0b13-49c6-4b1d-b790-7090eb82d4c4),volume_id=bf4c0b13-49c6-4b1d-b790-7090eb82d4c4,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:11:57Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-AttachVolumeTestJSON-volume-1944488700',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=bf4c0b13-49c6-4b1d-b790-7090eb82d4c4,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='2eb998c84b5b41e599af9636b929295d',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:14:16Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:14:16Z,use_quota=True,user_id='b9d4a317838b47d6a2a2e7eab7a453dd',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 16:16:35 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:11:57Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-AttachVolumeTestJSON-volume-1944488700',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=bf4c0b13-49c6-4b1d-b790-7090eb82d4c4,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='2eb998c84b5b41e599af9636b929295d',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:14:16Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:14:16Z,use_quota=True,user_id='b9d4a317838b47d6a2a2e7eab7a453dd',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:11:57Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-AttachVolumeTestJSON-volume-1944488700',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=bf4c0b13-49c6-4b1d-b790-7090eb82d4c4,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='2eb998c84b5b41e599af9636b929295d',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:14:16Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:14:16Z,use_quota=True,user_id='b9d4a317838b47d6a2a2e7eab7a453dd',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 16:16:35 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 16:16:35 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 16:16:35 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:16:35 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 16:16:35 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:16:35 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 16:16:35 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:16:35 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 16:16:35 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:16:35 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 16:16:35 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:16:35 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 16:16:35 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:16:35 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 16:16:35 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:16:35 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 16:16:35 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:16:35 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:16:35 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 16:16:35 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 16:16:35 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (fac9db02-eeb4-4c3c-8b2b-52719d146993) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:16:35 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (fac9db02-eeb4-4c3c-8b2b-52719d146993) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:16:35 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (465369c0-d203-478d-a929-0eb4d44505a5) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:16:35 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (465369c0-d203-478d-a929-0eb4d44505a5) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:16:35 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (431d0e7d-b66f-4827-bf8d-40d053555488) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:16:35 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (431d0e7d-b66f-4827-bf8d-40d053555488) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:16:35 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (814a3795-e586-4bcc-bc41-e322e51eb5a8) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:16:35 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Updating volume bf4c0b13-49c6-4b1d-b790-7090eb82d4c4 with {'scheduled_at': datetime.datetime(2023, 4, 17, 16, 16, 35, 309415), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 16:16:35 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Volume bf4c0b13-49c6-4b1d-b790-7090eb82d4c4: re-scheduling SchedulerAPI.create_volume attempt 2 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 16:16:35 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Volume bf4c0b13-49c6-4b1d-b790-7090eb82d4c4: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 16:16:35 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] deleating volume v_bf4c0b13-49c6-4b1d-b790-7090eb82d4c4 {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 16:16:35 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] get snapshots for volume v_bf4c0b13-49c6-4b1d-b790-7090eb82d4c4 {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 16:16:35 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_bf4c0b13-49c6-4b1d-b790-7090eb82d4c4/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:16:35 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_bf4c0b13-49c6-4b1d-b790-7090eb82d4c4/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:16:35 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Volume reschedule parameters: True retry: {'num_attempts': 3, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 16:16:35 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Flow 'volume_create_manager' (9806d060-778d-4f0e-b2ac-5470f8402414) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 16:16:35 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (a357ad05-6797-4829-b8ab-6b540d6dbeaa) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:16:35 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (a357ad05-6797-4829-b8ab-6b540d6dbeaa) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:11:57Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-AttachVolumeTestJSON-volume-1944488700',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=bf4c0b13-49c6-4b1d-b790-7090eb82d4c4,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='2eb998c84b5b41e599af9636b929295d',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:16:35Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:16:35Z,use_quota=True,user_id='b9d4a317838b47d6a2a2e7eab7a453dd',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:16:35 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (f8fb11c9-25cc-49cc-b72d-e79139f45094) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:16:35 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (f8fb11c9-25cc-49cc-b72d-e79139f45094) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:16:35 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (6714ee39-d859-45a1-b9b3-d363ebb6651b) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:16:35 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (6714ee39-d859-45a1-b9b3-d363ebb6651b) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': 'bf4c0b13-49c6-4b1d-b790-7090eb82d4c4', 'volume_name': 'volume-bf4c0b13-49c6-4b1d-b790-7090eb82d4c4', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:16:35 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (c94acc5c-7880-4a5f-a975-25ac889b3c98) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:16:35 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (c94acc5c-7880-4a5f-a975-25ac889b3c98) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:16:35 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (99b95cf0-ece7-459f-8846-6179d684ad4e) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:16:35 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Volume bf4c0b13-49c6-4b1d-b790-7090eb82d4c4: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-bf4c0b13-49c6-4b1d-b790-7090eb82d4c4', 'volume_size': 1} Apr 17 16:16:35 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] creating volume v_bf4c0b13-49c6-4b1d-b790-7090eb82d4c4. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 16:16:35 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] create volume {'name': 'v_bf4c0b13-49c6-4b1d-b790-7090eb82d4c4', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 16:16:35 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:16:35 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:16:38 user cinder-volume[84696]: DEBUG cinder.utils [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.746(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:16:38 user cinder-volume[84696]: DEBUG cinder.utils [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:16:38 user cinder-volume[84696]: DEBUG cinder.utils [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.911(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:16:38 user cinder-volume[84696]: DEBUG cinder.utils [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_bf4c0b13-49c6-4b1d-b790-7090eb82d4c4/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:16:38 user cinder-volume[84696]: DEBUG cinder.utils [req-bb0706f3-0b2c-473a-acc5-93c2ac742a5f req-0a3ae49c-837d-4b32-a241-6c27a7c860b1 tempest-ServerStableDeviceRescueTest-869290889 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 73.636(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:16:38 user cinder-volume[84696]: DEBUG cinder.utils [req-bb0706f3-0b2c-473a-acc5-93c2ac742a5f req-0a3ae49c-837d-4b32-a241-6c27a7c860b1 tempest-ServerStableDeviceRescueTest-869290889 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:32819ec1-2236-465f-a1c7-ee31aa4a0911/luns/v_32819ec1-2236-465f-a1c7-ee31aa4a0911 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:16:41 user cinder-volume[84696]: DEBUG cinder.utils [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.818(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:16:41 user cinder-volume[84696]: DEBUG cinder.utils [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:16:41 user cinder-volume[84696]: DEBUG cinder.utils [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.981(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:16:41 user cinder-volume[84696]: DEBUG cinder.utils [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_bf4c0b13-49c6-4b1d-b790-7090eb82d4c4/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:16:44 user cinder-volume[84696]: DEBUG cinder.utils [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 40.476(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:16:44 user cinder-volume[84696]: DEBUG cinder.utils [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_85fcb006-cd96-4018-b0e3-04872ff08267/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:16:44 user cinder-volume[84696]: DEBUG cinder.utils [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 40.317(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:16:44 user cinder-volume[84696]: DEBUG cinder.utils [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:16:47 user cinder-volume[84696]: DEBUG cinder.utils [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 72.844(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:16:47 user cinder-volume[84696]: DEBUG cinder.utils [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:16:47 user cinder-volume[84696]: DEBUG cinder.utils [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.126(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:16:47 user cinder-volume[84696]: DEBUG cinder.utils [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_bf4c0b13-49c6-4b1d-b790-7090eb82d4c4/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:16:47 user cinder-volume[84696]: DEBUG cinder.utils [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 11.965(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:16:47 user cinder-volume[84696]: DEBUG cinder.utils [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:16:50 user cinder-volume[84696]: DEBUG cinder.utils [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 136.487(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:16:50 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:16:50 user cinder-volume[84696]: INFO cinder.message.api [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Creating message record for request_id = req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc Apr 17 16:16:50 user cinder-volume[84696]: DEBUG cinder.utils [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 136.650(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:16:50 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_125cc0cd-98dc-41a8-a252-3ae03797fb6f/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:16:50 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (b576dc36-ce9c-49c5-adf5-5b1f237b943a) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 16:16:50 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (12dd2903-492e-4ff3-a822-b7b3de012387) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:16:50 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (12dd2903-492e-4ff3-a822-b7b3de012387) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:16:50 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Flow 'volume_create_manager' (b2538276-308a-4b31-bb9e-95fe32632134) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 16:16:50 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 16:16:50 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 16:16:50 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 16:16:50 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 16:16:50 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 16:16:50 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 16:16:50 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 16:16:50 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 16:16:50 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 16:16:50 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 16:16:50 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 16:16:50 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 16:16:50 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 16:16:50 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 16:16:50 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 16:16:50 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 16:16:50 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 16:16:50 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 16:16:50 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 16:16:50 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 16:16:50 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 16:16:50 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 16:16:50 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 16:16:50 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 16:16:50 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:16:50 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 16:16:50 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:16:50 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 16:16:50 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:16:50 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 16:16:50 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:16:50 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 16:16:50 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:16:50 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 16:16:50 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:16:50 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 16:16:50 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:16:50 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 16:16:50 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:16:50 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:16:50 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 16:16:50 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 16:16:50 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 16:16:50 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (6f7e4a34-1280-466d-ac2c-ccdf977b421e) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 16:16:50 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 16:16:50 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:12:14Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesTransfersV355Test-Volume-94958465',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=125cc0cd-98dc-41a8-a252-3ae03797fb6f,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='9796cc8ccc334a5b86ff098b10a0c413',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:14:34Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:14:34Z,use_quota=True,user_id='de05a598447a4e9fb3ff488ec2d84071',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 16:16:50 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:12:14Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesTransfersV355Test-Volume-94958465',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=125cc0cd-98dc-41a8-a252-3ae03797fb6f,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='9796cc8ccc334a5b86ff098b10a0c413',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:14:34Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:14:34Z,use_quota=True,user_id='de05a598447a4e9fb3ff488ec2d84071',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(125cc0cd-98dc-41a8-a252-3ae03797fb6f),volume_id=125cc0cd-98dc-41a8-a252-3ae03797fb6f,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': '125cc0cd-98dc-41a8-a252-3ae03797fb6f', 'volume_name': 'volume-125cc0cd-98dc-41a8-a252-3ae03797fb6f', 'volume_size': 1}} Apr 17 16:16:50 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': '125cc0cd-98dc-41a8-a252-3ae03797fb6f', 'volume': {'id': '125cc0cd-98dc-41a8-a252-3ae03797fb6f', '_name_id': None, 'ec2_id': None, 'user_id': 'de05a598447a4e9fb3ff488ec2d84071', 'project_id': '9796cc8ccc334a5b86ff098b10a0c413', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-VolumesTransfersV355Test-Volume-94958465', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'admin_metadata': {}, 'glance_metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T16:12:13.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-125cc0cd-98dc-41a8-a252-3ae03797fb6f', 'name_id': '125cc0cd-98dc-41a8-a252-3ae03797fb6f', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumesTransfersV355Test-Volume-94958465', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '9796cc8ccc334a5b86ff098b10a0c413', 'qos_specs': None, 'reservations': ['a45788dc-5f7c-49c5-baac-3ecbd8da3ffc', 'ede61d0b-a2d4-4097-88bb-c2d0d901d083', '70e9f0f6-daa6-48d0-a906-1bcc00ee96d7', 'b04070c8-a6ec-4747-a48e-c741818cad1d'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'de05a598447a4e9fb3ff488ec2d84071', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumesTransfersV355Test-Volume-94958465', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '9796cc8ccc334a5b86ff098b10a0c413', 'qos_specs': None, 'reservations': ['a45788dc-5f7c-49c5-baac-3ecbd8da3ffc', 'ede61d0b-a2d4-4097-88bb-c2d0d901d083', '70e9f0f6-daa6-48d0-a906-1bcc00ee96d7', 'b04070c8-a6ec-4747-a48e-c741818cad1d'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'de05a598447a4e9fb3ff488ec2d84071', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': 'de05a598447a4e9fb3ff488ec2d84071', 'metadata': {}, 'qos_specs': None, 'retry': {'num_attempts': 2, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']}}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(125cc0cd-98dc-41a8-a252-3ae03797fb6f),volume_id=125cc0cd-98dc-41a8-a252-3ae03797fb6f,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:12:14Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesTransfersV355Test-Volume-94958465',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=125cc0cd-98dc-41a8-a252-3ae03797fb6f,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='9796cc8ccc334a5b86ff098b10a0c413',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:14:34Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:14:34Z,use_quota=True,user_id='de05a598447a4e9fb3ff488ec2d84071',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 16:16:50 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:12:14Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesTransfersV355Test-Volume-94958465',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=125cc0cd-98dc-41a8-a252-3ae03797fb6f,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='9796cc8ccc334a5b86ff098b10a0c413',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:14:34Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:14:34Z,use_quota=True,user_id='de05a598447a4e9fb3ff488ec2d84071',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:12:14Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesTransfersV355Test-Volume-94958465',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=125cc0cd-98dc-41a8-a252-3ae03797fb6f,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='9796cc8ccc334a5b86ff098b10a0c413',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:14:34Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:14:34Z,use_quota=True,user_id='de05a598447a4e9fb3ff488ec2d84071',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 16:16:50 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 16:16:50 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 16:16:50 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:16:50 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 16:16:50 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:16:50 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 16:16:50 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:16:50 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 16:16:50 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:16:50 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 16:16:50 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:16:50 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 16:16:50 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:16:50 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 16:16:50 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:16:50 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 16:16:50 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:16:50 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:16:50 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 16:16:50 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 16:16:50 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (6f7e4a34-1280-466d-ac2c-ccdf977b421e) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:16:50 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (6f7e4a34-1280-466d-ac2c-ccdf977b421e) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:16:50 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (a605305a-8a58-477a-8eca-0d40f1f96fcc) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:16:50 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (a605305a-8a58-477a-8eca-0d40f1f96fcc) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:16:50 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (e0c30a52-1cbf-438a-9623-5050c5607675) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:16:50 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (e0c30a52-1cbf-438a-9623-5050c5607675) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:16:50 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (b41ac25a-a402-4ab8-92fa-6f3376c8c478) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:16:50 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Updating volume 125cc0cd-98dc-41a8-a252-3ae03797fb6f with {'scheduled_at': datetime.datetime(2023, 4, 17, 16, 16, 50, 639458), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 16:16:50 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Volume 125cc0cd-98dc-41a8-a252-3ae03797fb6f: re-scheduling SchedulerAPI.create_volume attempt 2 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 16:16:50 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Volume 125cc0cd-98dc-41a8-a252-3ae03797fb6f: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 16:16:50 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] deleating volume v_125cc0cd-98dc-41a8-a252-3ae03797fb6f {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 16:16:50 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] get snapshots for volume v_125cc0cd-98dc-41a8-a252-3ae03797fb6f {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 16:16:50 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_125cc0cd-98dc-41a8-a252-3ae03797fb6f/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:16:50 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_125cc0cd-98dc-41a8-a252-3ae03797fb6f/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:16:50 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Volume reschedule parameters: True retry: {'num_attempts': 3, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 16:16:50 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Flow 'volume_create_manager' (9b0fcca2-9468-40c5-b0ae-efc49b086cfc) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 16:16:50 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (d5d74489-1e5a-49f8-a0f0-4f1db3ebb077) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:16:50 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (d5d74489-1e5a-49f8-a0f0-4f1db3ebb077) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:12:14Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesTransfersV355Test-Volume-94958465',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=125cc0cd-98dc-41a8-a252-3ae03797fb6f,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='9796cc8ccc334a5b86ff098b10a0c413',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:16:51Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:16:51Z,use_quota=True,user_id='de05a598447a4e9fb3ff488ec2d84071',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:16:50 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (7e79f666-84e3-4fa2-b6f3-355d494345b0) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:16:50 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (7e79f666-84e3-4fa2-b6f3-355d494345b0) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:16:50 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (efd1ce56-2933-45b1-a352-89e6273c8a6d) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:16:50 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (efd1ce56-2933-45b1-a352-89e6273c8a6d) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': '125cc0cd-98dc-41a8-a252-3ae03797fb6f', 'volume_name': 'volume-125cc0cd-98dc-41a8-a252-3ae03797fb6f', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:16:50 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (6dc2e1e0-aa8c-44a7-9b3c-251a1af58781) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:16:50 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (6dc2e1e0-aa8c-44a7-9b3c-251a1af58781) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:16:50 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (7a706239-7838-42dc-9231-90cef3054ab0) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:16:50 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Volume 125cc0cd-98dc-41a8-a252-3ae03797fb6f: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-125cc0cd-98dc-41a8-a252-3ae03797fb6f', 'volume_size': 1} Apr 17 16:16:50 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] creating volume v_125cc0cd-98dc-41a8-a252-3ae03797fb6f. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 16:16:50 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] create volume {'name': 'v_125cc0cd-98dc-41a8-a252-3ae03797fb6f', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 16:16:50 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:16:50 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:16:53 user cinder-volume[84696]: DEBUG cinder.utils [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.777(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:16:53 user cinder-volume[84696]: DEBUG cinder.utils [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:16:53 user cinder-volume[84696]: DEBUG cinder.utils [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.962(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:16:53 user cinder-volume[84696]: DEBUG cinder.utils [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_125cc0cd-98dc-41a8-a252-3ae03797fb6f/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:16:56 user cinder-volume[84696]: DEBUG cinder.utils [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.178(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:16:56 user cinder-volume[84696]: DEBUG cinder.utils [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:16:56 user cinder-volume[84696]: DEBUG cinder.utils [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.341(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:16:56 user cinder-volume[84696]: DEBUG cinder.utils [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_bf4c0b13-49c6-4b1d-b790-7090eb82d4c4/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:16:56 user cinder-volume[84696]: DEBUG cinder.utils [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.851(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:16:56 user cinder-volume[84696]: DEBUG cinder.utils [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:16:56 user cinder-volume[84696]: DEBUG cinder.utils [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.035(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:16:56 user cinder-volume[84696]: DEBUG cinder.utils [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_125cc0cd-98dc-41a8-a252-3ae03797fb6f/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:17:03 user cinder-volume[84696]: DEBUG cinder.utils [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.943(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:17:03 user cinder-volume[84696]: DEBUG cinder.utils [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_125cc0cd-98dc-41a8-a252-3ae03797fb6f/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:17:03 user cinder-volume[84696]: DEBUG cinder.utils [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.762(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:17:03 user cinder-volume[84696]: DEBUG cinder.utils [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:17:04 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=16,cinder:UPDATE=4 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:17:04 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=15,cinder:UPDATE=6,cinder:INSERT=4 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:17:06 user cinder-volume[84696]: DEBUG cinder.utils [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 137.577(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:17:06 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:17:06 user cinder-volume[84696]: INFO cinder.message.api [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Creating message record for request_id = req-5a436bf3-8843-4b13-8b6b-92ee59a23185 Apr 17 16:17:06 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 16:17:06 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (d99e52ba-d0d9-4f61-be83-6ede164e256d) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 16:17:06 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 16:17:06 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:14:49Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-volume-992255480',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=07829a92-7f87-41a0-ad0d-04c53bb24b30,host='user@joviandss-2#joviandss-2',id=ae1eb1a7-8e56-418f-9032-a464645060fe,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='e10e34b8774b42d8a4cb1f78d72b411d',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:14:49Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:14:49Z,use_quota=True,user_id='e7392ed432ad41279967273f16b4a4c8',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(39cee13b-7a8b-43bd-ae52-5d071608f5ab),volume_type_id=39cee13b-7a8b-43bd-ae52-5d071608f5ab)}, 'provides': None} Apr 17 16:17:06 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:14:49Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-volume-992255480',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=07829a92-7f87-41a0-ad0d-04c53bb24b30,host='user@joviandss-2#joviandss-2',id=ae1eb1a7-8e56-418f-9032-a464645060fe,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='e10e34b8774b42d8a4cb1f78d72b411d',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:14:49Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:14:49Z,use_quota=True,user_id='e7392ed432ad41279967273f16b4a4c8',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(39cee13b-7a8b-43bd-ae52-5d071608f5ab),volume_type_id=39cee13b-7a8b-43bd-ae52-5d071608f5ab), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=07829a92-7f87-41a0-ad0d-04c53bb24b30,image_id=None,operation=,resource_backend='user@joviandss-2',snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(ae1eb1a7-8e56-418f-9032-a464645060fe),volume_id=ae1eb1a7-8e56-418f-9032-a464645060fe,volume_properties=VolumeProperties,volume_type=VolumeType(39cee13b-7a8b-43bd-ae52-5d071608f5ab))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': 'ae1eb1a7-8e56-418f-9032-a464645060fe', 'volume_name': 'volume-ae1eb1a7-8e56-418f-9032-a464645060fe', 'volume_size': 1}} Apr 17 16:17:06 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'retry': {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']}, 'request_spec': {'consistencygroup_id': None, 'group_id': '07829a92-7f87-41a0-ad0d-04c53bb24b30', 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': 'ae1eb1a7-8e56-418f-9032-a464645060fe', 'volume': {'id': 'ae1eb1a7-8e56-418f-9032-a464645060fe', '_name_id': None, 'ec2_id': None, 'user_id': 'e7392ed432ad41279967273f16b4a4c8', 'project_id': 'e10e34b8774b42d8a4cb1f78d72b411d', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-volume-992255480', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '39cee13b-7a8b-43bd-ae52-5d071608f5ab', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': '07829a92-7f87-41a0-ad0d-04c53bb24b30', 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'volume_type': {'id': '39cee13b-7a8b-43bd-ae52-5d071608f5ab', 'name': 'tempest-GroupsTest-volume-type-1704576', 'description': None, 'is_public': True, 'extra_specs': {}, 'qos_specs_id': None, 'created_at': '2023-04-17T16:14:46.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T16:14:48.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-ae1eb1a7-8e56-418f-9032-a464645060fe', 'name_id': 'ae1eb1a7-8e56-418f-9032-a464645060fe', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '39cee13b-7a8b-43bd-ae52-5d071608f5ab', 'name': 'tempest-GroupsTest-volume-type-1704576', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {}, 'qos_specs_id': None, 'created_at': '2023-04-17T16:14:46.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': '07829a92-7f87-41a0-ad0d-04c53bb24b30', 'display_description': None, 'display_name': 'tempest-volume-992255480', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': 'e10e34b8774b42d8a4cb1f78d72b411d', 'qos_specs': None, 'reservations': ['bfa3ce98-41d2-45e2-b7ff-023cf74674f2', '94fac8e5-378b-4745-89de-2454467d68c6', 'c020f3bc-c172-4b95-b5ba-37beec268319', '6cc9ae5a-aab4-4078-bba0-bc05dd3ecd71'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'e7392ed432ad41279967273f16b4a4c8', 'volume_type_id': '39cee13b-7a8b-43bd-ae52-5d071608f5ab'}, 'resource_backend': 'user@joviandss-2', 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': '07829a92-7f87-41a0-ad0d-04c53bb24b30', 'display_description': None, 'display_name': 'tempest-volume-992255480', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': 'e10e34b8774b42d8a4cb1f78d72b411d', 'qos_specs': None, 'reservations': ['bfa3ce98-41d2-45e2-b7ff-023cf74674f2', '94fac8e5-378b-4745-89de-2454467d68c6', 'c020f3bc-c172-4b95-b5ba-37beec268319', '6cc9ae5a-aab4-4078-bba0-bc05dd3ecd71'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'e7392ed432ad41279967273f16b4a4c8', 'volume_type_id': '39cee13b-7a8b-43bd-ae52-5d071608f5ab'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T16:14:46Z,deleted=False,deleted_at=None,description=None,extra_specs={},id=39cee13b-7a8b-43bd-ae52-5d071608f5ab,is_public=True,name='tempest-GroupsTest-volume-type-1704576',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T16:14:46Z,deleted=False,deleted_at=None,description=None,extra_specs={},id=39cee13b-7a8b-43bd-ae52-5d071608f5ab,is_public=True,name='tempest-GroupsTest-volume-type-1704576',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': 'e7392ed432ad41279967273f16b4a4c8', 'metadata': {}, 'qos_specs': None}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=07829a92-7f87-41a0-ad0d-04c53bb24b30,image_id=None,operation=,resource_backend='user@joviandss-2',snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(ae1eb1a7-8e56-418f-9032-a464645060fe),volume_id=ae1eb1a7-8e56-418f-9032-a464645060fe,volume_properties=VolumeProperties,volume_type=VolumeType(39cee13b-7a8b-43bd-ae52-5d071608f5ab)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:14:49Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-volume-992255480',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=07829a92-7f87-41a0-ad0d-04c53bb24b30,host='user@joviandss-2#joviandss-2',id=ae1eb1a7-8e56-418f-9032-a464645060fe,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='e10e34b8774b42d8a4cb1f78d72b411d',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:14:49Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:14:49Z,use_quota=True,user_id='e7392ed432ad41279967273f16b4a4c8',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(39cee13b-7a8b-43bd-ae52-5d071608f5ab),volume_type_id=39cee13b-7a8b-43bd-ae52-5d071608f5ab), 'context': }, 'provides': None} Apr 17 16:17:06 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:14:49Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-volume-992255480',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=07829a92-7f87-41a0-ad0d-04c53bb24b30,host='user@joviandss-2#joviandss-2',id=ae1eb1a7-8e56-418f-9032-a464645060fe,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='e10e34b8774b42d8a4cb1f78d72b411d',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:14:49Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:14:49Z,use_quota=True,user_id='e7392ed432ad41279967273f16b4a4c8',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(39cee13b-7a8b-43bd-ae52-5d071608f5ab),volume_type_id=39cee13b-7a8b-43bd-ae52-5d071608f5ab)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:14:49Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-volume-992255480',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=07829a92-7f87-41a0-ad0d-04c53bb24b30,host='user@joviandss-2#joviandss-2',id=ae1eb1a7-8e56-418f-9032-a464645060fe,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='e10e34b8774b42d8a4cb1f78d72b411d',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:14:49Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:14:49Z,use_quota=True,user_id='e7392ed432ad41279967273f16b4a4c8',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(39cee13b-7a8b-43bd-ae52-5d071608f5ab),volume_type_id=39cee13b-7a8b-43bd-ae52-5d071608f5ab)} Apr 17 16:17:06 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 16:17:06 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 16:17:06 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:17:06 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 16:17:06 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:17:06 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 16:17:06 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:17:06 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 16:17:06 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:17:06 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 16:17:06 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:17:06 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 16:17:06 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:17:06 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 16:17:06 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:17:06 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 16:17:06 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:17:06 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:17:06 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 16:17:06 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 16:17:06 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (d99e52ba-d0d9-4f61-be83-6ede164e256d) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:17:06 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (d99e52ba-d0d9-4f61-be83-6ede164e256d) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:17:06 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (d4dfa3a6-6ff3-4def-a3f8-a804a73fef86) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:17:06 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (d4dfa3a6-6ff3-4def-a3f8-a804a73fef86) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:17:06 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (db334af0-fd19-46f0-88ca-1fcdd66c739c) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:17:06 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (db334af0-fd19-46f0-88ca-1fcdd66c739c) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:17:06 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (15190e54-bc51-46cd-ac25-f5f4b36278fe) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:17:06 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Updating volume ae1eb1a7-8e56-418f-9032-a464645060fe with {'scheduled_at': datetime.datetime(2023, 4, 17, 16, 17, 6, 743035), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 16:17:06 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Volume ae1eb1a7-8e56-418f-9032-a464645060fe: re-scheduling SchedulerAPI.create_volume attempt 1 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 16:17:06 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Volume ae1eb1a7-8e56-418f-9032-a464645060fe: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 16:17:06 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] deleating volume v_ae1eb1a7-8e56-418f-9032-a464645060fe {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 16:17:06 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] get snapshots for volume v_ae1eb1a7-8e56-418f-9032-a464645060fe {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 16:17:06 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_ae1eb1a7-8e56-418f-9032-a464645060fe/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:17:06 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_ae1eb1a7-8e56-418f-9032-a464645060fe/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:17:06 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Volume reschedule parameters: True retry: {'num_attempts': 2, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 16:17:06 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Flow 'volume_create_manager' (0a6dcad9-ac8f-4ddd-a95d-1d74001447ea) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 16:17:06 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (df452e4f-c0ce-427e-8d41-9516da6f9599) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:17:06 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (df452e4f-c0ce-427e-8d41-9516da6f9599) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:14:49Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-volume-992255480',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=07829a92-7f87-41a0-ad0d-04c53bb24b30,host='user@joviandss-2#joviandss-2',id=ae1eb1a7-8e56-418f-9032-a464645060fe,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='e10e34b8774b42d8a4cb1f78d72b411d',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:17:07Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:17:07Z,use_quota=True,user_id='e7392ed432ad41279967273f16b4a4c8',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(39cee13b-7a8b-43bd-ae52-5d071608f5ab),volume_type_id=39cee13b-7a8b-43bd-ae52-5d071608f5ab)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:17:06 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (3e7569f1-ad43-445a-8c4b-e423989f8f30) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:17:06 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (3e7569f1-ad43-445a-8c4b-e423989f8f30) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:17:06 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (8a8d638d-b0bb-4740-a4da-8e6ceb149a5f) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:17:06 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (8a8d638d-b0bb-4740-a4da-8e6ceb149a5f) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': 'ae1eb1a7-8e56-418f-9032-a464645060fe', 'volume_name': 'volume-ae1eb1a7-8e56-418f-9032-a464645060fe', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:17:06 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (02d4484b-d14b-44cf-b789-93761591ab63) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:17:06 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (02d4484b-d14b-44cf-b789-93761591ab63) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:17:06 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (89b7b8a8-ff4e-48a7-a5ce-0c0562f2f22c) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:17:06 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Volume ae1eb1a7-8e56-418f-9032-a464645060fe: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-ae1eb1a7-8e56-418f-9032-a464645060fe', 'volume_size': 1} Apr 17 16:17:06 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] creating volume v_ae1eb1a7-8e56-418f-9032-a464645060fe. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 16:17:06 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] create volume {'name': 'v_ae1eb1a7-8e56-418f-9032-a464645060fe', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 16:17:06 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:17:06 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:17:09 user cinder-volume[84696]: DEBUG cinder.utils [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.814(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:17:09 user cinder-volume[84696]: DEBUG cinder.utils [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:17:09 user cinder-volume[84696]: DEBUG cinder.utils [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 74.681(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:17:09 user cinder-volume[84696]: DEBUG cinder.utils [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:17:09 user cinder-volume[84696]: DEBUG cinder.utils [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.992(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:17:09 user cinder-volume[84696]: DEBUG cinder.utils [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_ae1eb1a7-8e56-418f-9032-a464645060fe/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:17:09 user cinder-volume[84696]: DEBUG cinder.utils [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 74.920(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:17:09 user cinder-volume[84696]: DEBUG cinder.utils [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_0a7d4209-92e0-4900-8e98-36a62e68c247/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:17:09 user cinder-volume[84696]: DEBUG cinder.utils [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 74.917(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:17:09 user cinder-volume[84696]: DEBUG cinder.utils [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_49effcda-e787-4983-92b2-f8261052732d/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:17:12 user cinder-volume[84696]: DEBUG cinder.utils [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.977(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:17:12 user cinder-volume[84696]: DEBUG cinder.utils [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:17:12 user cinder-volume[84696]: DEBUG cinder.utils [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 22.161(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:17:12 user cinder-volume[84696]: DEBUG cinder.utils [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_125cc0cd-98dc-41a8-a252-3ae03797fb6f/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:17:12 user cinder-volume[84696]: DEBUG cinder.utils [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 37.308(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:17:12 user cinder-volume[84696]: DEBUG cinder.utils [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:17:12 user cinder-volume[84696]: DEBUG cinder.utils [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 37.471(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:17:12 user cinder-volume[84696]: DEBUG cinder.utils [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_bf4c0b13-49c6-4b1d-b790-7090eb82d4c4/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:17:12 user cinder-volume[84696]: DEBUG cinder.utils [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.890(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:17:12 user cinder-volume[84696]: DEBUG cinder.utils [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:17:12 user cinder-volume[84696]: DEBUG cinder.utils [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.064(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:17:12 user cinder-volume[84696]: DEBUG cinder.utils [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_ae1eb1a7-8e56-418f-9032-a464645060fe/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:17:18 user cinder-volume[84696]: DEBUG cinder.utils [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 74.876(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:17:18 user cinder-volume[84696]: DEBUG cinder.utils [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:17:18 user cinder-volume[84696]: DEBUG cinder.utils [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 75.037(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:17:18 user cinder-volume[84696]: DEBUG cinder.utils [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_85fcb006-cd96-4018-b0e3-04872ff08267/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:17:18 user cinder-volume[84696]: DEBUG cinder.utils [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.206(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:17:18 user cinder-volume[84696]: DEBUG cinder.utils [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_ae1eb1a7-8e56-418f-9032-a464645060fe/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:17:18 user cinder-volume[84696]: DEBUG cinder.utils [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.035(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:17:18 user cinder-volume[84696]: DEBUG cinder.utils [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:17:24 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-bb0706f3-0b2c-473a-acc5-93c2ac742a5f req-6b192c03-a049-4336-94b1-1dc0169da47f tempest-ServerStableDeviceRescueTest-869290889 None] terminate connection for 32819ec1-2236-465f-a1c7-ee31aa4a0911 {{(pid=84696) terminate_connection /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:1032}} Apr 17 16:17:24 user cinder-volume[84696]: INFO cinder.volume.manager [req-bb0706f3-0b2c-473a-acc5-93c2ac742a5f req-6b192c03-a049-4336-94b1-1dc0169da47f tempest-ServerStableDeviceRescueTest-869290889 None] Terminate volume connection completed successfully. Apr 17 16:17:24 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-bb0706f3-0b2c-473a-acc5-93c2ac742a5f req-6b192c03-a049-4336-94b1-1dc0169da47f tempest-ServerStableDeviceRescueTest-869290889 None] Deleting attachment fb44dce3-1af2-43cf-95fa-68ff027e0699. {{(pid=84696) attachment_delete /opt/stack/cinder/cinder/volume/manager.py:5006}} Apr 17 16:17:24 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-bb0706f3-0b2c-473a-acc5-93c2ac742a5f req-6b192c03-a049-4336-94b1-1dc0169da47f tempest-ServerStableDeviceRescueTest-869290889 None] remove_export for volume: 32819ec1-2236-465f-a1c7-ee31aa4a0911. {{(pid=84696) remove_export /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:718}} Apr 17 16:17:24 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-bb0706f3-0b2c-473a-acc5-93c2ac742a5f req-6b192c03-a049-4336-94b1-1dc0169da47f tempest-ServerStableDeviceRescueTest-869290889 None] remove export {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:912}} Apr 17 16:17:24 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-bb0706f3-0b2c-473a-acc5-93c2ac742a5f req-6b192c03-a049-4336-94b1-1dc0169da47f tempest-ServerStableDeviceRescueTest-869290889 None] detach volume:Volume(_name_id=None,admin_metadata=,attach_status='attached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:27:42Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-ServerStableDeviceRescueTest-volume-648918308',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=32819ec1-2236-465f-a1c7-ee31aa4a0911,launched_at=2023-04-17T15:27:46Z,metadata=,migration_status=None,multiattach=False,previous_status=None,project_id='6bdfb8642b9341f6ba289e920c33cf47',provider_auth='CHAP CmhLYVJy GWkyzUl8QJxnjY',provider_geometry=None,provider_id=None,provider_location='172.16.0.220:3260,1 iqn.2010-10.org.openstack:32819ec1-2236-465f-a1c7-ee31aa4a0911 0',replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:27:43Z,service_uuid='e628352e-c4cf-40eb-8389-cce40663026a',shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='in-use',terminated_at=None,updated_at=2023-04-17T15:27:57Z,use_quota=True,user_id='a972c04314724233bc8e33b64747fecf',volume_attachment=,volume_type=,volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c) from target:iqn.2010-10.org.openstack:32819ec1-2236-465f-a1c7-ee31aa4a0911. {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:913}} Apr 17 16:17:24 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [req-bb0706f3-0b2c-473a-acc5-93c2ac742a5f req-6b192c03-a049-4336-94b1-1dc0169da47f tempest-ServerStableDeviceRescueTest-869290889 None] detach volume v_32819ec1-2236-465f-a1c7-ee31aa4a0911 from target iqn.2010-10.org.openstack:32819ec1-2236-465f-a1c7-ee31aa4a0911 {{(pid=84696) detach_target_vol /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:606}} Apr 17 16:17:24 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-bb0706f3-0b2c-473a-acc5-93c2ac742a5f req-6b192c03-a049-4336-94b1-1dc0169da47f tempest-ServerStableDeviceRescueTest-869290889 None] Sending pool request DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:32819ec1-2236-465f-a1c7-ee31aa4a0911/luns/v_32819ec1-2236-465f-a1c7-ee31aa4a0911 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:17:24 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-bb0706f3-0b2c-473a-acc5-93c2ac742a5f req-6b192c03-a049-4336-94b1-1dc0169da47f tempest-ServerStableDeviceRescueTest-869290889 None] Sending DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:32819ec1-2236-465f-a1c7-ee31aa4a0911/luns/v_32819ec1-2236-465f-a1c7-ee31aa4a0911 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:17:25 user cinder-volume[84696]: DEBUG cinder.utils [req-bb0706f3-0b2c-473a-acc5-93c2ac742a5f req-6b192c03-a049-4336-94b1-1dc0169da47f tempest-ServerStableDeviceRescueTest-869290889 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 0.431(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:17:25 user cinder-volume[84696]: DEBUG cinder.utils [req-bb0706f3-0b2c-473a-acc5-93c2ac742a5f req-6b192c03-a049-4336-94b1-1dc0169da47f tempest-ServerStableDeviceRescueTest-869290889 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:32819ec1-2236-465f-a1c7-ee31aa4a0911/luns/v_32819ec1-2236-465f-a1c7-ee31aa4a0911 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:17:28 user cinder-volume[84696]: DEBUG cinder.utils [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.866(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:17:28 user cinder-volume[84696]: DEBUG cinder.utils [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_ae1eb1a7-8e56-418f-9032-a464645060fe/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:17:28 user cinder-volume[84696]: DEBUG cinder.utils [req-bb0706f3-0b2c-473a-acc5-93c2ac742a5f req-6b192c03-a049-4336-94b1-1dc0169da47f tempest-ServerStableDeviceRescueTest-869290889 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 3.952(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:17:28 user cinder-volume[84696]: DEBUG cinder.utils [req-bb0706f3-0b2c-473a-acc5-93c2ac742a5f req-6b192c03-a049-4336-94b1-1dc0169da47f tempest-ServerStableDeviceRescueTest-869290889 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:32819ec1-2236-465f-a1c7-ee31aa4a0911/luns/v_32819ec1-2236-465f-a1c7-ee31aa4a0911 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:17:28 user cinder-volume[84696]: DEBUG cinder.utils [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.696(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:17:28 user cinder-volume[84696]: DEBUG cinder.utils [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:17:31 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 140.090(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:17:31 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:17:31 user cinder-volume[84696]: ERROR oslo_service.periodic_task [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Error during VolumeManager.publish_service_capabilities: TypeError: 'NoneType' object is not subscriptable Apr 17 16:17:31 user cinder-volume[84696]: ERROR oslo_service.periodic_task Traceback (most recent call last): Apr 17 16:17:31 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/usr/local/lib/python3.10/dist-packages/oslo_service/periodic_task.py", line 216, in run_periodic_tasks Apr 17 16:17:31 user cinder-volume[84696]: ERROR oslo_service.periodic_task task(self, context) Apr 17 16:17:31 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/manager.py", line 2876, in publish_service_capabilities Apr 17 16:17:31 user cinder-volume[84696]: ERROR oslo_service.periodic_task self._report_driver_status(context) Apr 17 16:17:31 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/manager.py", line 2755, in _report_driver_status Apr 17 16:17:31 user cinder-volume[84696]: ERROR oslo_service.periodic_task volume_stats = get_stats() Apr 17 16:17:31 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/usr/local/lib/python3.10/dist-packages/oslo_utils/timeutils.py", line 319, in wrapper Apr 17 16:17:31 user cinder-volume[84696]: ERROR oslo_service.periodic_task result = func(*args, **kwargs) Apr 17 16:17:31 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/manager.py", line 2753, in get_stats Apr 17 16:17:31 user cinder-volume[84696]: ERROR oslo_service.periodic_task return self.driver.get_volume_stats(refresh=True) Apr 17 16:17:31 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/driver.py", line 666, in get_volume_stats Apr 17 16:17:31 user cinder-volume[84696]: ERROR oslo_service.periodic_task self._update_volume_stats() Apr 17 16:17:31 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 726, in _update_volume_stats Apr 17 16:17:31 user cinder-volume[84696]: ERROR oslo_service.periodic_task pool_stats = self.ra.get_pool_stats() Apr 17 16:17:31 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 897, in get_pool_stats Apr 17 16:17:31 user cinder-volume[84696]: ERROR oslo_service.periodic_task if not resp["error"] and resp["code"] == 200: Apr 17 16:17:31 user cinder-volume[84696]: ERROR oslo_service.periodic_task TypeError: 'NoneType' object is not subscriptable Apr 17 16:17:31 user cinder-volume[84696]: ERROR oslo_service.periodic_task Apr 17 16:17:31 user cinder-volume[84696]: DEBUG cinder.utils [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 40.859(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:17:31 user cinder-volume[84696]: DEBUG cinder.utils [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:17:31 user cinder-volume[84696]: DEBUG cinder.utils [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 41.043(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:17:31 user cinder-volume[84696]: DEBUG cinder.utils [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_125cc0cd-98dc-41a8-a252-3ae03797fb6f/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:17:31 user cinder-volume[84696]: WARNING oslo.service.loopingcall [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Function 'cinder.service.Service.periodic_tasks' run outlasted interval by 80.10 sec Apr 17 16:17:31 user cinder-volume[84696]: DEBUG oslo_service.periodic_task [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Running periodic task VolumeManager.publish_service_capabilities {{(pid=84696) run_periodic_tasks /usr/local/lib/python3.10/dist-packages/oslo_service/periodic_task.py:210}} Apr 17 16:17:31 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Updating volume stats {{(pid=84696) _update_volume_stats /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:724}} Apr 17 16:17:31 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Get pool Pool-0 fsprops {{(pid=84696) get_pool_stats /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:894}} Apr 17 16:17:31 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:17:31 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:17:34 user cinder-volume[84696]: DEBUG cinder.utils [req-bb0706f3-0b2c-473a-acc5-93c2ac742a5f req-6b192c03-a049-4336-94b1-1dc0169da47f tempest-ServerStableDeviceRescueTest-869290889 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 10.095(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:17:34 user cinder-volume[84696]: DEBUG cinder.utils [req-bb0706f3-0b2c-473a-acc5-93c2ac742a5f req-6b192c03-a049-4336-94b1-1dc0169da47f tempest-ServerStableDeviceRescueTest-869290889 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:32819ec1-2236-465f-a1c7-ee31aa4a0911/luns/v_32819ec1-2236-465f-a1c7-ee31aa4a0911 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:17:34 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 3.066(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:17:34 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:17:37 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.137(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:17:37 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:17:43 user cinder-volume[84696]: DEBUG cinder.utils [req-bb0706f3-0b2c-473a-acc5-93c2ac742a5f req-0a3ae49c-837d-4b32-a241-6c27a7c860b1 tempest-ServerStableDeviceRescueTest-869290889 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 139.359(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:17:43 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-bb0706f3-0b2c-473a-acc5-93c2ac742a5f req-0a3ae49c-837d-4b32-a241-6c27a7c860b1 tempest-ServerStableDeviceRescueTest-869290889 None] Geting None from DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:32819ec1-2236-465f-a1c7-ee31aa4a0911/luns/v_32819ec1-2236-465f-a1c7-ee31aa4a0911 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:17:43 user cinder-volume[84696]: WARNING cinder.volume.manager [req-bb0706f3-0b2c-473a-acc5-93c2ac742a5f req-0a3ae49c-837d-4b32-a241-6c27a7c860b1 tempest-ServerStableDeviceRescueTest-869290889 None] Failed to detach volume on the backend, ignoring failure 'NoneType' object is not subscriptable Apr 17 16:17:43 user cinder-volume[84696]: DEBUG cinder.utils [req-bb0706f3-0b2c-473a-acc5-93c2ac742a5f req-6b192c03-a049-4336-94b1-1dc0169da47f tempest-ServerStableDeviceRescueTest-869290889 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 19.313(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:17:43 user cinder-volume[84696]: DEBUG cinder.utils [req-bb0706f3-0b2c-473a-acc5-93c2ac742a5f req-6b192c03-a049-4336-94b1-1dc0169da47f tempest-ServerStableDeviceRescueTest-869290889 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:32819ec1-2236-465f-a1c7-ee31aa4a0911/luns/v_32819ec1-2236-465f-a1c7-ee31aa4a0911 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:17:44 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.284(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:17:44 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:17:44 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=9,cinder:UPDATE=6 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:17:44 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=16,cinder:INSERT=2,cinder:UPDATE=1 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:17:47 user cinder-volume[84696]: DEBUG cinder.utils [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 40.299(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:17:47 user cinder-volume[84696]: DEBUG cinder.utils [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_ae1eb1a7-8e56-418f-9032-a464645060fe/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:17:47 user cinder-volume[84696]: DEBUG cinder.utils [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 71.548(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:17:47 user cinder-volume[84696]: DEBUG cinder.utils [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:17:47 user cinder-volume[84696]: DEBUG cinder.utils [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 40.130(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:17:47 user cinder-volume[84696]: DEBUG cinder.utils [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:17:47 user cinder-volume[84696]: DEBUG cinder.utils [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 71.713(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:17:47 user cinder-volume[84696]: DEBUG cinder.utils [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_bf4c0b13-49c6-4b1d-b790-7090eb82d4c4/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:17:53 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.949(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:17:53 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:17:53 user cinder-volume[84696]: DEBUG cinder.utils [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 139.026(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:17:53 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:17:53 user cinder-volume[84696]: INFO cinder.message.api [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Creating message record for request_id = req-86e82c7b-2d41-4c71-a488-5361f4ac7ada Apr 17 16:17:53 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 16:17:53 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (38a5767d-d01e-4076-b859-6d3bbf0f9eac) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 16:17:53 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 16:17:53 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:15:34Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesTransfersV357Test-Volume-788739626',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=4725fbeb-e869-48db-a487-33998b955f83,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='1db59dbe212347b8a44e27ebed9e22fb',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:15:35Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:15:35Z,use_quota=True,user_id='ac9545120d3a48bbb11e4cd5e706b53b',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 16:17:53 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:15:34Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesTransfersV357Test-Volume-788739626',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=4725fbeb-e869-48db-a487-33998b955f83,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='1db59dbe212347b8a44e27ebed9e22fb',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:15:35Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:15:35Z,use_quota=True,user_id='ac9545120d3a48bbb11e4cd5e706b53b',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(4725fbeb-e869-48db-a487-33998b955f83),volume_id=4725fbeb-e869-48db-a487-33998b955f83,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': '4725fbeb-e869-48db-a487-33998b955f83', 'volume_name': 'volume-4725fbeb-e869-48db-a487-33998b955f83', 'volume_size': 1}} Apr 17 16:17:53 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'retry': {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']}, 'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': '4725fbeb-e869-48db-a487-33998b955f83', 'volume': {'id': '4725fbeb-e869-48db-a487-33998b955f83', '_name_id': None, 'ec2_id': None, 'user_id': 'ac9545120d3a48bbb11e4cd5e706b53b', 'project_id': '1db59dbe212347b8a44e27ebed9e22fb', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-VolumesTransfersV357Test-Volume-788739626', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T16:15:34.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-4725fbeb-e869-48db-a487-33998b955f83', 'name_id': '4725fbeb-e869-48db-a487-33998b955f83', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumesTransfersV357Test-Volume-788739626', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '1db59dbe212347b8a44e27ebed9e22fb', 'qos_specs': None, 'reservations': ['08ac1f5a-fa3d-4bd2-a307-674643af97ec', 'fb4977a0-e1cc-4948-8a75-8a55838deb4a', 'ee391e6b-8fe6-433e-8b00-937d5eb2b94a', '883ff3d2-1fda-44dd-a663-1d9481ae2b2b'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'ac9545120d3a48bbb11e4cd5e706b53b', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumesTransfersV357Test-Volume-788739626', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '1db59dbe212347b8a44e27ebed9e22fb', 'qos_specs': None, 'reservations': ['08ac1f5a-fa3d-4bd2-a307-674643af97ec', 'fb4977a0-e1cc-4948-8a75-8a55838deb4a', 'ee391e6b-8fe6-433e-8b00-937d5eb2b94a', '883ff3d2-1fda-44dd-a663-1d9481ae2b2b'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'ac9545120d3a48bbb11e4cd5e706b53b', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': 'ac9545120d3a48bbb11e4cd5e706b53b', 'metadata': {}, 'qos_specs': None}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(4725fbeb-e869-48db-a487-33998b955f83),volume_id=4725fbeb-e869-48db-a487-33998b955f83,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:15:34Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesTransfersV357Test-Volume-788739626',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=4725fbeb-e869-48db-a487-33998b955f83,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='1db59dbe212347b8a44e27ebed9e22fb',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:15:35Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:15:35Z,use_quota=True,user_id='ac9545120d3a48bbb11e4cd5e706b53b',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 16:17:53 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:15:34Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesTransfersV357Test-Volume-788739626',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=4725fbeb-e869-48db-a487-33998b955f83,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='1db59dbe212347b8a44e27ebed9e22fb',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:15:35Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:15:35Z,use_quota=True,user_id='ac9545120d3a48bbb11e4cd5e706b53b',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:15:34Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesTransfersV357Test-Volume-788739626',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=4725fbeb-e869-48db-a487-33998b955f83,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='1db59dbe212347b8a44e27ebed9e22fb',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:15:35Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:15:35Z,use_quota=True,user_id='ac9545120d3a48bbb11e4cd5e706b53b',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 16:17:53 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 16:17:53 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 16:17:53 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:17:53 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 16:17:53 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:17:53 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 16:17:53 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:17:53 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 16:17:53 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:17:53 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 16:17:53 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:17:53 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 16:17:53 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:17:53 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 16:17:53 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:17:53 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 16:17:53 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:17:53 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:17:53 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 16:17:53 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 16:17:53 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (38a5767d-d01e-4076-b859-6d3bbf0f9eac) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:17:53 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (38a5767d-d01e-4076-b859-6d3bbf0f9eac) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:17:53 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (32e7f0be-7eb2-4352-888d-75f5b25bf57f) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:17:53 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (32e7f0be-7eb2-4352-888d-75f5b25bf57f) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:17:53 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (27b4ae4a-d21f-45cb-96cf-4dd3cdc3cae8) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:17:53 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (27b4ae4a-d21f-45cb-96cf-4dd3cdc3cae8) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:17:53 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (136d9366-1462-4c4a-a656-b45f2e911887) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:17:53 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Updating volume 4725fbeb-e869-48db-a487-33998b955f83 with {'scheduled_at': datetime.datetime(2023, 4, 17, 16, 17, 53, 742339), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 16:17:53 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Volume 4725fbeb-e869-48db-a487-33998b955f83: re-scheduling SchedulerAPI.create_volume attempt 1 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 16:17:53 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Volume 4725fbeb-e869-48db-a487-33998b955f83: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 16:17:53 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] deleating volume v_4725fbeb-e869-48db-a487-33998b955f83 {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 16:17:53 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] get snapshots for volume v_4725fbeb-e869-48db-a487-33998b955f83 {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 16:17:53 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_4725fbeb-e869-48db-a487-33998b955f83/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:17:53 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_4725fbeb-e869-48db-a487-33998b955f83/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:17:53 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Volume reschedule parameters: True retry: {'num_attempts': 2, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 16:17:53 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Flow 'volume_create_manager' (e593c020-6a5b-4951-8d29-422a69e0c19c) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 16:17:53 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (54f43ff6-f144-496d-9c2c-4cf1066f8185) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:17:53 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (54f43ff6-f144-496d-9c2c-4cf1066f8185) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:15:34Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesTransfersV357Test-Volume-788739626',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=4725fbeb-e869-48db-a487-33998b955f83,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='1db59dbe212347b8a44e27ebed9e22fb',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:17:54Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:17:54Z,use_quota=True,user_id='ac9545120d3a48bbb11e4cd5e706b53b',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:17:53 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (6b890aa1-d214-4bf7-9f8c-9816e94ee7a7) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:17:53 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (6b890aa1-d214-4bf7-9f8c-9816e94ee7a7) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:17:53 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (448eaa75-bc5c-4e01-869a-024e30cc561c) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:17:53 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (448eaa75-bc5c-4e01-869a-024e30cc561c) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': '4725fbeb-e869-48db-a487-33998b955f83', 'volume_name': 'volume-4725fbeb-e869-48db-a487-33998b955f83', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:17:53 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (6ca55c73-125a-4133-834e-b9dffba0ab72) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:17:53 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (6ca55c73-125a-4133-834e-b9dffba0ab72) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:17:53 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (9509c9a6-48a4-4417-8239-a0022bd260b6) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:17:53 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Volume 4725fbeb-e869-48db-a487-33998b955f83: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-4725fbeb-e869-48db-a487-33998b955f83', 'volume_size': 1} Apr 17 16:17:53 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] creating volume v_4725fbeb-e869-48db-a487-33998b955f83. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 16:17:53 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] create volume {'name': 'v_4725fbeb-e869-48db-a487-33998b955f83', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 16:17:53 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:17:53 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:17:56 user cinder-volume[84696]: DEBUG cinder.utils [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.935(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:17:56 user cinder-volume[84696]: DEBUG cinder.utils [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_4725fbeb-e869-48db-a487-33998b955f83/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:17:56 user cinder-volume[84696]: DEBUG cinder.utils [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.755(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:17:56 user cinder-volume[84696]: DEBUG cinder.utils [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:17:59 user cinder-volume[84696]: DEBUG cinder.utils [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.011(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:17:59 user cinder-volume[84696]: DEBUG cinder.utils [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_4725fbeb-e869-48db-a487-33998b955f83/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:17:59 user cinder-volume[84696]: DEBUG cinder.utils [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.831(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:17:59 user cinder-volume[84696]: DEBUG cinder.utils [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:18:00 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Volume reschedule parameters: True retry: {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 16:18:00 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Flow 'volume_create_manager' (4fbdb665-19d0-4458-b393-02b7f20b5b51) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 16:18:00 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (8d479b1f-9718-4662-aa70-91f16dfc6bc1) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:18:00 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (8d479b1f-9718-4662-aa70-91f16dfc6bc1) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:18:00Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumeSnapshotQuotasNegativeTestJSON-Volume-1211070769',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=c9fa0a17-69d4-40d0-9d65-af6ae349cfd8,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='46b47eb1ecf7468abe1458b30e249be8',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:18:00Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:18:00Z,use_quota=True,user_id='48df3e98011f41a7938b3da3e07615c2',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:18:00 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (3965cc00-20f6-4545-ab02-6b9abc6c69ad) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:18:00 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (3965cc00-20f6-4545-ab02-6b9abc6c69ad) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:18:00 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (3694593c-86d7-4edc-857f-7b9d09e2cd9d) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:18:00 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (3694593c-86d7-4edc-857f-7b9d09e2cd9d) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': 'c9fa0a17-69d4-40d0-9d65-af6ae349cfd8', 'volume_name': 'volume-c9fa0a17-69d4-40d0-9d65-af6ae349cfd8', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:18:00 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (68c535b3-d589-4c42-bfd7-923c278eb17f) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:18:00 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (68c535b3-d589-4c42-bfd7-923c278eb17f) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:18:00 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (abaf268e-968d-49f2-bd1c-d7df9c978a17) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:18:00 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Volume c9fa0a17-69d4-40d0-9d65-af6ae349cfd8: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-c9fa0a17-69d4-40d0-9d65-af6ae349cfd8', 'volume_size': 1} Apr 17 16:18:00 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] creating volume v_c9fa0a17-69d4-40d0-9d65-af6ae349cfd8. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 16:18:00 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] create volume {'name': 'v_c9fa0a17-69d4-40d0-9d65-af6ae349cfd8', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 16:18:00 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:18:00 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:18:02 user cinder-volume[84696]: DEBUG cinder.utils [req-bb0706f3-0b2c-473a-acc5-93c2ac742a5f req-6b192c03-a049-4336-94b1-1dc0169da47f tempest-ServerStableDeviceRescueTest-869290889 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 38.191(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:18:02 user cinder-volume[84696]: DEBUG cinder.utils [req-bb0706f3-0b2c-473a-acc5-93c2ac742a5f req-6b192c03-a049-4336-94b1-1dc0169da47f tempest-ServerStableDeviceRescueTest-869290889 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:32819ec1-2236-465f-a1c7-ee31aa4a0911/luns/v_32819ec1-2236-465f-a1c7-ee31aa4a0911 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:18:02 user cinder-volume[84696]: DEBUG cinder.utils [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.681(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:18:02 user cinder-volume[84696]: DEBUG cinder.utils [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:18:05 user cinder-volume[84696]: DEBUG cinder.quota [None req-5269a4c4-2b6f-49d4-a9e8-bd52ab72602f tempest-GroupsTest-789764105 None] Created reservations ['ec78bf03-9fe0-49de-8187-4bf0b304a28a'] {{(pid=84696) reserve /opt/stack/cinder/cinder/quota.py:755}} Apr 17 16:18:05 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-5269a4c4-2b6f-49d4-a9e8-bd52ab72602f tempest-GroupsTest-789764105 None] Updating volume stats {{(pid=84696) _update_volume_stats /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:724}} Apr 17 16:18:05 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-5269a4c4-2b6f-49d4-a9e8-bd52ab72602f tempest-GroupsTest-789764105 None] Get pool Pool-0 fsprops {{(pid=84696) get_pool_stats /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:894}} Apr 17 16:18:05 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5269a4c4-2b6f-49d4-a9e8-bd52ab72602f tempest-GroupsTest-789764105 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:18:05 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5269a4c4-2b6f-49d4-a9e8-bd52ab72602f tempest-GroupsTest-789764105 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:18:05 user cinder-volume[84696]: DEBUG cinder.utils [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.151(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:18:05 user cinder-volume[84696]: DEBUG cinder.utils [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_4725fbeb-e869-48db-a487-33998b955f83/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:18:05 user cinder-volume[84696]: DEBUG cinder.utils [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 75.098(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:18:05 user cinder-volume[84696]: DEBUG cinder.utils [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:18:05 user cinder-volume[84696]: DEBUG cinder.utils [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 75.281(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:18:05 user cinder-volume[84696]: DEBUG cinder.utils [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_125cc0cd-98dc-41a8-a252-3ae03797fb6f/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:18:05 user cinder-volume[84696]: DEBUG cinder.utils [None req-5269a4c4-2b6f-49d4-a9e8-bd52ab72602f tempest-GroupsTest-789764105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 0.684(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:18:05 user cinder-volume[84696]: DEBUG cinder.utils [None req-5269a4c4-2b6f-49d4-a9e8-bd52ab72602f tempest-GroupsTest-789764105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:18:05 user cinder-volume[84696]: DEBUG cinder.utils [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.756(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:18:05 user cinder-volume[84696]: DEBUG cinder.utils [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:18:05 user cinder-volume[84696]: DEBUG cinder.utils [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 11.975(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:18:05 user cinder-volume[84696]: DEBUG cinder.utils [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:18:06 user cinder-volume[84696]: INFO cinder.volume.manager [None req-164aa17b-5907-464e-b650-af56f916c2ea tempest-GroupsV314Test-467284027 None] Group tempest-GroupsV314Test-Group-293982381: creating Apr 17 16:18:06 user cinder-volume[84696]: INFO cinder.volume.manager [None req-164aa17b-5907-464e-b650-af56f916c2ea tempest-GroupsV314Test-467284027 None] Group tempest-GroupsV314Test-Group-293982381: created successfully Apr 17 16:18:06 user cinder-volume[84696]: INFO cinder.volume.manager [None req-164aa17b-5907-464e-b650-af56f916c2ea tempest-GroupsV314Test-467284027 None] Create group completed successfully. Apr 17 16:18:06 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-2b4d4679-aa4c-41ed-b8f8-1972758fa754 tempest-GroupsTest-789764105 None] remove_export for volume: ae1eb1a7-8e56-418f-9032-a464645060fe. {{(pid=84696) remove_export /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:718}} Apr 17 16:18:06 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-2b4d4679-aa4c-41ed-b8f8-1972758fa754 tempest-GroupsTest-789764105 None] remove export {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:912}} Apr 17 16:18:06 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-2b4d4679-aa4c-41ed-b8f8-1972758fa754 tempest-GroupsTest-789764105 None] detach volume:Volume(_name_id=None,admin_metadata={},attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:14:49Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-volume-992255480',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=07829a92-7f87-41a0-ad0d-04c53bb24b30,host='user@joviandss-2#joviandss-2',id=ae1eb1a7-8e56-418f-9032-a464645060fe,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='e10e34b8774b42d8a4cb1f78d72b411d',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:17:07Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='deleting',terminated_at=None,updated_at=2023-04-17T16:18:07Z,use_quota=True,user_id='e7392ed432ad41279967273f16b4a4c8',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(39cee13b-7a8b-43bd-ae52-5d071608f5ab),volume_type_id=39cee13b-7a8b-43bd-ae52-5d071608f5ab) from target:iqn.2010-10.org.openstack:ae1eb1a7-8e56-418f-9032-a464645060fe. {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:913}} Apr 17 16:18:06 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-2b4d4679-aa4c-41ed-b8f8-1972758fa754 tempest-GroupsTest-789764105 None] detach volume v_ae1eb1a7-8e56-418f-9032-a464645060fe from target iqn.2010-10.org.openstack:ae1eb1a7-8e56-418f-9032-a464645060fe {{(pid=84696) detach_target_vol /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:606}} Apr 17 16:18:06 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-2b4d4679-aa4c-41ed-b8f8-1972758fa754 tempest-GroupsTest-789764105 None] Sending pool request DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:ae1eb1a7-8e56-418f-9032-a464645060fe/luns/v_ae1eb1a7-8e56-418f-9032-a464645060fe {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:18:06 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-2b4d4679-aa4c-41ed-b8f8-1972758fa754 tempest-GroupsTest-789764105 None] Sending DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:ae1eb1a7-8e56-418f-9032-a464645060fe/luns/v_ae1eb1a7-8e56-418f-9032-a464645060fe {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:18:08 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Volume reschedule parameters: True retry: {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 16:18:08 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Flow 'volume_create_manager' (75c8bceb-8d3c-46e1-8de8-c50a87b028f8) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 16:18:08 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (c96ba169-e55d-4bf5-90e7-5e8c5bbb9d4d) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:18:08 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (c96ba169-e55d-4bf5-90e7-5e8c5bbb9d4d) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:18:08Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-volume-769265242',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=f28ccf81-6553-4a98-83d7-400266578051,host='user@joviandss-2#joviandss-2',id=7f7b3b2e-e41d-466a-a3c2-e64e3d63b949,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='93c135fbb6f14928a9167adb3724faa3',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:18:08Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:18:08Z,use_quota=True,user_id='90edc8ffb805460794445bc433aa473f',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(be5f492b-b41f-4ad5-bffd-42eaafb454b4),volume_type_id=be5f492b-b41f-4ad5-bffd-42eaafb454b4)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:18:08 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (d7828182-b79b-4795-957b-3881571a5bc7) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:18:08 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (d7828182-b79b-4795-957b-3881571a5bc7) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:18:08 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (394d45e8-c2c8-420a-a0f1-43bb3616f1df) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:18:08 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (394d45e8-c2c8-420a-a0f1-43bb3616f1df) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': '7f7b3b2e-e41d-466a-a3c2-e64e3d63b949', 'volume_name': 'volume-7f7b3b2e-e41d-466a-a3c2-e64e3d63b949', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:18:08 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (9809a945-1c89-475a-ba2c-e4d032cdb120) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:18:08 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (9809a945-1c89-475a-ba2c-e4d032cdb120) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:18:08 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (caaf6b6c-87cc-4aee-ad2f-bf45311ff18d) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:18:08 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Volume 7f7b3b2e-e41d-466a-a3c2-e64e3d63b949: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-7f7b3b2e-e41d-466a-a3c2-e64e3d63b949', 'volume_size': 1} Apr 17 16:18:08 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] creating volume v_7f7b3b2e-e41d-466a-a3c2-e64e3d63b949. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 16:18:08 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] create volume {'name': 'v_7f7b3b2e-e41d-466a-a3c2-e64e3d63b949', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 16:18:08 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:18:08 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:18:09 user cinder-volume[84696]: DEBUG cinder.utils [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 0.933(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:18:09 user cinder-volume[84696]: DEBUG cinder.utils [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:18:09 user cinder-volume[84696]: DEBUG cinder.utils [None req-2b4d4679-aa4c-41ed-b8f8-1972758fa754 tempest-GroupsTest-789764105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.437(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:18:09 user cinder-volume[84696]: DEBUG cinder.utils [None req-2b4d4679-aa4c-41ed-b8f8-1972758fa754 tempest-GroupsTest-789764105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:ae1eb1a7-8e56-418f-9032-a464645060fe/luns/v_ae1eb1a7-8e56-418f-9032-a464645060fe (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:18:09 user cinder-volume[84696]: DEBUG cinder.utils [None req-5269a4c4-2b6f-49d4-a9e8-bd52ab72602f tempest-GroupsTest-789764105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 3.757(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:18:09 user cinder-volume[84696]: DEBUG cinder.utils [None req-5269a4c4-2b6f-49d4-a9e8-bd52ab72602f tempest-GroupsTest-789764105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:18:12 user cinder-volume[84696]: DEBUG cinder.utils [None req-2b4d4679-aa4c-41ed-b8f8-1972758fa754 tempest-GroupsTest-789764105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.506(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:18:12 user cinder-volume[84696]: DEBUG cinder.utils [None req-2b4d4679-aa4c-41ed-b8f8-1972758fa754 tempest-GroupsTest-789764105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:ae1eb1a7-8e56-418f-9032-a464645060fe/luns/v_ae1eb1a7-8e56-418f-9032-a464645060fe (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:18:12 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 40.377(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:18:12 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:18:12 user cinder-volume[84696]: DEBUG cinder.utils [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 4.006(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:18:12 user cinder-volume[84696]: DEBUG cinder.utils [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:18:12 user cinder-volume[84696]: DEBUG cinder.utils [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 11.898(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:18:12 user cinder-volume[84696]: DEBUG cinder.utils [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:18:15 user cinder-volume[84696]: DEBUG cinder.utils [None req-5269a4c4-2b6f-49d4-a9e8-bd52ab72602f tempest-GroupsTest-789764105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 9.897(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:18:15 user cinder-volume[84696]: DEBUG cinder.utils [None req-5269a4c4-2b6f-49d4-a9e8-bd52ab72602f tempest-GroupsTest-789764105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:18:15 user cinder-volume[84696]: DEBUG cinder.utils [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 140.321(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:18:15 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_0a7d4209-92e0-4900-8e98-36a62e68c247/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:18:15 user cinder-volume[84696]: WARNING cinder.volume.manager [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (19afd6eb-c20b-49ac-9713-0aa108e20290) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 16:18:15 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (9cf5485e-90e4-43ca-b95f-e75d7cad9b4b) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:18:15 user cinder-volume[84696]: WARNING cinder.volume.manager [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (9cf5485e-90e4-43ca-b95f-e75d7cad9b4b) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:18:15 user cinder-volume[84696]: WARNING cinder.volume.manager [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Flow 'volume_create_manager' (82856028-7c93-4c88-8de4-2c59742b7b78) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 16:18:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 16:18:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 16:18:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 16:18:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 16:18:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 16:18:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 16:18:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 16:18:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 16:18:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 16:18:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 16:18:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 16:18:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 16:18:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 16:18:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 16:18:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 16:18:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 16:18:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 16:18:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 16:18:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 16:18:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 16:18:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 16:18:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 16:18:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 16:18:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 16:18:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:18:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 16:18:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:18:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 16:18:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:18:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 16:18:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:18:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 16:18:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:18:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 16:18:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:18:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 16:18:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:18:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 16:18:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:18:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:18:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 16:18:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 16:18:15 user cinder-volume[84696]: DEBUG cinder.utils [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 140.104(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:18:15 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:18:15 user cinder-volume[84696]: INFO cinder.message.api [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Creating message record for request_id = req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 Apr 17 16:18:15 user cinder-volume[84696]: DEBUG cinder.utils [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.393(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:18:15 user cinder-volume[84696]: DEBUG cinder.utils [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_4725fbeb-e869-48db-a487-33998b955f83/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:18:15 user cinder-volume[84696]: DEBUG cinder.utils [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 140.343(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:18:15 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_49effcda-e787-4983-92b2-f8261052732d/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:18:15 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (1c72d6cd-9242-492c-a056-e0f57367779c) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 16:18:15 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (220f3894-090d-4a96-875a-84d6149caebb) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:18:15 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (220f3894-090d-4a96-875a-84d6149caebb) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:18:15 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Flow 'volume_create_manager' (dafcc11f-88ba-4457-a8aa-fb38e1e0bf03) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 16:18:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-40e48e44-d46b-4e9d-82c5-3b795728a90d tempest-VolumesTransfersV355Test-39117650 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 16:18:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 16:18:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 16:18:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 16:18:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 16:18:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 16:18:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 16:18:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 16:18:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 16:18:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 16:18:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 16:18:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 16:18:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 16:18:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 16:18:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 16:18:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 16:18:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 16:18:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 16:18:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 16:18:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 16:18:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 16:18:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 16:18:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 16:18:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 16:18:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:18:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 16:18:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:18:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 16:18:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:18:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 16:18:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:18:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 16:18:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:18:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 16:18:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:18:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 16:18:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:18:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 16:18:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:18:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:18:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 16:18:15 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 16:18:15 user cinder-volume[84696]: DEBUG cinder.utils [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.231(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:18:15 user cinder-volume[84696]: DEBUG cinder.utils [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:18:15 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 16:18:15 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=33,cinder:UPDATE=10 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:18:15 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=31,cinder:UPDATE=6,cinder:INSERT=4 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:18:15 user cinder-volume[84696]: WARNING cinder.volume.manager [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (d9b013a5-8ded-4da0-aceb-4e386c9783d5) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 16:18:15 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 16:18:15 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:11:18Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesGetTestJSON-Volume-1455502637',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=0a7d4209-92e0-4900-8e98-36a62e68c247,launched_at=None,metadata={Type='work'},migration_status=None,multiattach=False,previous_status=None,project_id='e330412972fa44b9bf60cd9afe4399cd',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:15:55Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:15:55Z,use_quota=True,user_id='e14688d19f1b44928fa8aceb2da0ac82',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 16:18:15 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:11:18Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesGetTestJSON-Volume-1455502637',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=0a7d4209-92e0-4900-8e98-36a62e68c247,launched_at=None,metadata={Type='work'},migration_status=None,multiattach=False,previous_status=None,project_id='e330412972fa44b9bf60cd9afe4399cd',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:15:55Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:15:55Z,use_quota=True,user_id='e14688d19f1b44928fa8aceb2da0ac82',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(0a7d4209-92e0-4900-8e98-36a62e68c247),volume_id=0a7d4209-92e0-4900-8e98-36a62e68c247,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': '0a7d4209-92e0-4900-8e98-36a62e68c247', 'volume_name': 'volume-0a7d4209-92e0-4900-8e98-36a62e68c247', 'volume_size': 1}} Apr 17 16:18:15 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': '0a7d4209-92e0-4900-8e98-36a62e68c247', 'volume': {'id': '0a7d4209-92e0-4900-8e98-36a62e68c247', '_name_id': None, 'ec2_id': None, 'user_id': 'e14688d19f1b44928fa8aceb2da0ac82', 'project_id': 'e330412972fa44b9bf60cd9afe4399cd', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-VolumesGetTestJSON-Volume-1455502637', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {'Type': 'work'}, 'admin_metadata': {}, 'glance_metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T16:11:17.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-0a7d4209-92e0-4900-8e98-36a62e68c247', 'name_id': '0a7d4209-92e0-4900-8e98-36a62e68c247', 'volume_metadata': [{'key': 'Type', 'value': 'work'}], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumesGetTestJSON-Volume-1455502637', 'encryption_key_id': None, 'metadata': {'Type': 'work'}, 'multiattach': False, 'project_id': 'e330412972fa44b9bf60cd9afe4399cd', 'qos_specs': None, 'reservations': ['a1d7a4e6-78b5-4539-8c86-495dc07de36d', '0ea4500b-e7dd-4705-a54f-22c9b06155b5', 'b2cf1ff4-8bad-445f-8b8b-6ebee58654f1', '9aeb8f74-fd45-412b-b0b4-1970c607402d'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'e14688d19f1b44928fa8aceb2da0ac82', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumesGetTestJSON-Volume-1455502637', 'encryption_key_id': None, 'metadata': {'Type': 'work'}, 'multiattach': False, 'project_id': 'e330412972fa44b9bf60cd9afe4399cd', 'qos_specs': None, 'reservations': ['a1d7a4e6-78b5-4539-8c86-495dc07de36d', '0ea4500b-e7dd-4705-a54f-22c9b06155b5', 'b2cf1ff4-8bad-445f-8b8b-6ebee58654f1', '9aeb8f74-fd45-412b-b0b4-1970c607402d'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'e14688d19f1b44928fa8aceb2da0ac82', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': 'e14688d19f1b44928fa8aceb2da0ac82', 'metadata': {'Type': 'work'}, 'qos_specs': None, 'retry': {'num_attempts': 3, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']}}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(0a7d4209-92e0-4900-8e98-36a62e68c247),volume_id=0a7d4209-92e0-4900-8e98-36a62e68c247,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:11:18Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesGetTestJSON-Volume-1455502637',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=0a7d4209-92e0-4900-8e98-36a62e68c247,launched_at=None,metadata={Type='work'},migration_status=None,multiattach=False,previous_status=None,project_id='e330412972fa44b9bf60cd9afe4399cd',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:15:55Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:15:55Z,use_quota=True,user_id='e14688d19f1b44928fa8aceb2da0ac82',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 16:18:15 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:11:18Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesGetTestJSON-Volume-1455502637',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=0a7d4209-92e0-4900-8e98-36a62e68c247,launched_at=None,metadata={Type='work'},migration_status=None,multiattach=False,previous_status=None,project_id='e330412972fa44b9bf60cd9afe4399cd',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:15:55Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:15:55Z,use_quota=True,user_id='e14688d19f1b44928fa8aceb2da0ac82',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:11:18Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesGetTestJSON-Volume-1455502637',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=0a7d4209-92e0-4900-8e98-36a62e68c247,launched_at=None,metadata={Type='work'},migration_status=None,multiattach=False,previous_status=None,project_id='e330412972fa44b9bf60cd9afe4399cd',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:15:55Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:15:55Z,use_quota=True,user_id='e14688d19f1b44928fa8aceb2da0ac82',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 16:18:15 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 16:18:15 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 16:18:15 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:18:15 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 16:18:15 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:18:15 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 16:18:15 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:18:15 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 16:18:15 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:18:15 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 16:18:15 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:18:15 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 16:18:15 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:18:15 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 16:18:15 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:18:15 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 16:18:15 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:18:15 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:18:15 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 16:18:15 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 16:18:15 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (d9b013a5-8ded-4da0-aceb-4e386c9783d5) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:18:15 user cinder-volume[84696]: WARNING cinder.volume.manager [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (d9b013a5-8ded-4da0-aceb-4e386c9783d5) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:18:15 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (d5a6989b-6ec3-4d84-b2d7-b5c3b7972f77) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:18:15 user cinder-volume[84696]: WARNING cinder.volume.manager [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (d5a6989b-6ec3-4d84-b2d7-b5c3b7972f77) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:18:15 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (50176311-b437-463c-afe1-be6646108689) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:18:15 user cinder-volume[84696]: WARNING cinder.volume.manager [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (50176311-b437-463c-afe1-be6646108689) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:18:15 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (635d913b-fca8-4373-98cf-065284347dae) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:18:15 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Updating volume 0a7d4209-92e0-4900-8e98-36a62e68c247 with {'scheduled_at': datetime.datetime(2023, 4, 17, 16, 18, 15, 273622), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 16:18:15 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Volume 0a7d4209-92e0-4900-8e98-36a62e68c247: re-scheduling SchedulerAPI.create_volume attempt 3 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 16:18:15 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Volume 0a7d4209-92e0-4900-8e98-36a62e68c247: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 16:18:15 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] deleating volume v_0a7d4209-92e0-4900-8e98-36a62e68c247 {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 16:18:15 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] get snapshots for volume v_0a7d4209-92e0-4900-8e98-36a62e68c247 {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 16:18:15 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_0a7d4209-92e0-4900-8e98-36a62e68c247/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:18:15 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_0a7d4209-92e0-4900-8e98-36a62e68c247/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:18:18 user cinder-volume[84696]: DEBUG cinder.utils [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.939(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:18:18 user cinder-volume[84696]: DEBUG cinder.utils [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_0a7d4209-92e0-4900-8e98-36a62e68c247/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:18:18 user cinder-volume[84696]: DEBUG cinder.utils [None req-2b4d4679-aa4c-41ed-b8f8-1972758fa754 tempest-GroupsTest-789764105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 11.656(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:18:18 user cinder-volume[84696]: DEBUG cinder.utils [None req-2b4d4679-aa4c-41ed-b8f8-1972758fa754 tempest-GroupsTest-789764105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:ae1eb1a7-8e56-418f-9032-a464645060fe/luns/v_ae1eb1a7-8e56-418f-9032-a464645060fe (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:18:18 user cinder-volume[84696]: DEBUG cinder.utils [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 10.155(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:18:18 user cinder-volume[84696]: DEBUG cinder.utils [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:18:21 user cinder-volume[84696]: DEBUG cinder.utils [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.007(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:18:21 user cinder-volume[84696]: DEBUG cinder.utils [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_0a7d4209-92e0-4900-8e98-36a62e68c247/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:18:21 user cinder-volume[84696]: DEBUG cinder.utils [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.112(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:18:21 user cinder-volume[84696]: DEBUG cinder.utils [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:18:21 user cinder-volume[84696]: DEBUG cinder.utils [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 74.368(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:18:21 user cinder-volume[84696]: DEBUG cinder.utils [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:18:21 user cinder-volume[84696]: DEBUG cinder.utils [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 74.541(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:18:21 user cinder-volume[84696]: DEBUG cinder.utils [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_ae1eb1a7-8e56-418f-9032-a464645060fe/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:18:24 user cinder-volume[84696]: DEBUG cinder.utils [None req-5269a4c4-2b6f-49d4-a9e8-bd52ab72602f tempest-GroupsTest-789764105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 19.113(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:18:24 user cinder-volume[84696]: DEBUG cinder.utils [None req-5269a4c4-2b6f-49d4-a9e8-bd52ab72602f tempest-GroupsTest-789764105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:18:24 user cinder-volume[84696]: DEBUG cinder.utils [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 140.285(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:18:24 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:18:24 user cinder-volume[84696]: INFO cinder.message.api [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Creating message record for request_id = req-726a1377-a691-4c8a-a56d-8b87aed706f7 Apr 17 16:18:24 user cinder-volume[84696]: DEBUG cinder.utils [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 140.450(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:18:24 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_85fcb006-cd96-4018-b0e3-04872ff08267/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:18:24 user cinder-volume[84696]: WARNING cinder.volume.manager [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (8f41e083-88b2-4f7e-8f3f-e49bca1072d7) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 16:18:24 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (a4e7d08a-d2ca-4ed8-a966-205fb3381725) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:18:24 user cinder-volume[84696]: WARNING cinder.volume.manager [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (a4e7d08a-d2ca-4ed8-a966-205fb3381725) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:18:24 user cinder-volume[84696]: WARNING cinder.volume.manager [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Flow 'volume_create_manager' (abab98a1-a3a3-491d-be7e-914cf2cd02b4) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 16:18:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 16:18:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 16:18:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 16:18:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 16:18:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 16:18:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 16:18:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 16:18:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 16:18:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 16:18:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 16:18:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 16:18:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 16:18:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 16:18:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 16:18:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 16:18:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 16:18:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 16:18:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 16:18:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 16:18:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 16:18:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 16:18:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 16:18:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 16:18:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 16:18:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:18:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 16:18:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:18:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 16:18:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:18:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 16:18:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:18:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 16:18:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:18:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 16:18:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:18:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 16:18:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:18:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 16:18:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:18:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:18:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 16:18:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 16:18:24 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 16:18:24 user cinder-volume[84696]: WARNING cinder.volume.manager [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (b1267986-f5a0-443c-b679-5606c00d9d41) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 16:18:24 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 16:18:24 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:11:28Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesSnapshotsTestJSON-volume-126016572',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=85fcb006-cd96-4018-b0e3-04872ff08267,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='34ab34dfa2e14c3785276a62a076f8e8',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:16:04Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:16:04Z,use_quota=True,user_id='c98520ee1d2c4eabaf6c186bd3ae7fee',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 16:18:24 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:11:28Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesSnapshotsTestJSON-volume-126016572',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=85fcb006-cd96-4018-b0e3-04872ff08267,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='34ab34dfa2e14c3785276a62a076f8e8',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:16:04Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:16:04Z,use_quota=True,user_id='c98520ee1d2c4eabaf6c186bd3ae7fee',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(85fcb006-cd96-4018-b0e3-04872ff08267),volume_id=85fcb006-cd96-4018-b0e3-04872ff08267,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': '85fcb006-cd96-4018-b0e3-04872ff08267', 'volume_name': 'volume-85fcb006-cd96-4018-b0e3-04872ff08267', 'volume_size': 1}} Apr 17 16:18:24 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': '85fcb006-cd96-4018-b0e3-04872ff08267', 'volume': {'id': '85fcb006-cd96-4018-b0e3-04872ff08267', '_name_id': None, 'ec2_id': None, 'user_id': 'c98520ee1d2c4eabaf6c186bd3ae7fee', 'project_id': '34ab34dfa2e14c3785276a62a076f8e8', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-VolumesSnapshotsTestJSON-volume-126016572', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'admin_metadata': {}, 'glance_metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T16:11:28.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-85fcb006-cd96-4018-b0e3-04872ff08267', 'name_id': '85fcb006-cd96-4018-b0e3-04872ff08267', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumesSnapshotsTestJSON-volume-126016572', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '34ab34dfa2e14c3785276a62a076f8e8', 'qos_specs': None, 'reservations': ['68a86b1f-7431-4aa0-8593-4b26c93f38d5', 'e3c2cd32-910b-402d-b512-12abf2966403', '3f9aa9b2-3b03-45b0-ab25-91483149a902', 'f37d20df-0298-46bc-8806-f2f882d834ac'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'c98520ee1d2c4eabaf6c186bd3ae7fee', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumesSnapshotsTestJSON-volume-126016572', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '34ab34dfa2e14c3785276a62a076f8e8', 'qos_specs': None, 'reservations': ['68a86b1f-7431-4aa0-8593-4b26c93f38d5', 'e3c2cd32-910b-402d-b512-12abf2966403', '3f9aa9b2-3b03-45b0-ab25-91483149a902', 'f37d20df-0298-46bc-8806-f2f882d834ac'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'c98520ee1d2c4eabaf6c186bd3ae7fee', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': 'c98520ee1d2c4eabaf6c186bd3ae7fee', 'metadata': {}, 'qos_specs': None, 'retry': {'num_attempts': 3, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']}}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(85fcb006-cd96-4018-b0e3-04872ff08267),volume_id=85fcb006-cd96-4018-b0e3-04872ff08267,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:11:28Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesSnapshotsTestJSON-volume-126016572',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=85fcb006-cd96-4018-b0e3-04872ff08267,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='34ab34dfa2e14c3785276a62a076f8e8',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:16:04Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:16:04Z,use_quota=True,user_id='c98520ee1d2c4eabaf6c186bd3ae7fee',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 16:18:24 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:11:28Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesSnapshotsTestJSON-volume-126016572',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=85fcb006-cd96-4018-b0e3-04872ff08267,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='34ab34dfa2e14c3785276a62a076f8e8',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:16:04Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:16:04Z,use_quota=True,user_id='c98520ee1d2c4eabaf6c186bd3ae7fee',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:11:28Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesSnapshotsTestJSON-volume-126016572',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=85fcb006-cd96-4018-b0e3-04872ff08267,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='34ab34dfa2e14c3785276a62a076f8e8',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:16:04Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:16:04Z,use_quota=True,user_id='c98520ee1d2c4eabaf6c186bd3ae7fee',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 16:18:24 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 16:18:24 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 16:18:24 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:18:24 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 16:18:24 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:18:24 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 16:18:24 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:18:24 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 16:18:24 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:18:24 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 16:18:24 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:18:24 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 16:18:24 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:18:24 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 16:18:24 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:18:24 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 16:18:24 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:18:24 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:18:24 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 16:18:24 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 16:18:24 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (b1267986-f5a0-443c-b679-5606c00d9d41) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:18:24 user cinder-volume[84696]: WARNING cinder.volume.manager [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (b1267986-f5a0-443c-b679-5606c00d9d41) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:18:24 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (b3f1f466-baa8-4acf-b117-af167237b331) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:18:24 user cinder-volume[84696]: WARNING cinder.volume.manager [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (b3f1f466-baa8-4acf-b117-af167237b331) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:18:24 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (0bc1b9f8-cfac-428e-98c1-ad90e610bf92) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:18:24 user cinder-volume[84696]: WARNING cinder.volume.manager [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (0bc1b9f8-cfac-428e-98c1-ad90e610bf92) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:18:24 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (b405f70a-8735-4c3f-9e23-c81972c36950) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:18:24 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Updating volume 85fcb006-cd96-4018-b0e3-04872ff08267 with {'scheduled_at': datetime.datetime(2023, 4, 17, 16, 18, 24, 458968), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 16:18:24 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Volume 85fcb006-cd96-4018-b0e3-04872ff08267: re-scheduling SchedulerAPI.create_volume attempt 3 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 16:18:24 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Volume 85fcb006-cd96-4018-b0e3-04872ff08267: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 16:18:24 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] deleating volume v_85fcb006-cd96-4018-b0e3-04872ff08267 {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 16:18:24 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] get snapshots for volume v_85fcb006-cd96-4018-b0e3-04872ff08267 {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 16:18:24 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_85fcb006-cd96-4018-b0e3-04872ff08267/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:18:24 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_85fcb006-cd96-4018-b0e3-04872ff08267/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:18:27 user cinder-volume[84696]: DEBUG cinder.utils [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.151(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:18:27 user cinder-volume[84696]: DEBUG cinder.utils [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_0a7d4209-92e0-4900-8e98-36a62e68c247/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:18:27 user cinder-volume[84696]: DEBUG cinder.utils [None req-2b4d4679-aa4c-41ed-b8f8-1972758fa754 tempest-GroupsTest-789764105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 20.869(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:18:27 user cinder-volume[84696]: DEBUG cinder.utils [None req-2b4d4679-aa4c-41ed-b8f8-1972758fa754 tempest-GroupsTest-789764105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:ae1eb1a7-8e56-418f-9032-a464645060fe/luns/v_ae1eb1a7-8e56-418f-9032-a464645060fe (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:18:27 user cinder-volume[84696]: DEBUG cinder.utils [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.972(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:18:27 user cinder-volume[84696]: DEBUG cinder.utils [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_85fcb006-cd96-4018-b0e3-04872ff08267/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:18:27 user cinder-volume[84696]: DEBUG cinder.utils [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 19.369(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:18:27 user cinder-volume[84696]: DEBUG cinder.utils [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:18:30 user cinder-volume[84696]: DEBUG cinder.utils [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.040(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:18:30 user cinder-volume[84696]: DEBUG cinder.utils [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_85fcb006-cd96-4018-b0e3-04872ff08267/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:18:33 user cinder-volume[84696]: DEBUG cinder.utils [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.617(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:18:33 user cinder-volume[84696]: DEBUG cinder.utils [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:18:33 user cinder-volume[84696]: DEBUG cinder.utils [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.800(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:18:33 user cinder-volume[84696]: DEBUG cinder.utils [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_4725fbeb-e869-48db-a487-33998b955f83/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:18:36 user cinder-volume[84696]: DEBUG cinder.utils [req-bb0706f3-0b2c-473a-acc5-93c2ac742a5f req-6b192c03-a049-4336-94b1-1dc0169da47f tempest-ServerStableDeviceRescueTest-869290889 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 71.983(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:18:36 user cinder-volume[84696]: DEBUG cinder.utils [req-bb0706f3-0b2c-473a-acc5-93c2ac742a5f req-6b192c03-a049-4336-94b1-1dc0169da47f tempest-ServerStableDeviceRescueTest-869290889 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:32819ec1-2236-465f-a1c7-ee31aa4a0911/luns/v_32819ec1-2236-465f-a1c7-ee31aa4a0911 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:18:36 user cinder-volume[84696]: DEBUG cinder.utils [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.186(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:18:36 user cinder-volume[84696]: DEBUG cinder.utils [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_85fcb006-cd96-4018-b0e3-04872ff08267/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:18:36 user cinder-volume[84696]: DEBUG cinder.utils [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.370(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:18:36 user cinder-volume[84696]: DEBUG cinder.utils [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_0a7d4209-92e0-4900-8e98-36a62e68c247/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:18:39 user cinder-volume[84696]: DEBUG cinder.utils [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.543(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:18:39 user cinder-volume[84696]: DEBUG cinder.utils [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:18:42 user cinder-volume[84696]: DEBUG cinder.utils [None req-5269a4c4-2b6f-49d4-a9e8-bd52ab72602f tempest-GroupsTest-789764105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 37.544(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:18:42 user cinder-volume[84696]: DEBUG cinder.utils [None req-5269a4c4-2b6f-49d4-a9e8-bd52ab72602f tempest-GroupsTest-789764105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:18:44 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=9,cinder:UPDATE=3,cinder:INSERT=1 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:18:45 user cinder-volume[84696]: DEBUG cinder.utils [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 37.797(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:18:45 user cinder-volume[84696]: DEBUG cinder.utils [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:18:45 user cinder-volume[84696]: DEBUG cinder.utils [None req-2b4d4679-aa4c-41ed-b8f8-1972758fa754 tempest-GroupsTest-789764105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.302(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:18:45 user cinder-volume[84696]: DEBUG cinder.utils [None req-2b4d4679-aa4c-41ed-b8f8-1972758fa754 tempest-GroupsTest-789764105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:ae1eb1a7-8e56-418f-9032-a464645060fe/luns/v_ae1eb1a7-8e56-418f-9032-a464645060fe (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:18:45 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 74.173(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:18:45 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:18:45 user cinder-volume[84696]: DEBUG cinder.utils [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.406(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:18:45 user cinder-volume[84696]: DEBUG cinder.utils [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_85fcb006-cd96-4018-b0e3-04872ff08267/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:18:51 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=8,cinder:UPDATE=4,cinder:INSERT=2 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:18:51 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Volume reschedule parameters: True retry: {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 16:18:51 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Flow 'volume_create_manager' (7ab40436-e8df-4d1d-8861-91491ac5dbc2) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 16:18:51 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (05bc28ea-f300-483a-bd08-a7ff9c91d60f) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:18:51 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (05bc28ea-f300-483a-bd08-a7ff9c91d60f) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:18:51Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesTransfersV357Test-Volume-1424534055',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=b900713d-b991-43a5-b403-483dfded7e11,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='1db59dbe212347b8a44e27ebed9e22fb',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:18:51Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:18:51Z,use_quota=True,user_id='ac9545120d3a48bbb11e4cd5e706b53b',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:18:51 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (b7ab0fe7-a3c6-4101-9e8e-1d3ad5b552da) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:18:51 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (b7ab0fe7-a3c6-4101-9e8e-1d3ad5b552da) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:18:51 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (cd687451-adf6-4402-994b-208b65cdafa9) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:18:51 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (cd687451-adf6-4402-994b-208b65cdafa9) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': 'b900713d-b991-43a5-b403-483dfded7e11', 'volume_name': 'volume-b900713d-b991-43a5-b403-483dfded7e11', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:18:51 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (5bc9a8da-86d4-4533-a4f2-8ee45e55019e) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:18:51 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (5bc9a8da-86d4-4533-a4f2-8ee45e55019e) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:18:51 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (4bc28d0c-b383-4171-8362-3ac6f477eb35) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:18:51 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Volume b900713d-b991-43a5-b403-483dfded7e11: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-b900713d-b991-43a5-b403-483dfded7e11', 'volume_size': 1} Apr 17 16:18:51 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] creating volume v_b900713d-b991-43a5-b403-483dfded7e11. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 16:18:51 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] create volume {'name': 'v_b900713d-b991-43a5-b403-483dfded7e11', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 16:18:51 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:18:51 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:18:52 user cinder-volume[84696]: DEBUG cinder.utils [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 136.505(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:18:52 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:18:52 user cinder-volume[84696]: INFO cinder.message.api [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Creating message record for request_id = req-e76412e0-459f-4be5-bc08-e01e70270c82 Apr 17 16:18:52 user cinder-volume[84696]: DEBUG cinder.utils [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 136.673(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:18:52 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_bf4c0b13-49c6-4b1d-b790-7090eb82d4c4/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:18:52 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (814a3795-e586-4bcc-bc41-e322e51eb5a8) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 16:18:52 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (dabb6e4d-e4a6-4a65-9999-f9d8abceef0c) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:18:52 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (dabb6e4d-e4a6-4a65-9999-f9d8abceef0c) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:18:52 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Flow 'volume_create_manager' (86ea0934-ade0-4b6a-b927-9976b9811491) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 16:18:52 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 16:18:52 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 16:18:52 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 16:18:52 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 16:18:52 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 16:18:52 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 16:18:52 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 16:18:52 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 16:18:52 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 16:18:52 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 16:18:52 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 16:18:52 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 16:18:52 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 16:18:52 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 16:18:52 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 16:18:52 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 16:18:52 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 16:18:52 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 16:18:52 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 16:18:52 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 16:18:52 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 16:18:52 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 16:18:52 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 16:18:52 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 16:18:52 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:18:52 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 16:18:52 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:18:52 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 16:18:52 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:18:52 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 16:18:52 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:18:52 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 16:18:52 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:18:52 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 16:18:52 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:18:52 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 16:18:52 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:18:52 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 16:18:52 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:18:52 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:18:52 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 16:18:52 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 16:18:52 user cinder-volume[84696]: DEBUG cinder.utils [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 0.601(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:18:52 user cinder-volume[84696]: DEBUG cinder.utils [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:18:52 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 16:18:52 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (99b95cf0-ece7-459f-8846-6179d684ad4e) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 16:18:52 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 16:18:52 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:11:57Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-AttachVolumeTestJSON-volume-1944488700',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=bf4c0b13-49c6-4b1d-b790-7090eb82d4c4,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='2eb998c84b5b41e599af9636b929295d',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:16:35Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:16:35Z,use_quota=True,user_id='b9d4a317838b47d6a2a2e7eab7a453dd',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 16:18:52 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:11:57Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-AttachVolumeTestJSON-volume-1944488700',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=bf4c0b13-49c6-4b1d-b790-7090eb82d4c4,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='2eb998c84b5b41e599af9636b929295d',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:16:35Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:16:35Z,use_quota=True,user_id='b9d4a317838b47d6a2a2e7eab7a453dd',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(bf4c0b13-49c6-4b1d-b790-7090eb82d4c4),volume_id=bf4c0b13-49c6-4b1d-b790-7090eb82d4c4,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': 'bf4c0b13-49c6-4b1d-b790-7090eb82d4c4', 'volume_name': 'volume-bf4c0b13-49c6-4b1d-b790-7090eb82d4c4', 'volume_size': 1}} Apr 17 16:18:52 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': 'bf4c0b13-49c6-4b1d-b790-7090eb82d4c4', 'volume': {'id': 'bf4c0b13-49c6-4b1d-b790-7090eb82d4c4', '_name_id': None, 'ec2_id': None, 'user_id': 'b9d4a317838b47d6a2a2e7eab7a453dd', 'project_id': '2eb998c84b5b41e599af9636b929295d', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-AttachVolumeTestJSON-volume-1944488700', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'admin_metadata': {}, 'glance_metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T16:11:57.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-bf4c0b13-49c6-4b1d-b790-7090eb82d4c4', 'name_id': 'bf4c0b13-49c6-4b1d-b790-7090eb82d4c4', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-AttachVolumeTestJSON-volume-1944488700', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '2eb998c84b5b41e599af9636b929295d', 'qos_specs': None, 'reservations': ['b181be02-433e-4478-a0ce-e1f2f6eeae64', 'fe95550b-8f6c-452e-b791-feeedb2022c3', '19f55cd7-af23-4f74-9f0b-5af38cf186a9', '2ba006a2-24af-4c11-bba8-0fe8459bb371'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'b9d4a317838b47d6a2a2e7eab7a453dd', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-AttachVolumeTestJSON-volume-1944488700', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '2eb998c84b5b41e599af9636b929295d', 'qos_specs': None, 'reservations': ['b181be02-433e-4478-a0ce-e1f2f6eeae64', 'fe95550b-8f6c-452e-b791-feeedb2022c3', '19f55cd7-af23-4f74-9f0b-5af38cf186a9', '2ba006a2-24af-4c11-bba8-0fe8459bb371'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'b9d4a317838b47d6a2a2e7eab7a453dd', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': 'b9d4a317838b47d6a2a2e7eab7a453dd', 'metadata': {}, 'qos_specs': None, 'retry': {'num_attempts': 3, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']}}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(bf4c0b13-49c6-4b1d-b790-7090eb82d4c4),volume_id=bf4c0b13-49c6-4b1d-b790-7090eb82d4c4,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:11:57Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-AttachVolumeTestJSON-volume-1944488700',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=bf4c0b13-49c6-4b1d-b790-7090eb82d4c4,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='2eb998c84b5b41e599af9636b929295d',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:16:35Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:16:35Z,use_quota=True,user_id='b9d4a317838b47d6a2a2e7eab7a453dd',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 16:18:52 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:11:57Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-AttachVolumeTestJSON-volume-1944488700',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=bf4c0b13-49c6-4b1d-b790-7090eb82d4c4,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='2eb998c84b5b41e599af9636b929295d',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:16:35Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:16:35Z,use_quota=True,user_id='b9d4a317838b47d6a2a2e7eab7a453dd',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:11:57Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-AttachVolumeTestJSON-volume-1944488700',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=bf4c0b13-49c6-4b1d-b790-7090eb82d4c4,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='2eb998c84b5b41e599af9636b929295d',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:16:35Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:16:35Z,use_quota=True,user_id='b9d4a317838b47d6a2a2e7eab7a453dd',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 16:18:52 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 16:18:52 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 16:18:52 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:18:52 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 16:18:52 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:18:52 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 16:18:52 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:18:52 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 16:18:52 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:18:52 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 16:18:52 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:18:52 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 16:18:52 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:18:52 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 16:18:52 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:18:52 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 16:18:52 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:18:52 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:18:52 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 16:18:52 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 16:18:52 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (99b95cf0-ece7-459f-8846-6179d684ad4e) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:18:52 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (99b95cf0-ece7-459f-8846-6179d684ad4e) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:18:52 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (c94acc5c-7880-4a5f-a975-25ac889b3c98) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:18:52 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (c94acc5c-7880-4a5f-a975-25ac889b3c98) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:18:52 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (6714ee39-d859-45a1-b9b3-d363ebb6651b) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:18:52 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (6714ee39-d859-45a1-b9b3-d363ebb6651b) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:18:52 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (f8fb11c9-25cc-49cc-b72d-e79139f45094) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:18:52 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Updating volume bf4c0b13-49c6-4b1d-b790-7090eb82d4c4 with {'scheduled_at': datetime.datetime(2023, 4, 17, 16, 18, 52, 107575), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 16:18:52 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Volume bf4c0b13-49c6-4b1d-b790-7090eb82d4c4: re-scheduling SchedulerAPI.create_volume attempt 3 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 16:18:52 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Volume bf4c0b13-49c6-4b1d-b790-7090eb82d4c4: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 16:18:52 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] deleating volume v_bf4c0b13-49c6-4b1d-b790-7090eb82d4c4 {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 16:18:52 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] get snapshots for volume v_bf4c0b13-49c6-4b1d-b790-7090eb82d4c4 {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 16:18:52 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_bf4c0b13-49c6-4b1d-b790-7090eb82d4c4/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:18:52 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_bf4c0b13-49c6-4b1d-b790-7090eb82d4c4/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:18:55 user cinder-volume[84696]: DEBUG cinder.utils [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.895(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:18:55 user cinder-volume[84696]: DEBUG cinder.utils [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_0a7d4209-92e0-4900-8e98-36a62e68c247/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:18:55 user cinder-volume[84696]: DEBUG cinder.utils [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 3.748(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:18:55 user cinder-volume[84696]: DEBUG cinder.utils [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:18:55 user cinder-volume[84696]: DEBUG cinder.utils [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 3.064(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:18:55 user cinder-volume[84696]: DEBUG cinder.utils [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_bf4c0b13-49c6-4b1d-b790-7090eb82d4c4/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:18:58 user cinder-volume[84696]: DEBUG cinder.utils [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.134(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:18:58 user cinder-volume[84696]: DEBUG cinder.utils [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_bf4c0b13-49c6-4b1d-b790-7090eb82d4c4/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:19:01 user cinder-volume[84696]: DEBUG cinder.utils [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 9.891(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:19:01 user cinder-volume[84696]: DEBUG cinder.utils [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:19:04 user cinder-volume[84696]: DEBUG cinder.utils [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.928(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:19:04 user cinder-volume[84696]: DEBUG cinder.utils [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_85fcb006-cd96-4018-b0e3-04872ff08267/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:19:04 user cinder-volume[84696]: DEBUG cinder.utils [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.279(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:19:04 user cinder-volume[84696]: DEBUG cinder.utils [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_bf4c0b13-49c6-4b1d-b790-7090eb82d4c4/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:19:04 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=4,cinder:INSERT=1,cinder:UPDATE=1 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:19:07 user cinder-volume[84696]: DEBUG cinder.utils [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 73.505(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:19:07 user cinder-volume[84696]: DEBUG cinder.utils [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:19:07 user cinder-volume[84696]: DEBUG cinder.utils [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 73.690(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:19:07 user cinder-volume[84696]: DEBUG cinder.utils [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_4725fbeb-e869-48db-a487-33998b955f83/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:19:10 user cinder-volume[84696]: DEBUG cinder.utils [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 139.705(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:19:10 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:19:10 user cinder-volume[84696]: INFO cinder.message.api [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Creating message record for request_id = req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc Apr 17 16:19:10 user cinder-volume[84696]: DEBUG cinder.utils [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 139.893(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:19:10 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_125cc0cd-98dc-41a8-a252-3ae03797fb6f/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:19:10 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (b41ac25a-a402-4ab8-92fa-6f3376c8c478) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 16:19:10 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (ef15c68a-a058-42d1-b8ed-a854e1ecb105) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:19:10 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (ef15c68a-a058-42d1-b8ed-a854e1ecb105) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:19:10 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Flow 'volume_create_manager' (20d31dd8-69af-4174-ae46-47214cef6a04) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 16:19:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 16:19:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 16:19:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 16:19:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 16:19:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 16:19:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 16:19:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 16:19:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 16:19:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 16:19:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 16:19:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 16:19:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 16:19:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 16:19:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 16:19:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 16:19:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 16:19:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 16:19:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 16:19:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 16:19:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 16:19:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 16:19:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 16:19:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 16:19:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 16:19:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:19:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 16:19:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:19:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 16:19:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:19:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 16:19:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:19:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 16:19:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:19:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 16:19:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:19:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 16:19:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:19:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 16:19:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:19:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:19:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 16:19:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 16:19:10 user cinder-volume[84696]: DEBUG cinder.utils [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 19.140(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:19:10 user cinder-volume[84696]: DEBUG cinder.utils [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:19:10 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 16:19:10 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (7a706239-7838-42dc-9231-90cef3054ab0) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 16:19:10 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 16:19:10 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:12:14Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesTransfersV355Test-Volume-94958465',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=125cc0cd-98dc-41a8-a252-3ae03797fb6f,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='9796cc8ccc334a5b86ff098b10a0c413',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:16:51Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:16:51Z,use_quota=True,user_id='de05a598447a4e9fb3ff488ec2d84071',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 16:19:10 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:12:14Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesTransfersV355Test-Volume-94958465',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=125cc0cd-98dc-41a8-a252-3ae03797fb6f,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='9796cc8ccc334a5b86ff098b10a0c413',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:16:51Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:16:51Z,use_quota=True,user_id='de05a598447a4e9fb3ff488ec2d84071',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(125cc0cd-98dc-41a8-a252-3ae03797fb6f),volume_id=125cc0cd-98dc-41a8-a252-3ae03797fb6f,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': '125cc0cd-98dc-41a8-a252-3ae03797fb6f', 'volume_name': 'volume-125cc0cd-98dc-41a8-a252-3ae03797fb6f', 'volume_size': 1}} Apr 17 16:19:10 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': '125cc0cd-98dc-41a8-a252-3ae03797fb6f', 'volume': {'id': '125cc0cd-98dc-41a8-a252-3ae03797fb6f', '_name_id': None, 'ec2_id': None, 'user_id': 'de05a598447a4e9fb3ff488ec2d84071', 'project_id': '9796cc8ccc334a5b86ff098b10a0c413', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-VolumesTransfersV355Test-Volume-94958465', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'admin_metadata': {}, 'glance_metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T16:12:13.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-125cc0cd-98dc-41a8-a252-3ae03797fb6f', 'name_id': '125cc0cd-98dc-41a8-a252-3ae03797fb6f', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumesTransfersV355Test-Volume-94958465', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '9796cc8ccc334a5b86ff098b10a0c413', 'qos_specs': None, 'reservations': ['a45788dc-5f7c-49c5-baac-3ecbd8da3ffc', 'ede61d0b-a2d4-4097-88bb-c2d0d901d083', '70e9f0f6-daa6-48d0-a906-1bcc00ee96d7', 'b04070c8-a6ec-4747-a48e-c741818cad1d'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'de05a598447a4e9fb3ff488ec2d84071', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumesTransfersV355Test-Volume-94958465', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '9796cc8ccc334a5b86ff098b10a0c413', 'qos_specs': None, 'reservations': ['a45788dc-5f7c-49c5-baac-3ecbd8da3ffc', 'ede61d0b-a2d4-4097-88bb-c2d0d901d083', '70e9f0f6-daa6-48d0-a906-1bcc00ee96d7', 'b04070c8-a6ec-4747-a48e-c741818cad1d'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'de05a598447a4e9fb3ff488ec2d84071', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': 'de05a598447a4e9fb3ff488ec2d84071', 'metadata': {}, 'qos_specs': None, 'retry': {'num_attempts': 3, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']}}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(125cc0cd-98dc-41a8-a252-3ae03797fb6f),volume_id=125cc0cd-98dc-41a8-a252-3ae03797fb6f,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:12:14Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesTransfersV355Test-Volume-94958465',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=125cc0cd-98dc-41a8-a252-3ae03797fb6f,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='9796cc8ccc334a5b86ff098b10a0c413',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:16:51Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:16:51Z,use_quota=True,user_id='de05a598447a4e9fb3ff488ec2d84071',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 16:19:10 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:12:14Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesTransfersV355Test-Volume-94958465',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=125cc0cd-98dc-41a8-a252-3ae03797fb6f,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='9796cc8ccc334a5b86ff098b10a0c413',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:16:51Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:16:51Z,use_quota=True,user_id='de05a598447a4e9fb3ff488ec2d84071',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:12:14Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesTransfersV355Test-Volume-94958465',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=125cc0cd-98dc-41a8-a252-3ae03797fb6f,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='9796cc8ccc334a5b86ff098b10a0c413',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:16:51Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:16:51Z,use_quota=True,user_id='de05a598447a4e9fb3ff488ec2d84071',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 16:19:10 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 16:19:10 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 16:19:10 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:19:10 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 16:19:10 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:19:10 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 16:19:10 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:19:10 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 16:19:10 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:19:10 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 16:19:10 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:19:10 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 16:19:10 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:19:10 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 16:19:10 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:19:10 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 16:19:10 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:19:10 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:19:10 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 16:19:10 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 16:19:10 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (7a706239-7838-42dc-9231-90cef3054ab0) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:19:10 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (7a706239-7838-42dc-9231-90cef3054ab0) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:19:10 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (6dc2e1e0-aa8c-44a7-9b3c-251a1af58781) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:19:10 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (6dc2e1e0-aa8c-44a7-9b3c-251a1af58781) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:19:10 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (efd1ce56-2933-45b1-a352-89e6273c8a6d) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:19:10 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (efd1ce56-2933-45b1-a352-89e6273c8a6d) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:19:10 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (7e79f666-84e3-4fa2-b6f3-355d494345b0) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:19:10 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Updating volume 125cc0cd-98dc-41a8-a252-3ae03797fb6f with {'scheduled_at': datetime.datetime(2023, 4, 17, 16, 19, 10, 647734), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 16:19:10 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Volume 125cc0cd-98dc-41a8-a252-3ae03797fb6f: re-scheduling SchedulerAPI.create_volume attempt 3 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 16:19:10 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Volume 125cc0cd-98dc-41a8-a252-3ae03797fb6f: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 16:19:10 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] deleating volume v_125cc0cd-98dc-41a8-a252-3ae03797fb6f {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 16:19:10 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] get snapshots for volume v_125cc0cd-98dc-41a8-a252-3ae03797fb6f {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 16:19:10 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_125cc0cd-98dc-41a8-a252-3ae03797fb6f/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:19:10 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_125cc0cd-98dc-41a8-a252-3ae03797fb6f/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:19:13 user cinder-volume[84696]: DEBUG cinder.utils [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.953(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:19:13 user cinder-volume[84696]: DEBUG cinder.utils [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_125cc0cd-98dc-41a8-a252-3ae03797fb6f/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:19:13 user cinder-volume[84696]: DEBUG cinder.utils [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 73.434(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:19:13 user cinder-volume[84696]: DEBUG cinder.utils [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:19:13 user cinder-volume[84696]: DEBUG cinder.utils [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.498(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:19:13 user cinder-volume[84696]: DEBUG cinder.utils [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_bf4c0b13-49c6-4b1d-b790-7090eb82d4c4/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:19:16 user cinder-volume[84696]: DEBUG cinder.utils [None req-5269a4c4-2b6f-49d4-a9e8-bd52ab72602f tempest-GroupsTest-789764105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 71.437(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:19:16 user cinder-volume[84696]: DEBUG cinder.utils [None req-5269a4c4-2b6f-49d4-a9e8-bd52ab72602f tempest-GroupsTest-789764105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:19:16 user cinder-volume[84696]: DEBUG cinder.utils [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.031(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:19:16 user cinder-volume[84696]: DEBUG cinder.utils [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_125cc0cd-98dc-41a8-a252-3ae03797fb6f/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:19:19 user cinder-volume[84696]: DEBUG cinder.utils [None req-2b4d4679-aa4c-41ed-b8f8-1972758fa754 tempest-GroupsTest-789764105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 73.186(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:19:19 user cinder-volume[84696]: DEBUG cinder.utils [None req-2b4d4679-aa4c-41ed-b8f8-1972758fa754 tempest-GroupsTest-789764105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:ae1eb1a7-8e56-418f-9032-a464645060fe/luns/v_ae1eb1a7-8e56-418f-9032-a464645060fe (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:19:19 user cinder-volume[84696]: DEBUG cinder.utils [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 71.685(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:19:19 user cinder-volume[84696]: DEBUG cinder.utils [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:19:22 user cinder-volume[84696]: DEBUG cinder.utils [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.169(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:19:22 user cinder-volume[84696]: DEBUG cinder.utils [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_125cc0cd-98dc-41a8-a252-3ae03797fb6f/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:19:24 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:INSERT=1,cinder:SELECT=4,cinder:UPDATE=1 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:19:24 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=14,cinder:UPDATE=6,cinder:INSERT=2 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:19:25 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-bb0706f3-0b2c-473a-acc5-93c2ac742a5f req-59119c1f-bf48-411e-9d13-48936b29a9e6 tempest-ServerStableDeviceRescueTest-869290889 None] terminate connection for 32819ec1-2236-465f-a1c7-ee31aa4a0911 {{(pid=84696) terminate_connection /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:1032}} Apr 17 16:19:25 user cinder-volume[84696]: INFO cinder.volume.manager [req-bb0706f3-0b2c-473a-acc5-93c2ac742a5f req-59119c1f-bf48-411e-9d13-48936b29a9e6 tempest-ServerStableDeviceRescueTest-869290889 None] Terminate volume connection completed successfully. Apr 17 16:19:25 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-bb0706f3-0b2c-473a-acc5-93c2ac742a5f req-59119c1f-bf48-411e-9d13-48936b29a9e6 tempest-ServerStableDeviceRescueTest-869290889 None] Deleting attachment fb44dce3-1af2-43cf-95fa-68ff027e0699. {{(pid=84696) attachment_delete /opt/stack/cinder/cinder/volume/manager.py:5006}} Apr 17 16:19:25 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-bb0706f3-0b2c-473a-acc5-93c2ac742a5f req-59119c1f-bf48-411e-9d13-48936b29a9e6 tempest-ServerStableDeviceRescueTest-869290889 None] remove_export for volume: 32819ec1-2236-465f-a1c7-ee31aa4a0911. {{(pid=84696) remove_export /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:718}} Apr 17 16:19:25 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-bb0706f3-0b2c-473a-acc5-93c2ac742a5f req-59119c1f-bf48-411e-9d13-48936b29a9e6 tempest-ServerStableDeviceRescueTest-869290889 None] remove export {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:912}} Apr 17 16:19:25 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [req-bb0706f3-0b2c-473a-acc5-93c2ac742a5f req-59119c1f-bf48-411e-9d13-48936b29a9e6 tempest-ServerStableDeviceRescueTest-869290889 None] detach volume:Volume(_name_id=None,admin_metadata=,attach_status='attached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T15:27:42Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-ServerStableDeviceRescueTest-volume-648918308',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=32819ec1-2236-465f-a1c7-ee31aa4a0911,launched_at=2023-04-17T15:27:46Z,metadata=,migration_status=None,multiattach=False,previous_status=None,project_id='6bdfb8642b9341f6ba289e920c33cf47',provider_auth='CHAP CmhLYVJy GWkyzUl8QJxnjY',provider_geometry=None,provider_id=None,provider_location='172.16.0.220:3260,1 iqn.2010-10.org.openstack:32819ec1-2236-465f-a1c7-ee31aa4a0911 0',replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T15:27:43Z,service_uuid='e628352e-c4cf-40eb-8389-cce40663026a',shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='in-use',terminated_at=None,updated_at=2023-04-17T15:27:57Z,use_quota=True,user_id='a972c04314724233bc8e33b64747fecf',volume_attachment=,volume_type=,volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c) from target:iqn.2010-10.org.openstack:32819ec1-2236-465f-a1c7-ee31aa4a0911. {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:913}} Apr 17 16:19:25 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [req-bb0706f3-0b2c-473a-acc5-93c2ac742a5f req-59119c1f-bf48-411e-9d13-48936b29a9e6 tempest-ServerStableDeviceRescueTest-869290889 None] detach volume v_32819ec1-2236-465f-a1c7-ee31aa4a0911 from target iqn.2010-10.org.openstack:32819ec1-2236-465f-a1c7-ee31aa4a0911 {{(pid=84696) detach_target_vol /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:606}} Apr 17 16:19:25 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-bb0706f3-0b2c-473a-acc5-93c2ac742a5f req-59119c1f-bf48-411e-9d13-48936b29a9e6 tempest-ServerStableDeviceRescueTest-869290889 None] Sending pool request DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:32819ec1-2236-465f-a1c7-ee31aa4a0911/luns/v_32819ec1-2236-465f-a1c7-ee31aa4a0911 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:19:25 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-bb0706f3-0b2c-473a-acc5-93c2ac742a5f req-59119c1f-bf48-411e-9d13-48936b29a9e6 tempest-ServerStableDeviceRescueTest-869290889 None] Sending DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:32819ec1-2236-465f-a1c7-ee31aa4a0911/luns/v_32819ec1-2236-465f-a1c7-ee31aa4a0911 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:19:25 user cinder-volume[84696]: DEBUG cinder.utils [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 138.973(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:19:25 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:19:25 user cinder-volume[84696]: INFO cinder.message.api [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Creating message record for request_id = req-5a436bf3-8843-4b13-8b6b-92ee59a23185 Apr 17 16:19:25 user cinder-volume[84696]: DEBUG cinder.utils [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 139.150(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:19:25 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_ae1eb1a7-8e56-418f-9032-a464645060fe/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:19:25 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (15190e54-bc51-46cd-ac25-f5f4b36278fe) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 16:19:25 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (81b9e09d-c5f4-4b6e-b7a8-368a1c23729f) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:19:25 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (81b9e09d-c5f4-4b6e-b7a8-368a1c23729f) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:19:25 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Flow 'volume_create_manager' (6edae92f-f435-48a3-b055-93905afcbb72) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 16:19:25 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 16:19:25 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 16:19:25 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 16:19:25 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 16:19:25 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 16:19:25 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 16:19:25 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 16:19:25 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 16:19:25 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 16:19:25 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 16:19:25 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 16:19:25 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 16:19:25 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 16:19:25 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 16:19:25 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 16:19:25 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 16:19:25 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 16:19:25 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 16:19:25 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 16:19:25 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 16:19:25 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 16:19:25 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 16:19:25 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 16:19:25 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 16:19:25 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:19:25 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 16:19:25 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:19:25 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 16:19:25 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:19:25 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 16:19:25 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:19:25 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 16:19:25 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:19:25 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 16:19:25 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:19:25 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 16:19:25 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:19:25 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 16:19:25 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:19:25 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:19:25 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 16:19:25 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 16:19:25 user cinder-volume[84696]: DEBUG cinder.utils [req-bb0706f3-0b2c-473a-acc5-93c2ac742a5f req-59119c1f-bf48-411e-9d13-48936b29a9e6 tempest-ServerStableDeviceRescueTest-869290889 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 0.895(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:19:25 user cinder-volume[84696]: DEBUG cinder.utils [req-bb0706f3-0b2c-473a-acc5-93c2ac742a5f req-59119c1f-bf48-411e-9d13-48936b29a9e6 tempest-ServerStableDeviceRescueTest-869290889 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:32819ec1-2236-465f-a1c7-ee31aa4a0911/luns/v_32819ec1-2236-465f-a1c7-ee31aa4a0911 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:19:25 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 16:19:25 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (89b7b8a8-ff4e-48a7-a5ce-0c0562f2f22c) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 16:19:25 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 16:19:25 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:14:49Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-volume-992255480',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=07829a92-7f87-41a0-ad0d-04c53bb24b30,host='user@joviandss-2#joviandss-2',id=ae1eb1a7-8e56-418f-9032-a464645060fe,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='e10e34b8774b42d8a4cb1f78d72b411d',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:17:07Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:17:07Z,use_quota=True,user_id='e7392ed432ad41279967273f16b4a4c8',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(39cee13b-7a8b-43bd-ae52-5d071608f5ab),volume_type_id=39cee13b-7a8b-43bd-ae52-5d071608f5ab)}, 'provides': None} Apr 17 16:19:25 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:14:49Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-volume-992255480',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=07829a92-7f87-41a0-ad0d-04c53bb24b30,host='user@joviandss-2#joviandss-2',id=ae1eb1a7-8e56-418f-9032-a464645060fe,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='e10e34b8774b42d8a4cb1f78d72b411d',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:17:07Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:17:07Z,use_quota=True,user_id='e7392ed432ad41279967273f16b4a4c8',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(39cee13b-7a8b-43bd-ae52-5d071608f5ab),volume_type_id=39cee13b-7a8b-43bd-ae52-5d071608f5ab), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=07829a92-7f87-41a0-ad0d-04c53bb24b30,image_id=None,operation=,resource_backend='user@joviandss-2',snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(ae1eb1a7-8e56-418f-9032-a464645060fe),volume_id=ae1eb1a7-8e56-418f-9032-a464645060fe,volume_properties=VolumeProperties,volume_type=VolumeType(39cee13b-7a8b-43bd-ae52-5d071608f5ab))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': 'ae1eb1a7-8e56-418f-9032-a464645060fe', 'volume_name': 'volume-ae1eb1a7-8e56-418f-9032-a464645060fe', 'volume_size': 1}} Apr 17 16:19:25 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'request_spec': {'consistencygroup_id': None, 'group_id': '07829a92-7f87-41a0-ad0d-04c53bb24b30', 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': 'ae1eb1a7-8e56-418f-9032-a464645060fe', 'volume': {'id': 'ae1eb1a7-8e56-418f-9032-a464645060fe', '_name_id': None, 'ec2_id': None, 'user_id': 'e7392ed432ad41279967273f16b4a4c8', 'project_id': 'e10e34b8774b42d8a4cb1f78d72b411d', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-volume-992255480', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '39cee13b-7a8b-43bd-ae52-5d071608f5ab', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': '07829a92-7f87-41a0-ad0d-04c53bb24b30', 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'admin_metadata': {}, 'glance_metadata': {}, 'volume_type': {'id': '39cee13b-7a8b-43bd-ae52-5d071608f5ab', 'name': 'tempest-GroupsTest-volume-type-1704576', 'description': None, 'is_public': True, 'extra_specs': {}, 'qos_specs_id': None, 'created_at': '2023-04-17T16:14:46.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T16:14:48.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-ae1eb1a7-8e56-418f-9032-a464645060fe', 'name_id': 'ae1eb1a7-8e56-418f-9032-a464645060fe', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '39cee13b-7a8b-43bd-ae52-5d071608f5ab', 'name': 'tempest-GroupsTest-volume-type-1704576', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {}, 'qos_specs_id': None, 'created_at': '2023-04-17T16:14:46.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': '07829a92-7f87-41a0-ad0d-04c53bb24b30', 'display_description': None, 'display_name': 'tempest-volume-992255480', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': 'e10e34b8774b42d8a4cb1f78d72b411d', 'qos_specs': None, 'reservations': ['bfa3ce98-41d2-45e2-b7ff-023cf74674f2', '94fac8e5-378b-4745-89de-2454467d68c6', 'c020f3bc-c172-4b95-b5ba-37beec268319', '6cc9ae5a-aab4-4078-bba0-bc05dd3ecd71'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'e7392ed432ad41279967273f16b4a4c8', 'volume_type_id': '39cee13b-7a8b-43bd-ae52-5d071608f5ab'}, 'resource_backend': 'user@joviandss-2', 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': '07829a92-7f87-41a0-ad0d-04c53bb24b30', 'display_description': None, 'display_name': 'tempest-volume-992255480', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': 'e10e34b8774b42d8a4cb1f78d72b411d', 'qos_specs': None, 'reservations': ['bfa3ce98-41d2-45e2-b7ff-023cf74674f2', '94fac8e5-378b-4745-89de-2454467d68c6', 'c020f3bc-c172-4b95-b5ba-37beec268319', '6cc9ae5a-aab4-4078-bba0-bc05dd3ecd71'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'e7392ed432ad41279967273f16b4a4c8', 'volume_type_id': '39cee13b-7a8b-43bd-ae52-5d071608f5ab'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T16:14:46Z,deleted=False,deleted_at=None,description=None,extra_specs={},id=39cee13b-7a8b-43bd-ae52-5d071608f5ab,is_public=True,name='tempest-GroupsTest-volume-type-1704576',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T16:14:46Z,deleted=False,deleted_at=None,description=None,extra_specs={},id=39cee13b-7a8b-43bd-ae52-5d071608f5ab,is_public=True,name='tempest-GroupsTest-volume-type-1704576',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': 'e7392ed432ad41279967273f16b4a4c8', 'metadata': {}, 'qos_specs': None, 'retry': {'num_attempts': 2, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']}}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=07829a92-7f87-41a0-ad0d-04c53bb24b30,image_id=None,operation=,resource_backend='user@joviandss-2',snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(ae1eb1a7-8e56-418f-9032-a464645060fe),volume_id=ae1eb1a7-8e56-418f-9032-a464645060fe,volume_properties=VolumeProperties,volume_type=VolumeType(39cee13b-7a8b-43bd-ae52-5d071608f5ab)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:14:49Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-volume-992255480',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=07829a92-7f87-41a0-ad0d-04c53bb24b30,host='user@joviandss-2#joviandss-2',id=ae1eb1a7-8e56-418f-9032-a464645060fe,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='e10e34b8774b42d8a4cb1f78d72b411d',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:17:07Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:17:07Z,use_quota=True,user_id='e7392ed432ad41279967273f16b4a4c8',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(39cee13b-7a8b-43bd-ae52-5d071608f5ab),volume_type_id=39cee13b-7a8b-43bd-ae52-5d071608f5ab), 'context': }, 'provides': None} Apr 17 16:19:25 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:14:49Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-volume-992255480',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=07829a92-7f87-41a0-ad0d-04c53bb24b30,host='user@joviandss-2#joviandss-2',id=ae1eb1a7-8e56-418f-9032-a464645060fe,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='e10e34b8774b42d8a4cb1f78d72b411d',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:17:07Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:17:07Z,use_quota=True,user_id='e7392ed432ad41279967273f16b4a4c8',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(39cee13b-7a8b-43bd-ae52-5d071608f5ab),volume_type_id=39cee13b-7a8b-43bd-ae52-5d071608f5ab)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:14:49Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-volume-992255480',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=07829a92-7f87-41a0-ad0d-04c53bb24b30,host='user@joviandss-2#joviandss-2',id=ae1eb1a7-8e56-418f-9032-a464645060fe,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='e10e34b8774b42d8a4cb1f78d72b411d',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:17:07Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:17:07Z,use_quota=True,user_id='e7392ed432ad41279967273f16b4a4c8',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(39cee13b-7a8b-43bd-ae52-5d071608f5ab),volume_type_id=39cee13b-7a8b-43bd-ae52-5d071608f5ab)} Apr 17 16:19:25 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 16:19:25 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 16:19:25 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:19:25 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 16:19:25 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:19:25 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 16:19:25 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:19:25 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 16:19:25 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:19:25 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 16:19:25 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:19:25 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 16:19:25 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:19:25 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 16:19:25 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:19:25 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 16:19:25 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:19:25 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:19:25 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 16:19:25 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 16:19:25 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (89b7b8a8-ff4e-48a7-a5ce-0c0562f2f22c) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:19:25 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (89b7b8a8-ff4e-48a7-a5ce-0c0562f2f22c) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:19:25 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (02d4484b-d14b-44cf-b789-93761591ab63) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:19:25 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (02d4484b-d14b-44cf-b789-93761591ab63) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:19:25 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (8a8d638d-b0bb-4740-a4da-8e6ceb149a5f) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:19:25 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (8a8d638d-b0bb-4740-a4da-8e6ceb149a5f) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:19:25 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (3e7569f1-ad43-445a-8c4b-e423989f8f30) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:19:25 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Updating volume ae1eb1a7-8e56-418f-9032-a464645060fe with {'scheduled_at': datetime.datetime(2023, 4, 17, 16, 19, 25, 997668), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 16:19:26 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Volume ae1eb1a7-8e56-418f-9032-a464645060fe: re-scheduling SchedulerAPI.create_volume attempt 2 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 16:19:26 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Volume ae1eb1a7-8e56-418f-9032-a464645060fe: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 16:19:26 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] deleating volume v_ae1eb1a7-8e56-418f-9032-a464645060fe {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 16:19:26 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] get snapshots for volume v_ae1eb1a7-8e56-418f-9032-a464645060fe {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 16:19:26 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_ae1eb1a7-8e56-418f-9032-a464645060fe/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:19:26 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_ae1eb1a7-8e56-418f-9032-a464645060fe/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:19:26 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Volume reschedule parameters: True retry: {'num_attempts': 3, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 16:19:26 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Flow 'volume_create_manager' (ae657be4-a652-4d1d-b136-310bcdde538f) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 16:19:26 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (2af8e7ff-4433-40a6-929a-d743fbd72ce9) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:19:26 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (2af8e7ff-4433-40a6-929a-d743fbd72ce9) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:14:49Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-volume-992255480',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=07829a92-7f87-41a0-ad0d-04c53bb24b30,host='user@joviandss-2#joviandss-2',id=ae1eb1a7-8e56-418f-9032-a464645060fe,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='e10e34b8774b42d8a4cb1f78d72b411d',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:19:26Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='deleting',terminated_at=None,updated_at=2023-04-17T16:19:26Z,use_quota=True,user_id='e7392ed432ad41279967273f16b4a4c8',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(39cee13b-7a8b-43bd-ae52-5d071608f5ab),volume_type_id=39cee13b-7a8b-43bd-ae52-5d071608f5ab)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:19:26 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (a83887a1-4d90-481a-a257-ad22ea100a28) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:19:26 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (a83887a1-4d90-481a-a257-ad22ea100a28) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:19:26 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (936eacde-57bc-4e09-acfb-84841b2415ba) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:19:26 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (936eacde-57bc-4e09-acfb-84841b2415ba) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'deleting', 'type': 'raw', 'volume_id': 'ae1eb1a7-8e56-418f-9032-a464645060fe', 'volume_name': 'volume-ae1eb1a7-8e56-418f-9032-a464645060fe', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:19:26 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (f6949b78-8685-4ca5-b86b-ff8946012815) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:19:26 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (f6949b78-8685-4ca5-b86b-ff8946012815) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:19:26 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (297c89d2-a771-48ab-a6e2-547a45d137e0) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:19:26 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Volume ae1eb1a7-8e56-418f-9032-a464645060fe: being created as raw with specification: {'status': 'deleting', 'volume_name': 'volume-ae1eb1a7-8e56-418f-9032-a464645060fe', 'volume_size': 1} Apr 17 16:19:26 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] creating volume v_ae1eb1a7-8e56-418f-9032-a464645060fe. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 16:19:26 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] create volume {'name': 'v_ae1eb1a7-8e56-418f-9032-a464645060fe', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 16:19:26 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:19:26 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:19:29 user cinder-volume[84696]: DEBUG cinder.utils [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 3.057(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:19:29 user cinder-volume[84696]: DEBUG cinder.utils [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_ae1eb1a7-8e56-418f-9032-a464645060fe/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:19:29 user cinder-volume[84696]: DEBUG cinder.utils [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.897(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:19:29 user cinder-volume[84696]: DEBUG cinder.utils [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:19:29 user cinder-volume[84696]: DEBUG cinder.utils [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 37.640(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:19:29 user cinder-volume[84696]: DEBUG cinder.utils [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:19:29 user cinder-volume[84696]: DEBUG cinder.utils [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 73.789(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:19:29 user cinder-volume[84696]: DEBUG cinder.utils [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_0a7d4209-92e0-4900-8e98-36a62e68c247/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:19:29 user cinder-volume[84696]: DEBUG cinder.utils [req-bb0706f3-0b2c-473a-acc5-93c2ac742a5f req-59119c1f-bf48-411e-9d13-48936b29a9e6 tempest-ServerStableDeviceRescueTest-869290889 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 4.048(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:19:29 user cinder-volume[84696]: DEBUG cinder.utils [req-bb0706f3-0b2c-473a-acc5-93c2ac742a5f req-59119c1f-bf48-411e-9d13-48936b29a9e6 tempest-ServerStableDeviceRescueTest-869290889 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:32819ec1-2236-465f-a1c7-ee31aa4a0911/luns/v_32819ec1-2236-465f-a1c7-ee31aa4a0911 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:19:32 user cinder-volume[84696]: DEBUG cinder.utils [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 40.022(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:19:32 user cinder-volume[84696]: DEBUG cinder.utils [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_bf4c0b13-49c6-4b1d-b790-7090eb82d4c4/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:19:32 user cinder-volume[84696]: DEBUG cinder.utils [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.483(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:19:32 user cinder-volume[84696]: DEBUG cinder.utils [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_125cc0cd-98dc-41a8-a252-3ae03797fb6f/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:19:32 user cinder-volume[84696]: DEBUG cinder.utils [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.131(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:19:32 user cinder-volume[84696]: DEBUG cinder.utils [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_ae1eb1a7-8e56-418f-9032-a464645060fe/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:19:32 user cinder-volume[84696]: DEBUG cinder.utils [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.970(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:19:32 user cinder-volume[84696]: DEBUG cinder.utils [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:19:35 user cinder-volume[84696]: DEBUG cinder.utils [req-bb0706f3-0b2c-473a-acc5-93c2ac742a5f req-59119c1f-bf48-411e-9d13-48936b29a9e6 tempest-ServerStableDeviceRescueTest-869290889 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 10.185(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:19:35 user cinder-volume[84696]: DEBUG cinder.utils [req-bb0706f3-0b2c-473a-acc5-93c2ac742a5f req-59119c1f-bf48-411e-9d13-48936b29a9e6 tempest-ServerStableDeviceRescueTest-869290889 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:32819ec1-2236-465f-a1c7-ee31aa4a0911/luns/v_32819ec1-2236-465f-a1c7-ee31aa4a0911 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:19:38 user cinder-volume[84696]: DEBUG cinder.utils [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.273(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:19:38 user cinder-volume[84696]: DEBUG cinder.utils [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_ae1eb1a7-8e56-418f-9032-a464645060fe/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:19:38 user cinder-volume[84696]: DEBUG cinder.utils [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.112(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:19:38 user cinder-volume[84696]: DEBUG cinder.utils [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:19:38 user cinder-volume[84696]: DEBUG cinder.utils [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 73.818(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:19:38 user cinder-volume[84696]: DEBUG cinder.utils [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_85fcb006-cd96-4018-b0e3-04872ff08267/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:19:41 user cinder-volume[84696]: DEBUG cinder.utils [req-bb0706f3-0b2c-473a-acc5-93c2ac742a5f req-6b192c03-a049-4336-94b1-1dc0169da47f tempest-ServerStableDeviceRescueTest-869290889 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 136.687(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:19:41 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-bb0706f3-0b2c-473a-acc5-93c2ac742a5f req-6b192c03-a049-4336-94b1-1dc0169da47f tempest-ServerStableDeviceRescueTest-869290889 None] Geting None from DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:32819ec1-2236-465f-a1c7-ee31aa4a0911/luns/v_32819ec1-2236-465f-a1c7-ee31aa4a0911 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:19:41 user cinder-volume[84696]: WARNING cinder.volume.manager [req-bb0706f3-0b2c-473a-acc5-93c2ac742a5f req-6b192c03-a049-4336-94b1-1dc0169da47f tempest-ServerStableDeviceRescueTest-869290889 None] Failed to detach volume on the backend, ignoring failure 'NoneType' object is not subscriptable Apr 17 16:19:44 user cinder-volume[84696]: DEBUG cinder.utils [req-bb0706f3-0b2c-473a-acc5-93c2ac742a5f req-59119c1f-bf48-411e-9d13-48936b29a9e6 tempest-ServerStableDeviceRescueTest-869290889 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 19.401(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:19:44 user cinder-volume[84696]: DEBUG cinder.utils [req-bb0706f3-0b2c-473a-acc5-93c2ac742a5f req-59119c1f-bf48-411e-9d13-48936b29a9e6 tempest-ServerStableDeviceRescueTest-869290889 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:32819ec1-2236-465f-a1c7-ee31aa4a0911/luns/v_32819ec1-2236-465f-a1c7-ee31aa4a0911 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:19:47 user cinder-volume[84696]: DEBUG cinder.utils [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.490(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:19:47 user cinder-volume[84696]: DEBUG cinder.utils [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_ae1eb1a7-8e56-418f-9032-a464645060fe/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:19:47 user cinder-volume[84696]: DEBUG cinder.utils [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.329(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:19:47 user cinder-volume[84696]: DEBUG cinder.utils [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:19:50 user cinder-volume[84696]: DEBUG cinder.utils [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.914(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:19:50 user cinder-volume[84696]: DEBUG cinder.utils [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_125cc0cd-98dc-41a8-a252-3ae03797fb6f/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:19:50 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 138.874(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:19:50 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:19:50 user cinder-volume[84696]: ERROR oslo_service.periodic_task [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Error during VolumeManager.publish_service_capabilities: TypeError: 'NoneType' object is not subscriptable Apr 17 16:19:50 user cinder-volume[84696]: ERROR oslo_service.periodic_task Traceback (most recent call last): Apr 17 16:19:50 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/usr/local/lib/python3.10/dist-packages/oslo_service/periodic_task.py", line 216, in run_periodic_tasks Apr 17 16:19:50 user cinder-volume[84696]: ERROR oslo_service.periodic_task task(self, context) Apr 17 16:19:50 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/manager.py", line 2876, in publish_service_capabilities Apr 17 16:19:50 user cinder-volume[84696]: ERROR oslo_service.periodic_task self._report_driver_status(context) Apr 17 16:19:50 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/manager.py", line 2755, in _report_driver_status Apr 17 16:19:50 user cinder-volume[84696]: ERROR oslo_service.periodic_task volume_stats = get_stats() Apr 17 16:19:50 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/usr/local/lib/python3.10/dist-packages/oslo_utils/timeutils.py", line 319, in wrapper Apr 17 16:19:50 user cinder-volume[84696]: ERROR oslo_service.periodic_task result = func(*args, **kwargs) Apr 17 16:19:50 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/manager.py", line 2753, in get_stats Apr 17 16:19:50 user cinder-volume[84696]: ERROR oslo_service.periodic_task return self.driver.get_volume_stats(refresh=True) Apr 17 16:19:50 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/driver.py", line 666, in get_volume_stats Apr 17 16:19:50 user cinder-volume[84696]: ERROR oslo_service.periodic_task self._update_volume_stats() Apr 17 16:19:50 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 726, in _update_volume_stats Apr 17 16:19:50 user cinder-volume[84696]: ERROR oslo_service.periodic_task pool_stats = self.ra.get_pool_stats() Apr 17 16:19:50 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 897, in get_pool_stats Apr 17 16:19:50 user cinder-volume[84696]: ERROR oslo_service.periodic_task if not resp["error"] and resp["code"] == 200: Apr 17 16:19:50 user cinder-volume[84696]: ERROR oslo_service.periodic_task TypeError: 'NoneType' object is not subscriptable Apr 17 16:19:50 user cinder-volume[84696]: ERROR oslo_service.periodic_task Apr 17 16:19:50 user cinder-volume[84696]: WARNING oslo.service.loopingcall [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Function 'cinder.service.Service.periodic_tasks' run outlasted interval by 78.88 sec Apr 17 16:19:50 user cinder-volume[84696]: DEBUG oslo_service.periodic_task [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Running periodic task VolumeManager.publish_service_capabilities {{(pid=84696) run_periodic_tasks /usr/local/lib/python3.10/dist-packages/oslo_service/periodic_task.py:210}} Apr 17 16:19:50 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Updating volume stats {{(pid=84696) _update_volume_stats /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:724}} Apr 17 16:19:50 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Get pool Pool-0 fsprops {{(pid=84696) get_pool_stats /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:894}} Apr 17 16:19:50 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:19:50 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:19:53 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 3.066(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:19:53 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:19:54 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=12,cinder:UPDATE=3 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:19:54 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=12,cinder:UPDATE=3,cinder:INSERT=2 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:19:56 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.138(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:19:56 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:20:03 user cinder-volume[84696]: DEBUG cinder.utils [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 72.066(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:20:03 user cinder-volume[84696]: DEBUG cinder.utils [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:20:03 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.924(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:20:03 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:20:03 user cinder-volume[84696]: DEBUG cinder.utils [req-bb0706f3-0b2c-473a-acc5-93c2ac742a5f req-59119c1f-bf48-411e-9d13-48936b29a9e6 tempest-ServerStableDeviceRescueTest-869290889 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 38.476(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:20:03 user cinder-volume[84696]: DEBUG cinder.utils [req-bb0706f3-0b2c-473a-acc5-93c2ac742a5f req-59119c1f-bf48-411e-9d13-48936b29a9e6 tempest-ServerStableDeviceRescueTest-869290889 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:32819ec1-2236-465f-a1c7-ee31aa4a0911/luns/v_32819ec1-2236-465f-a1c7-ee31aa4a0911 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:20:06 user cinder-volume[84696]: DEBUG cinder.utils [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 40.562(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:20:06 user cinder-volume[84696]: DEBUG cinder.utils [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_ae1eb1a7-8e56-418f-9032-a464645060fe/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:20:06 user cinder-volume[84696]: DEBUG cinder.utils [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 74.456(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:20:06 user cinder-volume[84696]: DEBUG cinder.utils [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_bf4c0b13-49c6-4b1d-b790-7090eb82d4c4/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:20:06 user cinder-volume[84696]: DEBUG cinder.utils [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 40.401(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:20:06 user cinder-volume[84696]: DEBUG cinder.utils [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:20:12 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 22.139(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:20:12 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:20:12 user cinder-volume[84696]: DEBUG cinder.utils [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 138.941(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:20:12 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_4725fbeb-e869-48db-a487-33998b955f83/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:20:12 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (136d9366-1462-4c4a-a656-b45f2e911887) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 16:20:12 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (252e12a0-0371-401d-9f23-c1e3320e16e6) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:20:12 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (252e12a0-0371-401d-9f23-c1e3320e16e6) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:20:12 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Flow 'volume_create_manager' (45002bb1-e3ac-48b3-afe5-acf3c897c114) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 16:20:12 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 16:20:12 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 16:20:12 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 16:20:12 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 16:20:12 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 16:20:12 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 16:20:12 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 16:20:12 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 16:20:12 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 16:20:12 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 16:20:12 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 16:20:12 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 16:20:12 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 16:20:12 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 16:20:12 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 16:20:12 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 16:20:12 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 16:20:12 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 16:20:12 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 16:20:12 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 16:20:12 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 16:20:12 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 16:20:12 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 16:20:12 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 16:20:12 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:20:12 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 16:20:12 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:20:12 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 16:20:12 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:20:12 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 16:20:12 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:20:12 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 16:20:12 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:20:12 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 16:20:12 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:20:12 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 16:20:12 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:20:12 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 16:20:12 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:20:12 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:20:12 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 16:20:12 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 16:20:12 user cinder-volume[84696]: DEBUG cinder.utils [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 138.786(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:20:12 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:20:12 user cinder-volume[84696]: INFO cinder.message.api [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Creating message record for request_id = req-86e82c7b-2d41-4c71-a488-5361f4ac7ada Apr 17 16:20:12 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 16:20:12 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (9509c9a6-48a4-4417-8239-a0022bd260b6) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 16:20:12 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 16:20:12 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:15:34Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesTransfersV357Test-Volume-788739626',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=4725fbeb-e869-48db-a487-33998b955f83,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='1db59dbe212347b8a44e27ebed9e22fb',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:17:54Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:17:54Z,use_quota=True,user_id='ac9545120d3a48bbb11e4cd5e706b53b',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 16:20:12 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:15:34Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesTransfersV357Test-Volume-788739626',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=4725fbeb-e869-48db-a487-33998b955f83,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='1db59dbe212347b8a44e27ebed9e22fb',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:17:54Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:17:54Z,use_quota=True,user_id='ac9545120d3a48bbb11e4cd5e706b53b',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(4725fbeb-e869-48db-a487-33998b955f83),volume_id=4725fbeb-e869-48db-a487-33998b955f83,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': '4725fbeb-e869-48db-a487-33998b955f83', 'volume_name': 'volume-4725fbeb-e869-48db-a487-33998b955f83', 'volume_size': 1}} Apr 17 16:20:12 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': '4725fbeb-e869-48db-a487-33998b955f83', 'volume': {'id': '4725fbeb-e869-48db-a487-33998b955f83', '_name_id': None, 'ec2_id': None, 'user_id': 'ac9545120d3a48bbb11e4cd5e706b53b', 'project_id': '1db59dbe212347b8a44e27ebed9e22fb', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-VolumesTransfersV357Test-Volume-788739626', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'admin_metadata': {}, 'glance_metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T16:15:34.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-4725fbeb-e869-48db-a487-33998b955f83', 'name_id': '4725fbeb-e869-48db-a487-33998b955f83', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumesTransfersV357Test-Volume-788739626', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '1db59dbe212347b8a44e27ebed9e22fb', 'qos_specs': None, 'reservations': ['08ac1f5a-fa3d-4bd2-a307-674643af97ec', 'fb4977a0-e1cc-4948-8a75-8a55838deb4a', 'ee391e6b-8fe6-433e-8b00-937d5eb2b94a', '883ff3d2-1fda-44dd-a663-1d9481ae2b2b'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'ac9545120d3a48bbb11e4cd5e706b53b', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumesTransfersV357Test-Volume-788739626', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '1db59dbe212347b8a44e27ebed9e22fb', 'qos_specs': None, 'reservations': ['08ac1f5a-fa3d-4bd2-a307-674643af97ec', 'fb4977a0-e1cc-4948-8a75-8a55838deb4a', 'ee391e6b-8fe6-433e-8b00-937d5eb2b94a', '883ff3d2-1fda-44dd-a663-1d9481ae2b2b'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'ac9545120d3a48bbb11e4cd5e706b53b', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': 'ac9545120d3a48bbb11e4cd5e706b53b', 'metadata': {}, 'qos_specs': None, 'retry': {'num_attempts': 2, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']}}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(4725fbeb-e869-48db-a487-33998b955f83),volume_id=4725fbeb-e869-48db-a487-33998b955f83,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:15:34Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesTransfersV357Test-Volume-788739626',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=4725fbeb-e869-48db-a487-33998b955f83,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='1db59dbe212347b8a44e27ebed9e22fb',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:17:54Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:17:54Z,use_quota=True,user_id='ac9545120d3a48bbb11e4cd5e706b53b',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 16:20:12 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:15:34Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesTransfersV357Test-Volume-788739626',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=4725fbeb-e869-48db-a487-33998b955f83,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='1db59dbe212347b8a44e27ebed9e22fb',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:17:54Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:17:54Z,use_quota=True,user_id='ac9545120d3a48bbb11e4cd5e706b53b',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:15:34Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesTransfersV357Test-Volume-788739626',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=4725fbeb-e869-48db-a487-33998b955f83,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='1db59dbe212347b8a44e27ebed9e22fb',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:17:54Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:17:54Z,use_quota=True,user_id='ac9545120d3a48bbb11e4cd5e706b53b',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 16:20:12 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 16:20:12 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 16:20:12 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:20:12 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 16:20:12 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:20:12 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 16:20:12 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:20:12 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 16:20:12 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:20:12 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 16:20:12 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:20:12 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 16:20:12 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:20:12 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 16:20:12 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:20:12 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 16:20:12 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:20:12 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:20:12 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 16:20:12 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 16:20:12 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (9509c9a6-48a4-4417-8239-a0022bd260b6) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:20:12 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (9509c9a6-48a4-4417-8239-a0022bd260b6) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:20:12 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (6ca55c73-125a-4133-834e-b9dffba0ab72) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:20:12 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (6ca55c73-125a-4133-834e-b9dffba0ab72) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:20:12 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (448eaa75-bc5c-4e01-869a-024e30cc561c) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:20:12 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (448eaa75-bc5c-4e01-869a-024e30cc561c) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:20:12 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (6b890aa1-d214-4bf7-9f8c-9816e94ee7a7) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:20:12 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Updating volume 4725fbeb-e869-48db-a487-33998b955f83 with {'scheduled_at': datetime.datetime(2023, 4, 17, 16, 20, 12, 835198), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 16:20:12 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Volume 4725fbeb-e869-48db-a487-33998b955f83: re-scheduling SchedulerAPI.create_volume attempt 2 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 16:20:12 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Volume 4725fbeb-e869-48db-a487-33998b955f83: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 16:20:12 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] deleating volume v_4725fbeb-e869-48db-a487-33998b955f83 {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 16:20:12 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] get snapshots for volume v_4725fbeb-e869-48db-a487-33998b955f83 {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 16:20:12 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_4725fbeb-e869-48db-a487-33998b955f83/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:20:12 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_4725fbeb-e869-48db-a487-33998b955f83/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:20:12 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Volume reschedule parameters: True retry: {'num_attempts': 3, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 16:20:12 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Flow 'volume_create_manager' (2f4b7229-a766-417a-884d-bb3f4475738e) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 16:20:13 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (5aa2bdf6-3a17-4601-943e-e453086434ef) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:20:13 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (5aa2bdf6-3a17-4601-943e-e453086434ef) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:15:34Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesTransfersV357Test-Volume-788739626',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=4725fbeb-e869-48db-a487-33998b955f83,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='1db59dbe212347b8a44e27ebed9e22fb',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:20:13Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:20:13Z,use_quota=True,user_id='ac9545120d3a48bbb11e4cd5e706b53b',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:20:13 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (0ea2e582-cdc9-48a0-a628-5209cfb9fbf9) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:20:13 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (0ea2e582-cdc9-48a0-a628-5209cfb9fbf9) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:20:13 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (01829bcc-10ab-4c69-92ea-f2f7a1c958ec) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:20:13 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (01829bcc-10ab-4c69-92ea-f2f7a1c958ec) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': '4725fbeb-e869-48db-a487-33998b955f83', 'volume_name': 'volume-4725fbeb-e869-48db-a487-33998b955f83', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:20:13 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (12e1189e-6a4e-4989-8030-9c5bc8251a16) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:20:13 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (12e1189e-6a4e-4989-8030-9c5bc8251a16) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:20:13 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (59b5ebbf-9898-4b59-82a2-bb496efdf3cb) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:20:13 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Volume 4725fbeb-e869-48db-a487-33998b955f83: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-4725fbeb-e869-48db-a487-33998b955f83', 'volume_size': 1} Apr 17 16:20:13 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] creating volume v_4725fbeb-e869-48db-a487-33998b955f83. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 16:20:13 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] create volume {'name': 'v_4725fbeb-e869-48db-a487-33998b955f83', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 16:20:13 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:20:13 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:20:15 user cinder-volume[84696]: DEBUG cinder.utils [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.926(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:20:15 user cinder-volume[84696]: DEBUG cinder.utils [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_4725fbeb-e869-48db-a487-33998b955f83/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:20:15 user cinder-volume[84696]: DEBUG cinder.utils [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.753(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:20:15 user cinder-volume[84696]: DEBUG cinder.utils [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:20:18 user cinder-volume[84696]: DEBUG cinder.utils [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.824(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:20:18 user cinder-volume[84696]: DEBUG cinder.utils [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:20:18 user cinder-volume[84696]: DEBUG cinder.utils [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.999(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:20:18 user cinder-volume[84696]: DEBUG cinder.utils [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_4725fbeb-e869-48db-a487-33998b955f83/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:20:18 user cinder-volume[84696]: DEBUG cinder.utils [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 138.681(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:20:18 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:20:18 user cinder-volume[84696]: INFO cinder.message.api [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Creating message record for request_id = req-f436a31c-1175-4e56-a839-11ba99c1cd60 Apr 17 16:20:18 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 16:20:18 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (abaf268e-968d-49f2-bd1c-d7df9c978a17) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 16:20:18 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 16:20:18 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:18:00Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumeSnapshotQuotasNegativeTestJSON-Volume-1211070769',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=c9fa0a17-69d4-40d0-9d65-af6ae349cfd8,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='46b47eb1ecf7468abe1458b30e249be8',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:18:00Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:18:00Z,use_quota=True,user_id='48df3e98011f41a7938b3da3e07615c2',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 16:20:18 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:18:00Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumeSnapshotQuotasNegativeTestJSON-Volume-1211070769',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=c9fa0a17-69d4-40d0-9d65-af6ae349cfd8,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='46b47eb1ecf7468abe1458b30e249be8',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:18:00Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:18:00Z,use_quota=True,user_id='48df3e98011f41a7938b3da3e07615c2',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(c9fa0a17-69d4-40d0-9d65-af6ae349cfd8),volume_id=c9fa0a17-69d4-40d0-9d65-af6ae349cfd8,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': 'c9fa0a17-69d4-40d0-9d65-af6ae349cfd8', 'volume_name': 'volume-c9fa0a17-69d4-40d0-9d65-af6ae349cfd8', 'volume_size': 1}} Apr 17 16:20:18 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'retry': {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']}, 'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': 'c9fa0a17-69d4-40d0-9d65-af6ae349cfd8', 'volume': {'id': 'c9fa0a17-69d4-40d0-9d65-af6ae349cfd8', '_name_id': None, 'ec2_id': None, 'user_id': '48df3e98011f41a7938b3da3e07615c2', 'project_id': '46b47eb1ecf7468abe1458b30e249be8', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-VolumeSnapshotQuotasNegativeTestJSON-Volume-1211070769', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T16:17:59.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-c9fa0a17-69d4-40d0-9d65-af6ae349cfd8', 'name_id': 'c9fa0a17-69d4-40d0-9d65-af6ae349cfd8', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumeSnapshotQuotasNegativeTestJSON-Volume-1211070769', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '46b47eb1ecf7468abe1458b30e249be8', 'qos_specs': None, 'reservations': ['019a7782-5155-4c89-a83b-235283c575e4', '0bfbe139-e9e6-4454-941c-cc4e8c814eb3', 'df625689-4086-45a1-a606-f6541e8045a3', '80b6f310-1127-40c3-bb09-d4d6b83f85c6'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': '48df3e98011f41a7938b3da3e07615c2', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumeSnapshotQuotasNegativeTestJSON-Volume-1211070769', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '46b47eb1ecf7468abe1458b30e249be8', 'qos_specs': None, 'reservations': ['019a7782-5155-4c89-a83b-235283c575e4', '0bfbe139-e9e6-4454-941c-cc4e8c814eb3', 'df625689-4086-45a1-a606-f6541e8045a3', '80b6f310-1127-40c3-bb09-d4d6b83f85c6'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': '48df3e98011f41a7938b3da3e07615c2', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': '48df3e98011f41a7938b3da3e07615c2', 'metadata': {}, 'qos_specs': None}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(c9fa0a17-69d4-40d0-9d65-af6ae349cfd8),volume_id=c9fa0a17-69d4-40d0-9d65-af6ae349cfd8,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:18:00Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumeSnapshotQuotasNegativeTestJSON-Volume-1211070769',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=c9fa0a17-69d4-40d0-9d65-af6ae349cfd8,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='46b47eb1ecf7468abe1458b30e249be8',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:18:00Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:18:00Z,use_quota=True,user_id='48df3e98011f41a7938b3da3e07615c2',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 16:20:18 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:18:00Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumeSnapshotQuotasNegativeTestJSON-Volume-1211070769',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=c9fa0a17-69d4-40d0-9d65-af6ae349cfd8,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='46b47eb1ecf7468abe1458b30e249be8',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:18:00Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:18:00Z,use_quota=True,user_id='48df3e98011f41a7938b3da3e07615c2',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:18:00Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumeSnapshotQuotasNegativeTestJSON-Volume-1211070769',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=c9fa0a17-69d4-40d0-9d65-af6ae349cfd8,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='46b47eb1ecf7468abe1458b30e249be8',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:18:00Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:18:00Z,use_quota=True,user_id='48df3e98011f41a7938b3da3e07615c2',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 16:20:18 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 16:20:18 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 16:20:18 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:20:18 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 16:20:18 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:20:18 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 16:20:18 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:20:18 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 16:20:18 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:20:18 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 16:20:18 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:20:18 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 16:20:18 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:20:18 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 16:20:18 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:20:18 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 16:20:18 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:20:18 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:20:18 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 16:20:18 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 16:20:18 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (abaf268e-968d-49f2-bd1c-d7df9c978a17) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:20:18 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (abaf268e-968d-49f2-bd1c-d7df9c978a17) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:20:18 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (68c535b3-d589-4c42-bfd7-923c278eb17f) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:20:18 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (68c535b3-d589-4c42-bfd7-923c278eb17f) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:20:18 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (3694593c-86d7-4edc-857f-7b9d09e2cd9d) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:20:18 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (3694593c-86d7-4edc-857f-7b9d09e2cd9d) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:20:18 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (3965cc00-20f6-4545-ab02-6b9abc6c69ad) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:20:18 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Updating volume c9fa0a17-69d4-40d0-9d65-af6ae349cfd8 with {'scheduled_at': datetime.datetime(2023, 4, 17, 16, 20, 18, 943766), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 16:20:18 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Volume c9fa0a17-69d4-40d0-9d65-af6ae349cfd8: re-scheduling SchedulerAPI.create_volume attempt 1 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 16:20:18 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Volume c9fa0a17-69d4-40d0-9d65-af6ae349cfd8: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 16:20:18 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] deleating volume v_c9fa0a17-69d4-40d0-9d65-af6ae349cfd8 {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 16:20:18 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] get snapshots for volume v_c9fa0a17-69d4-40d0-9d65-af6ae349cfd8 {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 16:20:18 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_c9fa0a17-69d4-40d0-9d65-af6ae349cfd8/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:20:18 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_c9fa0a17-69d4-40d0-9d65-af6ae349cfd8/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:20:19 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Volume reschedule parameters: True retry: {'num_attempts': 2, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 16:20:19 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Flow 'volume_create_manager' (ab0bf919-e871-4d35-a73a-16c12db71ae4) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 16:20:19 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (b6c8e53c-bd02-45b1-aa4e-f9c9016d7a0e) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:20:19 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (b6c8e53c-bd02-45b1-aa4e-f9c9016d7a0e) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:18:00Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumeSnapshotQuotasNegativeTestJSON-Volume-1211070769',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=c9fa0a17-69d4-40d0-9d65-af6ae349cfd8,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='46b47eb1ecf7468abe1458b30e249be8',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:20:19Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:20:19Z,use_quota=True,user_id='48df3e98011f41a7938b3da3e07615c2',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:20:19 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (85d4bcaf-ef93-4bed-8fe3-77dd128ef3ae) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:20:19 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (85d4bcaf-ef93-4bed-8fe3-77dd128ef3ae) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:20:19 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (fcf5d16f-4999-4f67-ae30-36a2355e93d4) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:20:19 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (fcf5d16f-4999-4f67-ae30-36a2355e93d4) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': 'c9fa0a17-69d4-40d0-9d65-af6ae349cfd8', 'volume_name': 'volume-c9fa0a17-69d4-40d0-9d65-af6ae349cfd8', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:20:19 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (1f136c37-038d-4b49-99b2-e7ce751248b7) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:20:19 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (1f136c37-038d-4b49-99b2-e7ce751248b7) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:20:19 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (9e273a19-870b-4921-99e6-3e9d030b9cc1) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:20:19 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Volume c9fa0a17-69d4-40d0-9d65-af6ae349cfd8: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-c9fa0a17-69d4-40d0-9d65-af6ae349cfd8', 'volume_size': 1} Apr 17 16:20:19 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] creating volume v_c9fa0a17-69d4-40d0-9d65-af6ae349cfd8. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 16:20:19 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] create volume {'name': 'v_c9fa0a17-69d4-40d0-9d65-af6ae349cfd8', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 16:20:19 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:20:19 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:20:21 user cinder-volume[84696]: DEBUG cinder.utils [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.979(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:20:21 user cinder-volume[84696]: DEBUG cinder.utils [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_c9fa0a17-69d4-40d0-9d65-af6ae349cfd8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:20:21 user cinder-volume[84696]: DEBUG cinder.utils [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.685(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:20:21 user cinder-volume[84696]: DEBUG cinder.utils [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:20:21 user cinder-volume[84696]: DEBUG cinder.utils [None req-5269a4c4-2b6f-49d4-a9e8-bd52ab72602f tempest-GroupsTest-789764105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 136.683(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:20:21 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5269a4c4-2b6f-49d4-a9e8-bd52ab72602f tempest-GroupsTest-789764105 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:20:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-5269a4c4-2b6f-49d4-a9e8-bd52ab72602f tempest-GroupsTest-789764105 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 16:20:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 16:20:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 16:20:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 16:20:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 16:20:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 16:20:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 16:20:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 16:20:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 3864, in delete_group Apr 17 16:20:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.publish_service_capabilities(context) Apr 17 16:20:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 2876, in publish_service_capabilities Apr 17 16:20:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self._report_driver_status(context) Apr 17 16:20:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 2755, in _report_driver_status Apr 17 16:20:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server volume_stats = get_stats() Apr 17 16:20:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/timeutils.py", line 319, in wrapper Apr 17 16:20:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(*args, **kwargs) Apr 17 16:20:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 2753, in get_stats Apr 17 16:20:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self.driver.get_volume_stats(refresh=True) Apr 17 16:20:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/driver.py", line 666, in get_volume_stats Apr 17 16:20:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self._update_volume_stats() Apr 17 16:20:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 726, in _update_volume_stats Apr 17 16:20:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server pool_stats = self.ra.get_pool_stats() Apr 17 16:20:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 897, in get_pool_stats Apr 17 16:20:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] == 200: Apr 17 16:20:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 16:20:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 16:20:24 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=12,cinder:INSERT=3,cinder:UPDATE=4 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:20:24 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:UPDATE=6,cinder:SELECT=15,cinder:INSERT=1 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:20:25 user cinder-volume[84696]: DEBUG cinder.utils [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 74.345(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:20:25 user cinder-volume[84696]: DEBUG cinder.utils [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_125cc0cd-98dc-41a8-a252-3ae03797fb6f/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:20:25 user cinder-volume[84696]: DEBUG cinder.utils [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.757(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:20:25 user cinder-volume[84696]: DEBUG cinder.utils [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:20:25 user cinder-volume[84696]: DEBUG cinder.utils [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.144(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:20:25 user cinder-volume[84696]: DEBUG cinder.utils [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_4725fbeb-e869-48db-a487-33998b955f83/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:20:25 user cinder-volume[84696]: DEBUG cinder.utils [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.054(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:20:25 user cinder-volume[84696]: DEBUG cinder.utils [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_c9fa0a17-69d4-40d0-9d65-af6ae349cfd8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:20:25 user cinder-volume[84696]: DEBUG cinder.utils [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 136.936(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:20:25 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:20:25 user cinder-volume[84696]: INFO cinder.message.api [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Creating message record for request_id = req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 Apr 17 16:20:25 user cinder-volume[84696]: DEBUG cinder.utils [None req-2b4d4679-aa4c-41ed-b8f8-1972758fa754 tempest-GroupsTest-789764105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 138.443(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:20:25 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-2b4d4679-aa4c-41ed-b8f8-1972758fa754 tempest-GroupsTest-789764105 None] Geting None from DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:ae1eb1a7-8e56-418f-9032-a464645060fe/luns/v_ae1eb1a7-8e56-418f-9032-a464645060fe {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:20:25 user cinder-volume[84696]: DEBUG cinder.utils [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 11.978(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:20:25 user cinder-volume[84696]: DEBUG cinder.utils [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:20:25 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 16:20:25 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (caaf6b6c-87cc-4aee-ad2f-bf45311ff18d) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 16:20:25 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 16:20:25 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:18:08Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-volume-769265242',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=f28ccf81-6553-4a98-83d7-400266578051,host='user@joviandss-2#joviandss-2',id=7f7b3b2e-e41d-466a-a3c2-e64e3d63b949,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='93c135fbb6f14928a9167adb3724faa3',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:18:08Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:18:08Z,use_quota=True,user_id='90edc8ffb805460794445bc433aa473f',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(be5f492b-b41f-4ad5-bffd-42eaafb454b4),volume_type_id=be5f492b-b41f-4ad5-bffd-42eaafb454b4)}, 'provides': None} Apr 17 16:20:25 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:18:08Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-volume-769265242',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=f28ccf81-6553-4a98-83d7-400266578051,host='user@joviandss-2#joviandss-2',id=7f7b3b2e-e41d-466a-a3c2-e64e3d63b949,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='93c135fbb6f14928a9167adb3724faa3',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:18:08Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:18:08Z,use_quota=True,user_id='90edc8ffb805460794445bc433aa473f',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(be5f492b-b41f-4ad5-bffd-42eaafb454b4),volume_type_id=be5f492b-b41f-4ad5-bffd-42eaafb454b4), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=f28ccf81-6553-4a98-83d7-400266578051,image_id=None,operation=,resource_backend='user@joviandss-2',snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(7f7b3b2e-e41d-466a-a3c2-e64e3d63b949),volume_id=7f7b3b2e-e41d-466a-a3c2-e64e3d63b949,volume_properties=VolumeProperties,volume_type=VolumeType(be5f492b-b41f-4ad5-bffd-42eaafb454b4))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': '7f7b3b2e-e41d-466a-a3c2-e64e3d63b949', 'volume_name': 'volume-7f7b3b2e-e41d-466a-a3c2-e64e3d63b949', 'volume_size': 1}} Apr 17 16:20:25 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'retry': {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']}, 'request_spec': {'consistencygroup_id': None, 'group_id': 'f28ccf81-6553-4a98-83d7-400266578051', 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': '7f7b3b2e-e41d-466a-a3c2-e64e3d63b949', 'volume': {'id': '7f7b3b2e-e41d-466a-a3c2-e64e3d63b949', '_name_id': None, 'ec2_id': None, 'user_id': '90edc8ffb805460794445bc433aa473f', 'project_id': '93c135fbb6f14928a9167adb3724faa3', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-volume-769265242', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': 'be5f492b-b41f-4ad5-bffd-42eaafb454b4', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': 'f28ccf81-6553-4a98-83d7-400266578051', 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'volume_type': {'id': 'be5f492b-b41f-4ad5-bffd-42eaafb454b4', 'name': 'tempest-GroupsV314Test-volume-type-1100522366', 'description': None, 'is_public': True, 'extra_specs': {}, 'qos_specs_id': None, 'created_at': '2023-04-17T16:18:06.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T16:18:07.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-7f7b3b2e-e41d-466a-a3c2-e64e3d63b949', 'name_id': '7f7b3b2e-e41d-466a-a3c2-e64e3d63b949', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': 'be5f492b-b41f-4ad5-bffd-42eaafb454b4', 'name': 'tempest-GroupsV314Test-volume-type-1100522366', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {}, 'qos_specs_id': None, 'created_at': '2023-04-17T16:18:06.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': 'f28ccf81-6553-4a98-83d7-400266578051', 'display_description': None, 'display_name': 'tempest-volume-769265242', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '93c135fbb6f14928a9167adb3724faa3', 'qos_specs': None, 'reservations': ['16413670-7ac2-4f17-a10d-ae77a197fc22', '05fb658c-9b25-41f9-b551-6c9fc7ca8983', 'bdd8bc3b-5d24-4a01-90e5-fca884142434', '6c4f66ea-ee22-430e-89df-e1bdd93b9f58'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': '90edc8ffb805460794445bc433aa473f', 'volume_type_id': 'be5f492b-b41f-4ad5-bffd-42eaafb454b4'}, 'resource_backend': 'user@joviandss-2', 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': 'f28ccf81-6553-4a98-83d7-400266578051', 'display_description': None, 'display_name': 'tempest-volume-769265242', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '93c135fbb6f14928a9167adb3724faa3', 'qos_specs': None, 'reservations': ['16413670-7ac2-4f17-a10d-ae77a197fc22', '05fb658c-9b25-41f9-b551-6c9fc7ca8983', 'bdd8bc3b-5d24-4a01-90e5-fca884142434', '6c4f66ea-ee22-430e-89df-e1bdd93b9f58'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': '90edc8ffb805460794445bc433aa473f', 'volume_type_id': 'be5f492b-b41f-4ad5-bffd-42eaafb454b4'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T16:18:06Z,deleted=False,deleted_at=None,description=None,extra_specs={},id=be5f492b-b41f-4ad5-bffd-42eaafb454b4,is_public=True,name='tempest-GroupsV314Test-volume-type-1100522366',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T16:18:06Z,deleted=False,deleted_at=None,description=None,extra_specs={},id=be5f492b-b41f-4ad5-bffd-42eaafb454b4,is_public=True,name='tempest-GroupsV314Test-volume-type-1100522366',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': '90edc8ffb805460794445bc433aa473f', 'metadata': {}, 'qos_specs': None}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=f28ccf81-6553-4a98-83d7-400266578051,image_id=None,operation=,resource_backend='user@joviandss-2',snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(7f7b3b2e-e41d-466a-a3c2-e64e3d63b949),volume_id=7f7b3b2e-e41d-466a-a3c2-e64e3d63b949,volume_properties=VolumeProperties,volume_type=VolumeType(be5f492b-b41f-4ad5-bffd-42eaafb454b4)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:18:08Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-volume-769265242',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=f28ccf81-6553-4a98-83d7-400266578051,host='user@joviandss-2#joviandss-2',id=7f7b3b2e-e41d-466a-a3c2-e64e3d63b949,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='93c135fbb6f14928a9167adb3724faa3',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:18:08Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:18:08Z,use_quota=True,user_id='90edc8ffb805460794445bc433aa473f',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(be5f492b-b41f-4ad5-bffd-42eaafb454b4),volume_type_id=be5f492b-b41f-4ad5-bffd-42eaafb454b4), 'context': }, 'provides': None} Apr 17 16:20:25 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:18:08Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-volume-769265242',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=f28ccf81-6553-4a98-83d7-400266578051,host='user@joviandss-2#joviandss-2',id=7f7b3b2e-e41d-466a-a3c2-e64e3d63b949,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='93c135fbb6f14928a9167adb3724faa3',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:18:08Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:18:08Z,use_quota=True,user_id='90edc8ffb805460794445bc433aa473f',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(be5f492b-b41f-4ad5-bffd-42eaafb454b4),volume_type_id=be5f492b-b41f-4ad5-bffd-42eaafb454b4)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:18:08Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-volume-769265242',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=f28ccf81-6553-4a98-83d7-400266578051,host='user@joviandss-2#joviandss-2',id=7f7b3b2e-e41d-466a-a3c2-e64e3d63b949,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='93c135fbb6f14928a9167adb3724faa3',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:18:08Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:18:08Z,use_quota=True,user_id='90edc8ffb805460794445bc433aa473f',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(be5f492b-b41f-4ad5-bffd-42eaafb454b4),volume_type_id=be5f492b-b41f-4ad5-bffd-42eaafb454b4)} Apr 17 16:20:25 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 16:20:25 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 16:20:25 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:20:25 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 16:20:25 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:20:25 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 16:20:25 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:20:25 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 16:20:25 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:20:25 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 16:20:25 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:20:25 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 16:20:25 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:20:25 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 16:20:25 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:20:25 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 16:20:25 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:20:25 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:20:25 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 16:20:25 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 16:20:25 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (caaf6b6c-87cc-4aee-ad2f-bf45311ff18d) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:20:25 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (caaf6b6c-87cc-4aee-ad2f-bf45311ff18d) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:20:25 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (9809a945-1c89-475a-ba2c-e4d032cdb120) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:20:25 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (9809a945-1c89-475a-ba2c-e4d032cdb120) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:20:25 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (394d45e8-c2c8-420a-a0f1-43bb3616f1df) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:20:25 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (394d45e8-c2c8-420a-a0f1-43bb3616f1df) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:20:25 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (d7828182-b79b-4795-957b-3881571a5bc7) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:20:25 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Updating volume 7f7b3b2e-e41d-466a-a3c2-e64e3d63b949 with {'scheduled_at': datetime.datetime(2023, 4, 17, 16, 20, 25, 97921), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 16:20:25 user cinder-volume[84696]: ERROR cinder.volume.manager [None req-2b4d4679-aa4c-41ed-b8f8-1972758fa754 tempest-GroupsTest-789764105 None] Delete group failed. Apr 17 16:20:25 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Volume 7f7b3b2e-e41d-466a-a3c2-e64e3d63b949: re-scheduling SchedulerAPI.create_volume attempt 1 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 16:20:25 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-2b4d4679-aa4c-41ed-b8f8-1972758fa754 tempest-GroupsTest-789764105 None] Exception during message handling: cinder.exception.VolumeDriverException: Volume driver reported an error: Delete group failed. Apr 17 16:20:25 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 16:20:25 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 16:20:25 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 16:20:25 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 16:20:25 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 16:20:25 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 16:20:25 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 16:20:25 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 3805, in delete_group Apr 17 16:20:25 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 16:20:25 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:20:25 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 16:20:25 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:20:25 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 16:20:25 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 3799, in delete_group Apr 17 16:20:25 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise exception.VolumeDriverException(message=msg) Apr 17 16:20:25 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server cinder.exception.VolumeDriverException: Volume driver reported an error: Delete group failed. Apr 17 16:20:25 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 16:20:25 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Volume 7f7b3b2e-e41d-466a-a3c2-e64e3d63b949: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 16:20:25 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] deleating volume v_7f7b3b2e-e41d-466a-a3c2-e64e3d63b949 {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 16:20:25 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] get snapshots for volume v_7f7b3b2e-e41d-466a-a3c2-e64e3d63b949 {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 16:20:25 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_7f7b3b2e-e41d-466a-a3c2-e64e3d63b949/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:20:25 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_7f7b3b2e-e41d-466a-a3c2-e64e3d63b949/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:20:25 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Volume reschedule parameters: True retry: {'num_attempts': 2, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 16:20:25 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Flow 'volume_create_manager' (2f24858c-65c0-4dd3-8b07-1964f233c23d) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 16:20:25 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (c41a31ea-c2c1-4456-a0cd-6a71f46175b9) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:20:25 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (c41a31ea-c2c1-4456-a0cd-6a71f46175b9) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:18:08Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-volume-769265242',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=f28ccf81-6553-4a98-83d7-400266578051,host='user@joviandss-2#joviandss-2',id=7f7b3b2e-e41d-466a-a3c2-e64e3d63b949,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='93c135fbb6f14928a9167adb3724faa3',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:20:25Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:20:25Z,use_quota=True,user_id='90edc8ffb805460794445bc433aa473f',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(be5f492b-b41f-4ad5-bffd-42eaafb454b4),volume_type_id=be5f492b-b41f-4ad5-bffd-42eaafb454b4)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:20:25 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (ed1e4b67-153e-4602-81cf-2a65404dafbc) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:20:25 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (ed1e4b67-153e-4602-81cf-2a65404dafbc) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:20:25 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (518b98fa-7d20-487d-8ea7-bdf1f37f4c72) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:20:25 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (518b98fa-7d20-487d-8ea7-bdf1f37f4c72) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': '7f7b3b2e-e41d-466a-a3c2-e64e3d63b949', 'volume_name': 'volume-7f7b3b2e-e41d-466a-a3c2-e64e3d63b949', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:20:25 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (44898e4d-1d99-453e-9276-3228f547ddf8) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:20:25 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (44898e4d-1d99-453e-9276-3228f547ddf8) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:20:25 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (679460c5-9bf5-4df2-99d1-0df6c3fa1060) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:20:25 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Volume 7f7b3b2e-e41d-466a-a3c2-e64e3d63b949: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-7f7b3b2e-e41d-466a-a3c2-e64e3d63b949', 'volume_size': 1} Apr 17 16:20:25 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] creating volume v_7f7b3b2e-e41d-466a-a3c2-e64e3d63b949. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 16:20:25 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] create volume {'name': 'v_7f7b3b2e-e41d-466a-a3c2-e64e3d63b949', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 16:20:25 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:20:25 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:20:28 user cinder-volume[84696]: DEBUG cinder.utils [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.958(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:20:28 user cinder-volume[84696]: DEBUG cinder.utils [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_7f7b3b2e-e41d-466a-a3c2-e64e3d63b949/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:20:28 user cinder-volume[84696]: DEBUG cinder.utils [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.788(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:20:28 user cinder-volume[84696]: DEBUG cinder.utils [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:20:31 user cinder-volume[84696]: DEBUG cinder.utils [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 11.900(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:20:31 user cinder-volume[84696]: DEBUG cinder.utils [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:20:31 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 40.572(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:20:31 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:20:31 user cinder-volume[84696]: DEBUG cinder.utils [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.034(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:20:31 user cinder-volume[84696]: DEBUG cinder.utils [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_7f7b3b2e-e41d-466a-a3c2-e64e3d63b949/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:20:31 user cinder-volume[84696]: DEBUG cinder.utils [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.201(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:20:31 user cinder-volume[84696]: DEBUG cinder.utils [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_c9fa0a17-69d4-40d0-9d65-af6ae349cfd8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:20:31 user cinder-volume[84696]: DEBUG cinder.utils [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.866(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:20:31 user cinder-volume[84696]: DEBUG cinder.utils [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:20:34 user cinder-volume[84696]: DEBUG cinder.utils [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 138.935(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:20:34 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_0a7d4209-92e0-4900-8e98-36a62e68c247/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:20:34 user cinder-volume[84696]: WARNING cinder.volume.manager [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (635d913b-fca8-4373-98cf-065284347dae) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 16:20:34 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (3f23333b-0a38-4265-81bf-f9f7aa177600) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:20:34 user cinder-volume[84696]: WARNING cinder.volume.manager [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (3f23333b-0a38-4265-81bf-f9f7aa177600) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:20:34 user cinder-volume[84696]: WARNING cinder.volume.manager [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Flow 'volume_create_manager' (2c280583-ddcd-4eb7-9be2-f7658b26a36f) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 16:20:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [req-482d9d54-8fe6-4470-94f4-7e368edd2a1a req-530db70f-e58e-4ff0-ae5f-4bc68b7f2df1 tempest-VolumesGetTestJSON-223039682 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 16:20:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 16:20:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 16:20:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 16:20:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 16:20:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 16:20:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 16:20:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 16:20:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 16:20:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 16:20:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 16:20:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 16:20:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 16:20:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 16:20:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 16:20:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 16:20:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 16:20:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 16:20:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 16:20:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 16:20:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 16:20:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 16:20:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 16:20:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 16:20:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:20:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 16:20:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:20:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 16:20:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:20:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 16:20:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:20:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 16:20:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:20:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 16:20:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:20:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 16:20:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:20:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 16:20:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:20:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:20:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 16:20:34 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 16:20:34 user cinder-volume[84696]: DEBUG cinder.utils [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.203(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:20:34 user cinder-volume[84696]: DEBUG cinder.utils [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:20:34 user cinder-volume[84696]: DEBUG cinder.utils [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.378(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:20:34 user cinder-volume[84696]: DEBUG cinder.utils [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_4725fbeb-e869-48db-a487-33998b955f83/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:20:37 user cinder-volume[84696]: DEBUG cinder.utils [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.003(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:20:37 user cinder-volume[84696]: DEBUG cinder.utils [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:20:37 user cinder-volume[84696]: DEBUG cinder.utils [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.175(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:20:37 user cinder-volume[84696]: DEBUG cinder.utils [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_7f7b3b2e-e41d-466a-a3c2-e64e3d63b949/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:20:37 user cinder-volume[84696]: DEBUG cinder.utils [req-bb0706f3-0b2c-473a-acc5-93c2ac742a5f req-59119c1f-bf48-411e-9d13-48936b29a9e6 tempest-ServerStableDeviceRescueTest-869290889 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 72.267(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:20:37 user cinder-volume[84696]: DEBUG cinder.utils [req-bb0706f3-0b2c-473a-acc5-93c2ac742a5f req-59119c1f-bf48-411e-9d13-48936b29a9e6 tempest-ServerStableDeviceRescueTest-869290889 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:32819ec1-2236-465f-a1c7-ee31aa4a0911/luns/v_32819ec1-2236-465f-a1c7-ee31aa4a0911 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:20:40 user cinder-volume[84696]: DEBUG cinder.utils [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 74.191(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:20:40 user cinder-volume[84696]: DEBUG cinder.utils [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:20:40 user cinder-volume[84696]: DEBUG cinder.utils [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.118(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:20:40 user cinder-volume[84696]: DEBUG cinder.utils [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:20:40 user cinder-volume[84696]: DEBUG cinder.utils [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 74.357(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:20:40 user cinder-volume[84696]: DEBUG cinder.utils [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_ae1eb1a7-8e56-418f-9032-a464645060fe/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:20:40 user cinder-volume[84696]: DEBUG cinder.utils [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.415(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:20:40 user cinder-volume[84696]: DEBUG cinder.utils [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_c9fa0a17-69d4-40d0-9d65-af6ae349cfd8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:20:43 user cinder-volume[84696]: DEBUG cinder.utils [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 138.968(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:20:43 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_85fcb006-cd96-4018-b0e3-04872ff08267/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:20:43 user cinder-volume[84696]: WARNING cinder.volume.manager [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (b405f70a-8735-4c3f-9e23-c81972c36950) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 16:20:43 user cinder-volume[84696]: DEBUG cinder.volume.manager [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (8fcb7af7-c087-4742-a88f-580b970d9d51) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:20:43 user cinder-volume[84696]: WARNING cinder.volume.manager [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (8fcb7af7-c087-4742-a88f-580b970d9d51) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:20:43 user cinder-volume[84696]: WARNING cinder.volume.manager [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Flow 'volume_create_manager' (7554f208-702b-4fcd-840a-a9924da76383) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 16:20:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [req-03b99b1c-6c29-4a54-8914-aee31ee083b6 req-726a1377-a691-4c8a-a56d-8b87aed706f7 tempest-VolumesSnapshotsTestJSON-2126874200 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 16:20:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 16:20:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 16:20:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 16:20:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 16:20:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 16:20:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 16:20:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 16:20:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 16:20:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 16:20:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 16:20:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 16:20:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 16:20:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 16:20:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 16:20:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 16:20:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 16:20:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 16:20:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 16:20:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 16:20:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 16:20:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 16:20:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 16:20:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 16:20:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:20:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 16:20:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:20:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 16:20:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:20:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 16:20:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:20:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 16:20:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:20:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 16:20:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:20:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 16:20:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:20:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 16:20:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:20:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:20:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 16:20:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 16:20:44 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=6,cinder:UPDATE=4 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:20:46 user cinder-volume[84696]: DEBUG cinder.utils [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.390(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:20:46 user cinder-volume[84696]: DEBUG cinder.utils [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_7f7b3b2e-e41d-466a-a3c2-e64e3d63b949/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:20:46 user cinder-volume[84696]: DEBUG cinder.utils [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.220(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:20:46 user cinder-volume[84696]: DEBUG cinder.utils [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:20:52 user cinder-volume[84696]: DEBUG cinder.utils [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.790(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:20:52 user cinder-volume[84696]: DEBUG cinder.utils [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_4725fbeb-e869-48db-a487-33998b955f83/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:20:52 user cinder-volume[84696]: DEBUG cinder.utils [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.617(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:20:52 user cinder-volume[84696]: DEBUG cinder.utils [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:20:54 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:INSERT=2,cinder:SELECT=12,cinder:UPDATE=3 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:20:58 user cinder-volume[84696]: DEBUG cinder.utils [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.548(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:20:58 user cinder-volume[84696]: DEBUG cinder.utils [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:20:58 user cinder-volume[84696]: DEBUG cinder.utils [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.844(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:20:58 user cinder-volume[84696]: DEBUG cinder.utils [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_c9fa0a17-69d4-40d0-9d65-af6ae349cfd8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:21:04 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 74.362(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:21:04 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:21:04 user cinder-volume[84696]: DEBUG cinder.utils [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.826(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:21:04 user cinder-volume[84696]: DEBUG cinder.utils [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_7f7b3b2e-e41d-466a-a3c2-e64e3d63b949/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:21:04 user cinder-volume[84696]: DEBUG cinder.utils [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.657(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:21:04 user cinder-volume[84696]: DEBUG cinder.utils [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:21:08 user cinder-volume[84696]: DEBUG cinder.utils [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 136.583(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:21:08 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:21:08 user cinder-volume[84696]: INFO cinder.message.api [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Creating message record for request_id = req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf Apr 17 16:21:08 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 16:21:08 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (4bc28d0c-b383-4171-8362-3ac6f477eb35) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 16:21:08 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 16:21:08 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:18:51Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesTransfersV357Test-Volume-1424534055',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=b900713d-b991-43a5-b403-483dfded7e11,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='1db59dbe212347b8a44e27ebed9e22fb',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:18:51Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:18:51Z,use_quota=True,user_id='ac9545120d3a48bbb11e4cd5e706b53b',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 16:21:08 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:18:51Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesTransfersV357Test-Volume-1424534055',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=b900713d-b991-43a5-b403-483dfded7e11,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='1db59dbe212347b8a44e27ebed9e22fb',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:18:51Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:18:51Z,use_quota=True,user_id='ac9545120d3a48bbb11e4cd5e706b53b',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(b900713d-b991-43a5-b403-483dfded7e11),volume_id=b900713d-b991-43a5-b403-483dfded7e11,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': 'b900713d-b991-43a5-b403-483dfded7e11', 'volume_name': 'volume-b900713d-b991-43a5-b403-483dfded7e11', 'volume_size': 1}} Apr 17 16:21:08 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'retry': {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']}, 'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': 'b900713d-b991-43a5-b403-483dfded7e11', 'volume': {'id': 'b900713d-b991-43a5-b403-483dfded7e11', '_name_id': None, 'ec2_id': None, 'user_id': 'ac9545120d3a48bbb11e4cd5e706b53b', 'project_id': '1db59dbe212347b8a44e27ebed9e22fb', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-VolumesTransfersV357Test-Volume-1424534055', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T16:18:51.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-b900713d-b991-43a5-b403-483dfded7e11', 'name_id': 'b900713d-b991-43a5-b403-483dfded7e11', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumesTransfersV357Test-Volume-1424534055', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '1db59dbe212347b8a44e27ebed9e22fb', 'qos_specs': None, 'reservations': ['b40cb061-d9e6-4c02-bdc0-5d0cb5271144', '8ba69a37-4e7c-4eef-b721-634a940f66f4', 'a799c29a-0d31-4b62-9e2b-b056b451bb3c', 'e510dee0-051c-4a24-a14a-7acbc9a22867'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'ac9545120d3a48bbb11e4cd5e706b53b', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumesTransfersV357Test-Volume-1424534055', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '1db59dbe212347b8a44e27ebed9e22fb', 'qos_specs': None, 'reservations': ['b40cb061-d9e6-4c02-bdc0-5d0cb5271144', '8ba69a37-4e7c-4eef-b721-634a940f66f4', 'a799c29a-0d31-4b62-9e2b-b056b451bb3c', 'e510dee0-051c-4a24-a14a-7acbc9a22867'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'ac9545120d3a48bbb11e4cd5e706b53b', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': 'ac9545120d3a48bbb11e4cd5e706b53b', 'metadata': {}, 'qos_specs': None}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(b900713d-b991-43a5-b403-483dfded7e11),volume_id=b900713d-b991-43a5-b403-483dfded7e11,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:18:51Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesTransfersV357Test-Volume-1424534055',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=b900713d-b991-43a5-b403-483dfded7e11,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='1db59dbe212347b8a44e27ebed9e22fb',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:18:51Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:18:51Z,use_quota=True,user_id='ac9545120d3a48bbb11e4cd5e706b53b',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 16:21:08 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:18:51Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesTransfersV357Test-Volume-1424534055',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=b900713d-b991-43a5-b403-483dfded7e11,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='1db59dbe212347b8a44e27ebed9e22fb',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:18:51Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:18:51Z,use_quota=True,user_id='ac9545120d3a48bbb11e4cd5e706b53b',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:18:51Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesTransfersV357Test-Volume-1424534055',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=b900713d-b991-43a5-b403-483dfded7e11,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='1db59dbe212347b8a44e27ebed9e22fb',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:18:51Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:18:51Z,use_quota=True,user_id='ac9545120d3a48bbb11e4cd5e706b53b',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 16:21:08 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 16:21:08 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 16:21:08 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:21:08 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 16:21:08 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:21:08 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 16:21:08 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:21:08 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 16:21:08 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:21:08 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 16:21:08 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:21:08 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 16:21:08 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:21:08 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 16:21:08 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:21:08 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 16:21:08 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:21:08 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:21:08 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 16:21:08 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 16:21:08 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (4bc28d0c-b383-4171-8362-3ac6f477eb35) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:21:08 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (4bc28d0c-b383-4171-8362-3ac6f477eb35) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:21:08 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (5bc9a8da-86d4-4533-a4f2-8ee45e55019e) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:21:08 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (5bc9a8da-86d4-4533-a4f2-8ee45e55019e) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:21:08 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (cd687451-adf6-4402-994b-208b65cdafa9) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:21:08 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (cd687451-adf6-4402-994b-208b65cdafa9) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:21:08 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (b7ab0fe7-a3c6-4101-9e8e-1d3ad5b552da) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:21:08 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Updating volume b900713d-b991-43a5-b403-483dfded7e11 with {'scheduled_at': datetime.datetime(2023, 4, 17, 16, 21, 8, 91644), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 16:21:08 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Volume b900713d-b991-43a5-b403-483dfded7e11: re-scheduling SchedulerAPI.create_volume attempt 1 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 16:21:08 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Volume b900713d-b991-43a5-b403-483dfded7e11: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 16:21:08 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] deleating volume v_b900713d-b991-43a5-b403-483dfded7e11 {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 16:21:08 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] get snapshots for volume v_b900713d-b991-43a5-b403-483dfded7e11 {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 16:21:08 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_b900713d-b991-43a5-b403-483dfded7e11/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:21:08 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_b900713d-b991-43a5-b403-483dfded7e11/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:21:08 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Volume reschedule parameters: True retry: {'num_attempts': 2, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 16:21:08 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Flow 'volume_create_manager' (f05b4bd3-9089-4b61-bdf1-73a44b40be8c) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 16:21:08 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (f2d11083-eeae-4c90-b3b0-925cc0f83d7e) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:21:08 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (f2d11083-eeae-4c90-b3b0-925cc0f83d7e) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:18:51Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesTransfersV357Test-Volume-1424534055',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=b900713d-b991-43a5-b403-483dfded7e11,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='1db59dbe212347b8a44e27ebed9e22fb',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:21:08Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:21:08Z,use_quota=True,user_id='ac9545120d3a48bbb11e4cd5e706b53b',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:21:08 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (ac6e72c4-80c1-44d3-8c37-428687e574ca) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:21:08 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (ac6e72c4-80c1-44d3-8c37-428687e574ca) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:21:08 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (cb00741d-99d9-404e-b7a5-683616a3f865) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:21:08 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (cb00741d-99d9-404e-b7a5-683616a3f865) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': 'b900713d-b991-43a5-b403-483dfded7e11', 'volume_name': 'volume-b900713d-b991-43a5-b403-483dfded7e11', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:21:08 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (45ea267e-7e55-4f48-9d6c-7a692e92f91f) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:21:08 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (45ea267e-7e55-4f48-9d6c-7a692e92f91f) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:21:08 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (0049b7bf-3755-4315-9906-42130ce19b91) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:21:08 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Volume b900713d-b991-43a5-b403-483dfded7e11: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-b900713d-b991-43a5-b403-483dfded7e11', 'volume_size': 1} Apr 17 16:21:08 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] creating volume v_b900713d-b991-43a5-b403-483dfded7e11. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 16:21:08 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] create volume {'name': 'v_b900713d-b991-43a5-b403-483dfded7e11', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 16:21:08 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:21:08 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:21:11 user cinder-volume[84696]: DEBUG cinder.utils [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.920(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:21:11 user cinder-volume[84696]: DEBUG cinder.utils [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_b900713d-b991-43a5-b403-483dfded7e11/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:21:11 user cinder-volume[84696]: DEBUG cinder.utils [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 138.968(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:21:11 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_bf4c0b13-49c6-4b1d-b790-7090eb82d4c4/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:21:11 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (f8fb11c9-25cc-49cc-b72d-e79139f45094) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 16:21:11 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (a357ad05-6797-4829-b8ab-6b540d6dbeaa) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:21:11 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (a357ad05-6797-4829-b8ab-6b540d6dbeaa) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:21:11 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Flow 'volume_create_manager' (9806d060-778d-4f0e-b2ac-5470f8402414) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 16:21:11 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-e76412e0-459f-4be5-bc08-e01e70270c82 tempest-AttachVolumeTestJSON-1193954556 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 16:21:11 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 16:21:11 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 16:21:11 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 16:21:11 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 16:21:11 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 16:21:11 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 16:21:11 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 16:21:11 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 16:21:11 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 16:21:11 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 16:21:11 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 16:21:11 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 16:21:11 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 16:21:11 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 16:21:11 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 16:21:11 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 16:21:11 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 16:21:11 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 16:21:11 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 16:21:11 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 16:21:11 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 16:21:11 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 16:21:11 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 16:21:11 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:21:11 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 16:21:11 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:21:11 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 16:21:11 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:21:11 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 16:21:11 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:21:11 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 16:21:11 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:21:11 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 16:21:11 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:21:11 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 16:21:11 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:21:11 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 16:21:11 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:21:11 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:21:11 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 16:21:11 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 16:21:11 user cinder-volume[84696]: DEBUG cinder.utils [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.764(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:21:11 user cinder-volume[84696]: DEBUG cinder.utils [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:21:14 user cinder-volume[84696]: DEBUG cinder.utils [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.815(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:21:14 user cinder-volume[84696]: DEBUG cinder.utils [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:21:14 user cinder-volume[84696]: DEBUG cinder.utils [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.995(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:21:14 user cinder-volume[84696]: DEBUG cinder.utils [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_b900713d-b991-43a5-b403-483dfded7e11/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:21:20 user cinder-volume[84696]: DEBUG cinder.utils [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.439(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:21:20 user cinder-volume[84696]: DEBUG cinder.utils [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:21:20 user cinder-volume[84696]: DEBUG cinder.utils [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.617(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:21:20 user cinder-volume[84696]: DEBUG cinder.utils [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_b900713d-b991-43a5-b403-483dfded7e11/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:21:23 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=9,cinder:UPDATE=4,cinder:INSERT=1 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:21:23 user cinder-volume[84696]: INFO cinder.volume.manager [None req-14569ee4-fd22-4f97-ac17-cb6ac7a50989 tempest-GroupsTest-789764105 None] Group tempest-GroupsTest-Group-179260869: creating Apr 17 16:21:23 user cinder-volume[84696]: INFO cinder.volume.manager [None req-14569ee4-fd22-4f97-ac17-cb6ac7a50989 tempest-GroupsTest-789764105 None] Group tempest-GroupsTest-Group-179260869: created successfully Apr 17 16:21:23 user cinder-volume[84696]: INFO cinder.volume.manager [None req-14569ee4-fd22-4f97-ac17-cb6ac7a50989 tempest-GroupsTest-789764105 None] Create group completed successfully. Apr 17 16:21:24 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Volume reschedule parameters: True retry: {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 16:21:24 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Flow 'volume_create_manager' (8fe5f406-9d74-4724-8669-269bd8374179) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 16:21:24 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (9c018aa7-8849-4897-ab64-f67667e36734) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:21:24 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (9c018aa7-8849-4897-ab64-f67667e36734) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:21:24Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-AbsoluteLimitsTests-Volume-1339022265',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=94610127-c8b8-4df4-9e39-a1f41b5a9b98,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='2d881d9502184c74b36e1c50acad1f80',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:21:24Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:21:24Z,use_quota=True,user_id='e460a7fb86b4454788ec1950b15434a4',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:21:24 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (b549076a-a06e-48df-a37e-acd89eb0ce7d) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:21:24 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (b549076a-a06e-48df-a37e-acd89eb0ce7d) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:21:24 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (45fc99b4-3004-4757-add8-c527531a59a7) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:21:24 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (45fc99b4-3004-4757-add8-c527531a59a7) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': '94610127-c8b8-4df4-9e39-a1f41b5a9b98', 'volume_name': 'volume-94610127-c8b8-4df4-9e39-a1f41b5a9b98', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:21:24 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (59d3448b-fe78-4b00-89bb-bc7abcca6374) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:21:24 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (59d3448b-fe78-4b00-89bb-bc7abcca6374) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:21:24 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (4985f125-71bc-4dbb-ae8b-c135918369bd) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:21:24 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Volume 94610127-c8b8-4df4-9e39-a1f41b5a9b98: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-94610127-c8b8-4df4-9e39-a1f41b5a9b98', 'volume_size': 1} Apr 17 16:21:24 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] creating volume v_94610127-c8b8-4df4-9e39-a1f41b5a9b98. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 16:21:24 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] create volume {'name': 'v_94610127-c8b8-4df4-9e39-a1f41b5a9b98', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 16:21:24 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:21:24 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:21:24 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=20,cinder:UPDATE=4,cinder:INSERT=1 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:21:24 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Volume reschedule parameters: True retry: {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 16:21:24 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Flow 'volume_create_manager' (a315cb75-5740-45a4-9e24-b7f1a9e6a279) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 16:21:24 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (cf9fc825-9cbf-4a8f-8652-ab5cc09e6dfe) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:21:24 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (cf9fc825-9cbf-4a8f-8652-ab5cc09e6dfe) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:21:25Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-volume-1452364054',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=fe50cd00-dcd4-4b21-a4e9-6f226f8c6ac1,host='user@joviandss-2#joviandss-2',id=02aba338-fa31-4e46-8fcf-7f86191e44d1,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='e10e34b8774b42d8a4cb1f78d72b411d',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:21:25Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:21:25Z,use_quota=True,user_id='e7392ed432ad41279967273f16b4a4c8',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(400bec8d-bd01-4057-8fb8-49a786cd39f1),volume_type_id=400bec8d-bd01-4057-8fb8-49a786cd39f1)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:21:24 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (d38aba9e-fa93-4bd1-9c3a-5884f9099207) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:21:24 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (d38aba9e-fa93-4bd1-9c3a-5884f9099207) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:21:24 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (e8bb6e08-bb48-464c-9d05-1378354a8f6b) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:21:24 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (e8bb6e08-bb48-464c-9d05-1378354a8f6b) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': '02aba338-fa31-4e46-8fcf-7f86191e44d1', 'volume_name': 'volume-02aba338-fa31-4e46-8fcf-7f86191e44d1', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:21:24 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (18ad6f8d-a5b5-42ca-b79f-9ef4a14cb402) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:21:24 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (18ad6f8d-a5b5-42ca-b79f-9ef4a14cb402) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:21:24 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (8b9bca99-a013-4bf7-86d0-049714fd79dc) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:21:24 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Volume 02aba338-fa31-4e46-8fcf-7f86191e44d1: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-02aba338-fa31-4e46-8fcf-7f86191e44d1', 'volume_size': 1} Apr 17 16:21:24 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] creating volume v_02aba338-fa31-4e46-8fcf-7f86191e44d1. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 16:21:24 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] create volume {'name': 'v_02aba338-fa31-4e46-8fcf-7f86191e44d1', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 16:21:24 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:21:24 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:21:25 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-684f3c97-1d86-45f4-a649-1a4efd9f0051 tempest-GroupsV314Test-467284027 None] remove_export for volume: 7f7b3b2e-e41d-466a-a3c2-e64e3d63b949. {{(pid=84696) remove_export /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:718}} Apr 17 16:21:25 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-684f3c97-1d86-45f4-a649-1a4efd9f0051 tempest-GroupsV314Test-467284027 None] remove export {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:912}} Apr 17 16:21:25 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-684f3c97-1d86-45f4-a649-1a4efd9f0051 tempest-GroupsV314Test-467284027 None] detach volume:Volume(_name_id=None,admin_metadata={},attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:18:08Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-volume-769265242',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=f28ccf81-6553-4a98-83d7-400266578051,host='user@joviandss-2#joviandss-2',id=7f7b3b2e-e41d-466a-a3c2-e64e3d63b949,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='93c135fbb6f14928a9167adb3724faa3',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:20:25Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='deleting',terminated_at=None,updated_at=2023-04-17T16:21:25Z,use_quota=True,user_id='90edc8ffb805460794445bc433aa473f',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(be5f492b-b41f-4ad5-bffd-42eaafb454b4),volume_type_id=be5f492b-b41f-4ad5-bffd-42eaafb454b4) from target:iqn.2010-10.org.openstack:7f7b3b2e-e41d-466a-a3c2-e64e3d63b949. {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:913}} Apr 17 16:21:25 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-684f3c97-1d86-45f4-a649-1a4efd9f0051 tempest-GroupsV314Test-467284027 None] detach volume v_7f7b3b2e-e41d-466a-a3c2-e64e3d63b949 from target iqn.2010-10.org.openstack:7f7b3b2e-e41d-466a-a3c2-e64e3d63b949 {{(pid=84696) detach_target_vol /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:606}} Apr 17 16:21:25 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-684f3c97-1d86-45f4-a649-1a4efd9f0051 tempest-GroupsV314Test-467284027 None] Sending pool request DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:7f7b3b2e-e41d-466a-a3c2-e64e3d63b949/luns/v_7f7b3b2e-e41d-466a-a3c2-e64e3d63b949 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:21:25 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-684f3c97-1d86-45f4-a649-1a4efd9f0051 tempest-GroupsV314Test-467284027 None] Sending DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:7f7b3b2e-e41d-466a-a3c2-e64e3d63b949/luns/v_7f7b3b2e-e41d-466a-a3c2-e64e3d63b949 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:21:27 user cinder-volume[84696]: DEBUG cinder.utils [None req-684f3c97-1d86-45f4-a649-1a4efd9f0051 tempest-GroupsV314Test-467284027 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.096(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:21:27 user cinder-volume[84696]: DEBUG cinder.utils [None req-684f3c97-1d86-45f4-a649-1a4efd9f0051 tempest-GroupsV314Test-467284027 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:7f7b3b2e-e41d-466a-a3c2-e64e3d63b949/luns/v_7f7b3b2e-e41d-466a-a3c2-e64e3d63b949 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:21:27 user cinder-volume[84696]: DEBUG cinder.utils [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 3.073(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:21:27 user cinder-volume[84696]: DEBUG cinder.utils [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:21:27 user cinder-volume[84696]: DEBUG cinder.utils [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 74.289(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:21:27 user cinder-volume[84696]: DEBUG cinder.utils [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_4725fbeb-e869-48db-a487-33998b955f83/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:21:27 user cinder-volume[84696]: DEBUG cinder.utils [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 74.116(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:21:27 user cinder-volume[84696]: DEBUG cinder.utils [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:21:27 user cinder-volume[84696]: DEBUG cinder.utils [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.262(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:21:27 user cinder-volume[84696]: DEBUG cinder.utils [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:21:30 user cinder-volume[84696]: DEBUG cinder.utils [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.879(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:21:30 user cinder-volume[84696]: DEBUG cinder.utils [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:21:30 user cinder-volume[84696]: DEBUG cinder.utils [None req-684f3c97-1d86-45f4-a649-1a4efd9f0051 tempest-GroupsV314Test-467284027 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.169(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:21:30 user cinder-volume[84696]: DEBUG cinder.utils [None req-684f3c97-1d86-45f4-a649-1a4efd9f0051 tempest-GroupsV314Test-467284027 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:7f7b3b2e-e41d-466a-a3c2-e64e3d63b949/luns/v_7f7b3b2e-e41d-466a-a3c2-e64e3d63b949 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:21:30 user cinder-volume[84696]: DEBUG cinder.utils [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 22.058(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:21:30 user cinder-volume[84696]: DEBUG cinder.utils [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_b900713d-b991-43a5-b403-483dfded7e11/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:21:30 user cinder-volume[84696]: DEBUG cinder.utils [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 139.565(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:21:30 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_125cc0cd-98dc-41a8-a252-3ae03797fb6f/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:21:30 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (7e79f666-84e3-4fa2-b6f3-355d494345b0) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 16:21:30 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (d5d74489-1e5a-49f8-a0f0-4f1db3ebb077) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:21:30 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (d5d74489-1e5a-49f8-a0f0-4f1db3ebb077) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:21:30 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Flow 'volume_create_manager' (9b0fcca2-9468-40c5-b0ae-efc49b086cfc) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 16:21:30 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-c6a1a049-a5f6-42d9-8682-cac7fc8587cc tempest-VolumesTransfersV355Test-39117650 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 16:21:30 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 16:21:30 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 16:21:30 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 16:21:30 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 16:21:30 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 16:21:30 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 16:21:30 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 16:21:30 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 16:21:30 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 16:21:30 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 16:21:30 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 16:21:30 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 16:21:30 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 16:21:30 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 16:21:30 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 16:21:30 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 16:21:30 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 16:21:30 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 16:21:30 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 16:21:30 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 16:21:30 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 16:21:30 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 16:21:30 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 16:21:30 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:21:30 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 16:21:30 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:21:30 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 16:21:30 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:21:30 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 16:21:30 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:21:30 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 16:21:30 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:21:30 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 16:21:30 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:21:30 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 16:21:30 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:21:30 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 16:21:30 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:21:30 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:21:30 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 16:21:30 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 16:21:30 user cinder-volume[84696]: DEBUG cinder.utils [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.352(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:21:30 user cinder-volume[84696]: DEBUG cinder.utils [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:21:30 user cinder-volume[84696]: DEBUG cinder.utils [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.169(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:21:30 user cinder-volume[84696]: DEBUG cinder.utils [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:21:33 user cinder-volume[84696]: DEBUG cinder.utils [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 74.044(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:21:33 user cinder-volume[84696]: DEBUG cinder.utils [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:21:33 user cinder-volume[84696]: DEBUG cinder.utils [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 74.341(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:21:33 user cinder-volume[84696]: DEBUG cinder.utils [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_c9fa0a17-69d4-40d0-9d65-af6ae349cfd8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:21:36 user cinder-volume[84696]: DEBUG cinder.utils [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 11.473(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:21:36 user cinder-volume[84696]: DEBUG cinder.utils [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:21:36 user cinder-volume[84696]: DEBUG cinder.utils [None req-684f3c97-1d86-45f4-a649-1a4efd9f0051 tempest-GroupsV314Test-467284027 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 11.313(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:21:36 user cinder-volume[84696]: DEBUG cinder.utils [None req-684f3c97-1d86-45f4-a649-1a4efd9f0051 tempest-GroupsV314Test-467284027 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:7f7b3b2e-e41d-466a-a3c2-e64e3d63b949/luns/v_7f7b3b2e-e41d-466a-a3c2-e64e3d63b949 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:21:36 user cinder-volume[84696]: DEBUG cinder.utils [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.289(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:21:36 user cinder-volume[84696]: DEBUG cinder.utils [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:21:39 user cinder-volume[84696]: DEBUG cinder.utils [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 74.318(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:21:39 user cinder-volume[84696]: DEBUG cinder.utils [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_7f7b3b2e-e41d-466a-a3c2-e64e3d63b949/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:21:39 user cinder-volume[84696]: DEBUG cinder.utils [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 74.149(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:21:39 user cinder-volume[84696]: DEBUG cinder.utils [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:21:42 user cinder-volume[84696]: DEBUG cinder.utils [req-bb0706f3-0b2c-473a-acc5-93c2ac742a5f req-59119c1f-bf48-411e-9d13-48936b29a9e6 tempest-ServerStableDeviceRescueTest-869290889 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 137.481(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:21:42 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [req-bb0706f3-0b2c-473a-acc5-93c2ac742a5f req-59119c1f-bf48-411e-9d13-48936b29a9e6 tempest-ServerStableDeviceRescueTest-869290889 None] Geting None from DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:32819ec1-2236-465f-a1c7-ee31aa4a0911/luns/v_32819ec1-2236-465f-a1c7-ee31aa4a0911 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:21:42 user cinder-volume[84696]: WARNING cinder.volume.manager [req-bb0706f3-0b2c-473a-acc5-93c2ac742a5f req-59119c1f-bf48-411e-9d13-48936b29a9e6 tempest-ServerStableDeviceRescueTest-869290889 None] Failed to detach volume on the backend, ignoring failure 'NoneType' object is not subscriptable Apr 17 16:21:44 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=13,cinder:UPDATE=4 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:21:45 user cinder-volume[84696]: DEBUG cinder.utils [None req-684f3c97-1d86-45f4-a649-1a4efd9f0051 tempest-GroupsV314Test-467284027 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 20.528(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:21:45 user cinder-volume[84696]: DEBUG cinder.utils [None req-684f3c97-1d86-45f4-a649-1a4efd9f0051 tempest-GroupsV314Test-467284027 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:7f7b3b2e-e41d-466a-a3c2-e64e3d63b949/luns/v_7f7b3b2e-e41d-466a-a3c2-e64e3d63b949 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:21:45 user cinder-volume[84696]: DEBUG cinder.utils [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 20.691(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:21:45 user cinder-volume[84696]: DEBUG cinder.utils [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:21:45 user cinder-volume[84696]: DEBUG cinder.utils [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 139.572(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:21:45 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_ae1eb1a7-8e56-418f-9032-a464645060fe/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:21:45 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (3e7569f1-ad43-445a-8c4b-e423989f8f30) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 16:21:45 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (df452e4f-c0ce-427e-8d41-9516da6f9599) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:21:45 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (df452e4f-c0ce-427e-8d41-9516da6f9599) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:21:45 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Flow 'volume_create_manager' (0a6dcad9-ac8f-4ddd-a95d-1d74001447ea) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 16:21:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 16:21:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 16:21:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 16:21:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 16:21:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 16:21:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 16:21:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 16:21:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 16:21:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 16:21:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 16:21:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 16:21:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 16:21:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 16:21:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 16:21:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 16:21:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 16:21:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 16:21:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 16:21:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 16:21:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 16:21:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 16:21:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 16:21:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 16:21:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 16:21:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:21:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 16:21:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:21:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 16:21:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:21:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 16:21:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:21:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 16:21:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:21:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 16:21:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:21:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 16:21:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:21:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 16:21:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:21:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:21:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 16:21:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 16:21:45 user cinder-volume[84696]: DEBUG cinder.utils [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 139.427(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:21:45 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:21:45 user cinder-volume[84696]: INFO cinder.message.api [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Creating message record for request_id = req-5a436bf3-8843-4b13-8b6b-92ee59a23185 Apr 17 16:21:45 user cinder-volume[84696]: DEBUG cinder.utils [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.527(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:21:45 user cinder-volume[84696]: DEBUG cinder.utils [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:21:45 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 16:21:45 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (297c89d2-a771-48ab-a6e2-547a45d137e0) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 16:21:45 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 16:21:45 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:14:49Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-volume-992255480',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=07829a92-7f87-41a0-ad0d-04c53bb24b30,host='user@joviandss-2#joviandss-2',id=ae1eb1a7-8e56-418f-9032-a464645060fe,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='e10e34b8774b42d8a4cb1f78d72b411d',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:19:26Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='deleting',terminated_at=None,updated_at=2023-04-17T16:19:26Z,use_quota=True,user_id='e7392ed432ad41279967273f16b4a4c8',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(39cee13b-7a8b-43bd-ae52-5d071608f5ab),volume_type_id=39cee13b-7a8b-43bd-ae52-5d071608f5ab)}, 'provides': None} Apr 17 16:21:45 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:14:49Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-volume-992255480',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=07829a92-7f87-41a0-ad0d-04c53bb24b30,host='user@joviandss-2#joviandss-2',id=ae1eb1a7-8e56-418f-9032-a464645060fe,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='e10e34b8774b42d8a4cb1f78d72b411d',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:19:26Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='deleting',terminated_at=None,updated_at=2023-04-17T16:19:26Z,use_quota=True,user_id='e7392ed432ad41279967273f16b4a4c8',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(39cee13b-7a8b-43bd-ae52-5d071608f5ab),volume_type_id=39cee13b-7a8b-43bd-ae52-5d071608f5ab), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=07829a92-7f87-41a0-ad0d-04c53bb24b30,image_id=None,operation=,resource_backend='user@joviandss-2',snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(ae1eb1a7-8e56-418f-9032-a464645060fe),volume_id=ae1eb1a7-8e56-418f-9032-a464645060fe,volume_properties=VolumeProperties,volume_type=VolumeType(39cee13b-7a8b-43bd-ae52-5d071608f5ab))}, 'provides': {'status': 'deleting', 'type': 'raw', 'volume_id': 'ae1eb1a7-8e56-418f-9032-a464645060fe', 'volume_name': 'volume-ae1eb1a7-8e56-418f-9032-a464645060fe', 'volume_size': 1}} Apr 17 16:21:45 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'request_spec': {'consistencygroup_id': None, 'group_id': '07829a92-7f87-41a0-ad0d-04c53bb24b30', 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': 'ae1eb1a7-8e56-418f-9032-a464645060fe', 'volume': {'id': 'ae1eb1a7-8e56-418f-9032-a464645060fe', '_name_id': None, 'ec2_id': None, 'user_id': 'e7392ed432ad41279967273f16b4a4c8', 'project_id': 'e10e34b8774b42d8a4cb1f78d72b411d', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-volume-992255480', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '39cee13b-7a8b-43bd-ae52-5d071608f5ab', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': '07829a92-7f87-41a0-ad0d-04c53bb24b30', 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'admin_metadata': {}, 'glance_metadata': {}, 'volume_type': {'id': '39cee13b-7a8b-43bd-ae52-5d071608f5ab', 'name': 'tempest-GroupsTest-volume-type-1704576', 'description': None, 'is_public': True, 'extra_specs': {}, 'qos_specs_id': None, 'created_at': '2023-04-17T16:14:46.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T16:14:48.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-ae1eb1a7-8e56-418f-9032-a464645060fe', 'name_id': 'ae1eb1a7-8e56-418f-9032-a464645060fe', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '39cee13b-7a8b-43bd-ae52-5d071608f5ab', 'name': 'tempest-GroupsTest-volume-type-1704576', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {}, 'qos_specs_id': None, 'created_at': '2023-04-17T16:14:46.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': '07829a92-7f87-41a0-ad0d-04c53bb24b30', 'display_description': None, 'display_name': 'tempest-volume-992255480', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': 'e10e34b8774b42d8a4cb1f78d72b411d', 'qos_specs': None, 'reservations': ['bfa3ce98-41d2-45e2-b7ff-023cf74674f2', '94fac8e5-378b-4745-89de-2454467d68c6', 'c020f3bc-c172-4b95-b5ba-37beec268319', '6cc9ae5a-aab4-4078-bba0-bc05dd3ecd71'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'e7392ed432ad41279967273f16b4a4c8', 'volume_type_id': '39cee13b-7a8b-43bd-ae52-5d071608f5ab'}, 'resource_backend': 'user@joviandss-2', 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': '07829a92-7f87-41a0-ad0d-04c53bb24b30', 'display_description': None, 'display_name': 'tempest-volume-992255480', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': 'e10e34b8774b42d8a4cb1f78d72b411d', 'qos_specs': None, 'reservations': ['bfa3ce98-41d2-45e2-b7ff-023cf74674f2', '94fac8e5-378b-4745-89de-2454467d68c6', 'c020f3bc-c172-4b95-b5ba-37beec268319', '6cc9ae5a-aab4-4078-bba0-bc05dd3ecd71'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'e7392ed432ad41279967273f16b4a4c8', 'volume_type_id': '39cee13b-7a8b-43bd-ae52-5d071608f5ab'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T16:14:46Z,deleted=False,deleted_at=None,description=None,extra_specs={},id=39cee13b-7a8b-43bd-ae52-5d071608f5ab,is_public=True,name='tempest-GroupsTest-volume-type-1704576',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T16:14:46Z,deleted=False,deleted_at=None,description=None,extra_specs={},id=39cee13b-7a8b-43bd-ae52-5d071608f5ab,is_public=True,name='tempest-GroupsTest-volume-type-1704576',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': 'e7392ed432ad41279967273f16b4a4c8', 'metadata': {}, 'qos_specs': None, 'retry': {'num_attempts': 3, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']}}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=07829a92-7f87-41a0-ad0d-04c53bb24b30,image_id=None,operation=,resource_backend='user@joviandss-2',snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(ae1eb1a7-8e56-418f-9032-a464645060fe),volume_id=ae1eb1a7-8e56-418f-9032-a464645060fe,volume_properties=VolumeProperties,volume_type=VolumeType(39cee13b-7a8b-43bd-ae52-5d071608f5ab)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:14:49Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-volume-992255480',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=07829a92-7f87-41a0-ad0d-04c53bb24b30,host='user@joviandss-2#joviandss-2',id=ae1eb1a7-8e56-418f-9032-a464645060fe,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='e10e34b8774b42d8a4cb1f78d72b411d',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:19:26Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='deleting',terminated_at=None,updated_at=2023-04-17T16:19:26Z,use_quota=True,user_id='e7392ed432ad41279967273f16b4a4c8',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(39cee13b-7a8b-43bd-ae52-5d071608f5ab),volume_type_id=39cee13b-7a8b-43bd-ae52-5d071608f5ab), 'context': }, 'provides': None} Apr 17 16:21:45 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:14:49Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-volume-992255480',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=07829a92-7f87-41a0-ad0d-04c53bb24b30,host='user@joviandss-2#joviandss-2',id=ae1eb1a7-8e56-418f-9032-a464645060fe,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='e10e34b8774b42d8a4cb1f78d72b411d',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:19:26Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='deleting',terminated_at=None,updated_at=2023-04-17T16:19:26Z,use_quota=True,user_id='e7392ed432ad41279967273f16b4a4c8',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(39cee13b-7a8b-43bd-ae52-5d071608f5ab),volume_type_id=39cee13b-7a8b-43bd-ae52-5d071608f5ab)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:14:49Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-volume-992255480',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=07829a92-7f87-41a0-ad0d-04c53bb24b30,host='user@joviandss-2#joviandss-2',id=ae1eb1a7-8e56-418f-9032-a464645060fe,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='e10e34b8774b42d8a4cb1f78d72b411d',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:19:26Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='deleting',terminated_at=None,updated_at=2023-04-17T16:19:26Z,use_quota=True,user_id='e7392ed432ad41279967273f16b4a4c8',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(39cee13b-7a8b-43bd-ae52-5d071608f5ab),volume_type_id=39cee13b-7a8b-43bd-ae52-5d071608f5ab)} Apr 17 16:21:45 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 16:21:45 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 16:21:45 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:21:45 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 16:21:45 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:21:45 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 16:21:45 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:21:45 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 16:21:45 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:21:45 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 16:21:45 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:21:45 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 16:21:45 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:21:45 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 16:21:45 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:21:45 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 16:21:45 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:21:45 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:21:45 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 16:21:45 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 16:21:45 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (297c89d2-a771-48ab-a6e2-547a45d137e0) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:21:45 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (297c89d2-a771-48ab-a6e2-547a45d137e0) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:21:45 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (f6949b78-8685-4ca5-b86b-ff8946012815) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:21:45 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (f6949b78-8685-4ca5-b86b-ff8946012815) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:21:45 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (936eacde-57bc-4e09-acfb-84841b2415ba) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:21:45 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (936eacde-57bc-4e09-acfb-84841b2415ba) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:21:45 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (a83887a1-4d90-481a-a257-ad22ea100a28) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:21:45 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Updating volume ae1eb1a7-8e56-418f-9032-a464645060fe with {'scheduled_at': datetime.datetime(2023, 4, 17, 16, 21, 45, 676190), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 16:21:45 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Volume ae1eb1a7-8e56-418f-9032-a464645060fe: re-scheduling SchedulerAPI.create_volume attempt 3 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 16:21:45 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Volume ae1eb1a7-8e56-418f-9032-a464645060fe: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 16:21:45 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] deleating volume v_ae1eb1a7-8e56-418f-9032-a464645060fe {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 16:21:45 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] get snapshots for volume v_ae1eb1a7-8e56-418f-9032-a464645060fe {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 16:21:45 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_ae1eb1a7-8e56-418f-9032-a464645060fe/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:21:45 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_ae1eb1a7-8e56-418f-9032-a464645060fe/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:21:48 user cinder-volume[84696]: DEBUG cinder.utils [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 40.311(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:21:48 user cinder-volume[84696]: DEBUG cinder.utils [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:21:48 user cinder-volume[84696]: DEBUG cinder.utils [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.966(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:21:48 user cinder-volume[84696]: DEBUG cinder.utils [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_ae1eb1a7-8e56-418f-9032-a464645060fe/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:21:48 user cinder-volume[84696]: DEBUG cinder.utils [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 40.491(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:21:48 user cinder-volume[84696]: DEBUG cinder.utils [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_b900713d-b991-43a5-b403-483dfded7e11/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:21:51 user cinder-volume[84696]: DEBUG cinder.utils [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.041(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:21:51 user cinder-volume[84696]: DEBUG cinder.utils [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_ae1eb1a7-8e56-418f-9032-a464645060fe/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:21:58 user cinder-volume[84696]: DEBUG cinder.utils [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 13.108(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:21:58 user cinder-volume[84696]: DEBUG cinder.utils [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_ae1eb1a7-8e56-418f-9032-a464645060fe/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:22:01 user cinder-volume[84696]: DEBUG cinder.utils [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 37.791(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:22:01 user cinder-volume[84696]: DEBUG cinder.utils [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:22:01 user cinder-volume[84696]: DEBUG cinder.utils [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 36.978(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:22:01 user cinder-volume[84696]: DEBUG cinder.utils [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:22:01 user cinder-volume[84696]: DEBUG cinder.utils [None req-684f3c97-1d86-45f4-a649-1a4efd9f0051 tempest-GroupsV314Test-467284027 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 36.818(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:22:01 user cinder-volume[84696]: DEBUG cinder.utils [None req-684f3c97-1d86-45f4-a649-1a4efd9f0051 tempest-GroupsV314Test-467284027 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:7f7b3b2e-e41d-466a-a3c2-e64e3d63b949/luns/v_7f7b3b2e-e41d-466a-a3c2-e64e3d63b949 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:22:04 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=13,cinder:UPDATE=2,cinder:INSERT=1 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:22:08 user cinder-volume[84696]: DEBUG cinder.utils [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 23.192(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:22:08 user cinder-volume[84696]: DEBUG cinder.utils [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_ae1eb1a7-8e56-418f-9032-a464645060fe/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:22:11 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Volume reschedule parameters: True retry: {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 16:22:11 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Flow 'volume_create_manager' (481e1710-a15b-499f-8369-3ea87288fff6) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 16:22:11 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (08f1b9ad-1bbb-446c-85cd-796b0907142d) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:22:11 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (08f1b9ad-1bbb-446c-85cd-796b0907142d) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:22:11Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesNegativeTest-Volume-375402817',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=2e8580be-c7f2-4d3a-9007-d192a5b029b3,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='935190a703294690b79c182d789f31e4',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:22:11Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:22:11Z,use_quota=True,user_id='9296aa0c28024371bac0c53928bff421',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:22:11 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (242c0d4b-5c14-4989-a5d0-47d010c8ba81) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:22:11 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (242c0d4b-5c14-4989-a5d0-47d010c8ba81) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:22:11 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (b12c230c-770d-447e-a211-66cab96aa07a) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:22:11 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (b12c230c-770d-447e-a211-66cab96aa07a) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': '2e8580be-c7f2-4d3a-9007-d192a5b029b3', 'volume_name': 'volume-2e8580be-c7f2-4d3a-9007-d192a5b029b3', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:22:11 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (3408d748-cf33-44b9-9c31-4ef573d30a9a) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:22:11 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (3408d748-cf33-44b9-9c31-4ef573d30a9a) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:22:11 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (987bfcf0-d50d-414e-819e-cbf51ab05454) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:22:11 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Volume 2e8580be-c7f2-4d3a-9007-d192a5b029b3: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-2e8580be-c7f2-4d3a-9007-d192a5b029b3', 'volume_size': 1} Apr 17 16:22:11 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] creating volume v_2e8580be-c7f2-4d3a-9007-d192a5b029b3. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 16:22:11 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] create volume {'name': 'v_2e8580be-c7f2-4d3a-9007-d192a5b029b3', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 16:22:11 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:22:11 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:22:11 user cinder-volume[84696]: DEBUG cinder.utils [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 0.437(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:22:11 user cinder-volume[84696]: DEBUG cinder.utils [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:22:11 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 141.373(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:22:11 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:22:11 user cinder-volume[84696]: ERROR oslo_service.periodic_task [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Error during VolumeManager.publish_service_capabilities: TypeError: 'NoneType' object is not subscriptable Apr 17 16:22:11 user cinder-volume[84696]: ERROR oslo_service.periodic_task Traceback (most recent call last): Apr 17 16:22:11 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/usr/local/lib/python3.10/dist-packages/oslo_service/periodic_task.py", line 216, in run_periodic_tasks Apr 17 16:22:11 user cinder-volume[84696]: ERROR oslo_service.periodic_task task(self, context) Apr 17 16:22:11 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/manager.py", line 2876, in publish_service_capabilities Apr 17 16:22:11 user cinder-volume[84696]: ERROR oslo_service.periodic_task self._report_driver_status(context) Apr 17 16:22:11 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/manager.py", line 2755, in _report_driver_status Apr 17 16:22:11 user cinder-volume[84696]: ERROR oslo_service.periodic_task volume_stats = get_stats() Apr 17 16:22:11 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/usr/local/lib/python3.10/dist-packages/oslo_utils/timeutils.py", line 319, in wrapper Apr 17 16:22:11 user cinder-volume[84696]: ERROR oslo_service.periodic_task result = func(*args, **kwargs) Apr 17 16:22:11 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/manager.py", line 2753, in get_stats Apr 17 16:22:11 user cinder-volume[84696]: ERROR oslo_service.periodic_task return self.driver.get_volume_stats(refresh=True) Apr 17 16:22:11 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/driver.py", line 666, in get_volume_stats Apr 17 16:22:11 user cinder-volume[84696]: ERROR oslo_service.periodic_task self._update_volume_stats() Apr 17 16:22:11 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 726, in _update_volume_stats Apr 17 16:22:11 user cinder-volume[84696]: ERROR oslo_service.periodic_task pool_stats = self.ra.get_pool_stats() Apr 17 16:22:11 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 897, in get_pool_stats Apr 17 16:22:11 user cinder-volume[84696]: ERROR oslo_service.periodic_task if not resp["error"] and resp["code"] == 200: Apr 17 16:22:11 user cinder-volume[84696]: ERROR oslo_service.periodic_task TypeError: 'NoneType' object is not subscriptable Apr 17 16:22:11 user cinder-volume[84696]: ERROR oslo_service.periodic_task Apr 17 16:22:11 user cinder-volume[84696]: WARNING oslo.service.loopingcall [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Function 'cinder.service.Service.periodic_tasks' run outlasted interval by 81.38 sec Apr 17 16:22:11 user cinder-volume[84696]: DEBUG oslo_service.periodic_task [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Running periodic task VolumeManager.publish_service_capabilities {{(pid=84696) run_periodic_tasks /usr/local/lib/python3.10/dist-packages/oslo_service/periodic_task.py:210}} Apr 17 16:22:11 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Updating volume stats {{(pid=84696) _update_volume_stats /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:724}} Apr 17 16:22:11 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Get pool Pool-0 fsprops {{(pid=84696) get_pool_stats /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:894}} Apr 17 16:22:11 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:22:11 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:22:15 user cinder-volume[84696]: DEBUG cinder.utils [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 3.512(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:22:15 user cinder-volume[84696]: DEBUG cinder.utils [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:22:15 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 3.069(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:22:15 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:22:18 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.135(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:22:18 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:22:21 user cinder-volume[84696]: DEBUG cinder.utils [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 9.652(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:22:21 user cinder-volume[84696]: DEBUG cinder.utils [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:22:21 user cinder-volume[84696]: DEBUG cinder.utils [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 73.001(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:22:21 user cinder-volume[84696]: DEBUG cinder.utils [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_b900713d-b991-43a5-b403-483dfded7e11/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:22:21 user cinder-volume[84696]: DEBUG cinder.utils [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 72.825(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:22:21 user cinder-volume[84696]: DEBUG cinder.utils [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:22:24 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.279(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:22:24 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:22:24 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=5 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:22:24 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=9,cinder:INSERT=1,cinder:UPDATE=5 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:22:27 user cinder-volume[84696]: DEBUG cinder.utils [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 41.620(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:22:27 user cinder-volume[84696]: DEBUG cinder.utils [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_ae1eb1a7-8e56-418f-9032-a464645060fe/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:22:30 user cinder-volume[84696]: DEBUG cinder.utils [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 18.868(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:22:30 user cinder-volume[84696]: DEBUG cinder.utils [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:22:33 user cinder-volume[84696]: DEBUG cinder.utils [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 140.590(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:22:33 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_4725fbeb-e869-48db-a487-33998b955f83/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:22:33 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (6b890aa1-d214-4bf7-9f8c-9816e94ee7a7) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 16:22:33 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (54f43ff6-f144-496d-9c2c-4cf1066f8185) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:22:33 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (54f43ff6-f144-496d-9c2c-4cf1066f8185) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:22:33 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Flow 'volume_create_manager' (e593c020-6a5b-4951-8d29-422a69e0c19c) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 16:22:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 16:22:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 16:22:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 16:22:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 16:22:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 16:22:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 16:22:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 16:22:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 16:22:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 16:22:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 16:22:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 16:22:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 16:22:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 16:22:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 16:22:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 16:22:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 16:22:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 16:22:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 16:22:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 16:22:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 16:22:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 16:22:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 16:22:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 16:22:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 16:22:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:22:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 16:22:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:22:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 16:22:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:22:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 16:22:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:22:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 16:22:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:22:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 16:22:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:22:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 16:22:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:22:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 16:22:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:22:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:22:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 16:22:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 16:22:33 user cinder-volume[84696]: DEBUG cinder.utils [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 140.441(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:22:33 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:22:33 user cinder-volume[84696]: INFO cinder.message.api [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Creating message record for request_id = req-86e82c7b-2d41-4c71-a488-5361f4ac7ada Apr 17 16:22:33 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.527(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:22:33 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:22:33 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 16:22:33 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (59b5ebbf-9898-4b59-82a2-bb496efdf3cb) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 16:22:33 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 16:22:33 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:15:34Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesTransfersV357Test-Volume-788739626',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=4725fbeb-e869-48db-a487-33998b955f83,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='1db59dbe212347b8a44e27ebed9e22fb',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:20:13Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:20:13Z,use_quota=True,user_id='ac9545120d3a48bbb11e4cd5e706b53b',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 16:22:33 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:15:34Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesTransfersV357Test-Volume-788739626',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=4725fbeb-e869-48db-a487-33998b955f83,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='1db59dbe212347b8a44e27ebed9e22fb',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:20:13Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:20:13Z,use_quota=True,user_id='ac9545120d3a48bbb11e4cd5e706b53b',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(4725fbeb-e869-48db-a487-33998b955f83),volume_id=4725fbeb-e869-48db-a487-33998b955f83,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': '4725fbeb-e869-48db-a487-33998b955f83', 'volume_name': 'volume-4725fbeb-e869-48db-a487-33998b955f83', 'volume_size': 1}} Apr 17 16:22:33 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': '4725fbeb-e869-48db-a487-33998b955f83', 'volume': {'id': '4725fbeb-e869-48db-a487-33998b955f83', '_name_id': None, 'ec2_id': None, 'user_id': 'ac9545120d3a48bbb11e4cd5e706b53b', 'project_id': '1db59dbe212347b8a44e27ebed9e22fb', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-VolumesTransfersV357Test-Volume-788739626', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'admin_metadata': {}, 'glance_metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T16:15:34.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-4725fbeb-e869-48db-a487-33998b955f83', 'name_id': '4725fbeb-e869-48db-a487-33998b955f83', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumesTransfersV357Test-Volume-788739626', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '1db59dbe212347b8a44e27ebed9e22fb', 'qos_specs': None, 'reservations': ['08ac1f5a-fa3d-4bd2-a307-674643af97ec', 'fb4977a0-e1cc-4948-8a75-8a55838deb4a', 'ee391e6b-8fe6-433e-8b00-937d5eb2b94a', '883ff3d2-1fda-44dd-a663-1d9481ae2b2b'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'ac9545120d3a48bbb11e4cd5e706b53b', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumesTransfersV357Test-Volume-788739626', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '1db59dbe212347b8a44e27ebed9e22fb', 'qos_specs': None, 'reservations': ['08ac1f5a-fa3d-4bd2-a307-674643af97ec', 'fb4977a0-e1cc-4948-8a75-8a55838deb4a', 'ee391e6b-8fe6-433e-8b00-937d5eb2b94a', '883ff3d2-1fda-44dd-a663-1d9481ae2b2b'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'ac9545120d3a48bbb11e4cd5e706b53b', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': 'ac9545120d3a48bbb11e4cd5e706b53b', 'metadata': {}, 'qos_specs': None, 'retry': {'num_attempts': 3, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']}}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(4725fbeb-e869-48db-a487-33998b955f83),volume_id=4725fbeb-e869-48db-a487-33998b955f83,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:15:34Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesTransfersV357Test-Volume-788739626',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=4725fbeb-e869-48db-a487-33998b955f83,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='1db59dbe212347b8a44e27ebed9e22fb',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:20:13Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:20:13Z,use_quota=True,user_id='ac9545120d3a48bbb11e4cd5e706b53b',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 16:22:33 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:15:34Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesTransfersV357Test-Volume-788739626',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=4725fbeb-e869-48db-a487-33998b955f83,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='1db59dbe212347b8a44e27ebed9e22fb',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:20:13Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:20:13Z,use_quota=True,user_id='ac9545120d3a48bbb11e4cd5e706b53b',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:15:34Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesTransfersV357Test-Volume-788739626',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=4725fbeb-e869-48db-a487-33998b955f83,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='1db59dbe212347b8a44e27ebed9e22fb',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:20:13Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:20:13Z,use_quota=True,user_id='ac9545120d3a48bbb11e4cd5e706b53b',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 16:22:33 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 16:22:33 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 16:22:33 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:22:33 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 16:22:33 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:22:33 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 16:22:33 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:22:33 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 16:22:33 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:22:33 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 16:22:33 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:22:33 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 16:22:33 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:22:33 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 16:22:33 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:22:33 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 16:22:33 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:22:33 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:22:33 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 16:22:33 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 16:22:33 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (59b5ebbf-9898-4b59-82a2-bb496efdf3cb) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:22:33 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (59b5ebbf-9898-4b59-82a2-bb496efdf3cb) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:22:33 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (12e1189e-6a4e-4989-8030-9c5bc8251a16) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:22:33 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (12e1189e-6a4e-4989-8030-9c5bc8251a16) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:22:33 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (01829bcc-10ab-4c69-92ea-f2f7a1c958ec) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:22:33 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (01829bcc-10ab-4c69-92ea-f2f7a1c958ec) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:22:33 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (0ea2e582-cdc9-48a0-a628-5209cfb9fbf9) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:22:33 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Updating volume 4725fbeb-e869-48db-a487-33998b955f83 with {'scheduled_at': datetime.datetime(2023, 4, 17, 16, 22, 33, 561673), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 16:22:33 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Volume 4725fbeb-e869-48db-a487-33998b955f83: re-scheduling SchedulerAPI.create_volume attempt 3 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 16:22:33 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Volume 4725fbeb-e869-48db-a487-33998b955f83: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 16:22:33 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] deleating volume v_4725fbeb-e869-48db-a487-33998b955f83 {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 16:22:33 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] get snapshots for volume v_4725fbeb-e869-48db-a487-33998b955f83 {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 16:22:33 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_4725fbeb-e869-48db-a487-33998b955f83/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:22:33 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_4725fbeb-e869-48db-a487-33998b955f83/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:22:36 user cinder-volume[84696]: DEBUG cinder.utils [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 72.447(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:22:36 user cinder-volume[84696]: DEBUG cinder.utils [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:22:36 user cinder-volume[84696]: DEBUG cinder.utils [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 71.635(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:22:36 user cinder-volume[84696]: DEBUG cinder.utils [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:22:36 user cinder-volume[84696]: DEBUG cinder.utils [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.942(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:22:36 user cinder-volume[84696]: DEBUG cinder.utils [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_4725fbeb-e869-48db-a487-33998b955f83/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:22:36 user cinder-volume[84696]: DEBUG cinder.utils [None req-684f3c97-1d86-45f4-a649-1a4efd9f0051 tempest-GroupsV314Test-467284027 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 71.478(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:22:36 user cinder-volume[84696]: DEBUG cinder.utils [None req-684f3c97-1d86-45f4-a649-1a4efd9f0051 tempest-GroupsV314Test-467284027 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:7f7b3b2e-e41d-466a-a3c2-e64e3d63b949/luns/v_7f7b3b2e-e41d-466a-a3c2-e64e3d63b949 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:22:39 user cinder-volume[84696]: DEBUG cinder.utils [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.014(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:22:39 user cinder-volume[84696]: DEBUG cinder.utils [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_4725fbeb-e869-48db-a487-33998b955f83/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:22:39 user cinder-volume[84696]: DEBUG cinder.utils [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 140.648(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:22:39 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_c9fa0a17-69d4-40d0-9d65-af6ae349cfd8/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:22:39 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (3965cc00-20f6-4545-ab02-6b9abc6c69ad) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 16:22:39 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (8d479b1f-9718-4662-aa70-91f16dfc6bc1) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:22:39 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (8d479b1f-9718-4662-aa70-91f16dfc6bc1) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:22:39 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Flow 'volume_create_manager' (4fbdb665-19d0-4458-b393-02b7f20b5b51) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 16:22:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 16:22:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 16:22:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 16:22:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 16:22:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 16:22:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 16:22:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 16:22:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 16:22:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 16:22:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 16:22:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 16:22:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 16:22:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 16:22:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 16:22:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 16:22:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 16:22:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 16:22:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 16:22:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 16:22:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 16:22:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 16:22:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 16:22:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 16:22:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 16:22:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:22:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 16:22:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:22:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 16:22:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:22:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 16:22:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:22:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 16:22:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:22:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 16:22:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:22:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 16:22:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:22:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 16:22:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:22:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:22:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 16:22:39 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 16:22:39 user cinder-volume[84696]: DEBUG cinder.utils [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 140.376(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:22:39 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:22:39 user cinder-volume[84696]: INFO cinder.message.api [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Creating message record for request_id = req-f436a31c-1175-4e56-a839-11ba99c1cd60 Apr 17 16:22:39 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 16:22:39 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (9e273a19-870b-4921-99e6-3e9d030b9cc1) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 16:22:39 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 16:22:39 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:18:00Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumeSnapshotQuotasNegativeTestJSON-Volume-1211070769',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=c9fa0a17-69d4-40d0-9d65-af6ae349cfd8,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='46b47eb1ecf7468abe1458b30e249be8',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:20:19Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:20:19Z,use_quota=True,user_id='48df3e98011f41a7938b3da3e07615c2',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 16:22:39 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:18:00Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumeSnapshotQuotasNegativeTestJSON-Volume-1211070769',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=c9fa0a17-69d4-40d0-9d65-af6ae349cfd8,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='46b47eb1ecf7468abe1458b30e249be8',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:20:19Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:20:19Z,use_quota=True,user_id='48df3e98011f41a7938b3da3e07615c2',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(c9fa0a17-69d4-40d0-9d65-af6ae349cfd8),volume_id=c9fa0a17-69d4-40d0-9d65-af6ae349cfd8,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': 'c9fa0a17-69d4-40d0-9d65-af6ae349cfd8', 'volume_name': 'volume-c9fa0a17-69d4-40d0-9d65-af6ae349cfd8', 'volume_size': 1}} Apr 17 16:22:39 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': 'c9fa0a17-69d4-40d0-9d65-af6ae349cfd8', 'volume': {'id': 'c9fa0a17-69d4-40d0-9d65-af6ae349cfd8', '_name_id': None, 'ec2_id': None, 'user_id': '48df3e98011f41a7938b3da3e07615c2', 'project_id': '46b47eb1ecf7468abe1458b30e249be8', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-VolumeSnapshotQuotasNegativeTestJSON-Volume-1211070769', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'admin_metadata': {}, 'glance_metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T16:17:59.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-c9fa0a17-69d4-40d0-9d65-af6ae349cfd8', 'name_id': 'c9fa0a17-69d4-40d0-9d65-af6ae349cfd8', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumeSnapshotQuotasNegativeTestJSON-Volume-1211070769', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '46b47eb1ecf7468abe1458b30e249be8', 'qos_specs': None, 'reservations': ['019a7782-5155-4c89-a83b-235283c575e4', '0bfbe139-e9e6-4454-941c-cc4e8c814eb3', 'df625689-4086-45a1-a606-f6541e8045a3', '80b6f310-1127-40c3-bb09-d4d6b83f85c6'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': '48df3e98011f41a7938b3da3e07615c2', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumeSnapshotQuotasNegativeTestJSON-Volume-1211070769', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '46b47eb1ecf7468abe1458b30e249be8', 'qos_specs': None, 'reservations': ['019a7782-5155-4c89-a83b-235283c575e4', '0bfbe139-e9e6-4454-941c-cc4e8c814eb3', 'df625689-4086-45a1-a606-f6541e8045a3', '80b6f310-1127-40c3-bb09-d4d6b83f85c6'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': '48df3e98011f41a7938b3da3e07615c2', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': '48df3e98011f41a7938b3da3e07615c2', 'metadata': {}, 'qos_specs': None, 'retry': {'num_attempts': 2, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']}}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(c9fa0a17-69d4-40d0-9d65-af6ae349cfd8),volume_id=c9fa0a17-69d4-40d0-9d65-af6ae349cfd8,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:18:00Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumeSnapshotQuotasNegativeTestJSON-Volume-1211070769',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=c9fa0a17-69d4-40d0-9d65-af6ae349cfd8,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='46b47eb1ecf7468abe1458b30e249be8',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:20:19Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:20:19Z,use_quota=True,user_id='48df3e98011f41a7938b3da3e07615c2',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 16:22:39 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:18:00Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumeSnapshotQuotasNegativeTestJSON-Volume-1211070769',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=c9fa0a17-69d4-40d0-9d65-af6ae349cfd8,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='46b47eb1ecf7468abe1458b30e249be8',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:20:19Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:20:19Z,use_quota=True,user_id='48df3e98011f41a7938b3da3e07615c2',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:18:00Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumeSnapshotQuotasNegativeTestJSON-Volume-1211070769',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=c9fa0a17-69d4-40d0-9d65-af6ae349cfd8,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='46b47eb1ecf7468abe1458b30e249be8',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:20:19Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:20:19Z,use_quota=True,user_id='48df3e98011f41a7938b3da3e07615c2',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 16:22:39 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 16:22:39 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 16:22:39 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:22:39 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 16:22:39 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:22:39 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 16:22:39 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:22:39 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 16:22:39 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:22:39 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 16:22:39 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:22:39 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 16:22:39 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:22:39 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 16:22:39 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:22:39 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 16:22:39 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:22:39 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:22:39 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 16:22:39 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 16:22:39 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (9e273a19-870b-4921-99e6-3e9d030b9cc1) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:22:39 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (9e273a19-870b-4921-99e6-3e9d030b9cc1) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:22:39 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (1f136c37-038d-4b49-99b2-e7ce751248b7) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:22:39 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (1f136c37-038d-4b49-99b2-e7ce751248b7) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:22:39 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (fcf5d16f-4999-4f67-ae30-36a2355e93d4) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:22:39 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (fcf5d16f-4999-4f67-ae30-36a2355e93d4) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:22:39 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (85d4bcaf-ef93-4bed-8fe3-77dd128ef3ae) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:22:39 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Updating volume c9fa0a17-69d4-40d0-9d65-af6ae349cfd8 with {'scheduled_at': datetime.datetime(2023, 4, 17, 16, 22, 39, 698088), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 16:22:39 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Volume c9fa0a17-69d4-40d0-9d65-af6ae349cfd8: re-scheduling SchedulerAPI.create_volume attempt 2 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 16:22:39 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Volume c9fa0a17-69d4-40d0-9d65-af6ae349cfd8: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 16:22:39 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] deleating volume v_c9fa0a17-69d4-40d0-9d65-af6ae349cfd8 {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 16:22:39 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] get snapshots for volume v_c9fa0a17-69d4-40d0-9d65-af6ae349cfd8 {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 16:22:39 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_c9fa0a17-69d4-40d0-9d65-af6ae349cfd8/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:22:39 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_c9fa0a17-69d4-40d0-9d65-af6ae349cfd8/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:22:39 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Volume reschedule parameters: True retry: {'num_attempts': 3, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 16:22:39 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Flow 'volume_create_manager' (fe1e710e-d987-444a-80f3-30b4d3d04581) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 16:22:39 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (1fb8c9f4-3ea6-4074-a3b7-6276b466d7e4) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:22:39 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (1fb8c9f4-3ea6-4074-a3b7-6276b466d7e4) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:18:00Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumeSnapshotQuotasNegativeTestJSON-Volume-1211070769',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=c9fa0a17-69d4-40d0-9d65-af6ae349cfd8,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='46b47eb1ecf7468abe1458b30e249be8',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:22:40Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:22:40Z,use_quota=True,user_id='48df3e98011f41a7938b3da3e07615c2',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:22:39 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (28e9bdbf-3834-419e-8596-3394b9b72ead) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:22:39 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (28e9bdbf-3834-419e-8596-3394b9b72ead) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:22:39 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (1f63cb8c-4036-4247-8a6b-493a2cda3fa9) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:22:39 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (1f63cb8c-4036-4247-8a6b-493a2cda3fa9) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': 'c9fa0a17-69d4-40d0-9d65-af6ae349cfd8', 'volume_name': 'volume-c9fa0a17-69d4-40d0-9d65-af6ae349cfd8', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:22:39 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (86f3be07-58ed-409e-bd03-d75ee4497c5c) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:22:39 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (86f3be07-58ed-409e-bd03-d75ee4497c5c) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:22:39 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (a388ee1d-0d27-46cd-a4cf-0ab732a282d7) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:22:39 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Volume c9fa0a17-69d4-40d0-9d65-af6ae349cfd8: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-c9fa0a17-69d4-40d0-9d65-af6ae349cfd8', 'volume_size': 1} Apr 17 16:22:39 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] creating volume v_c9fa0a17-69d4-40d0-9d65-af6ae349cfd8. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 16:22:39 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] create volume {'name': 'v_c9fa0a17-69d4-40d0-9d65-af6ae349cfd8', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 16:22:39 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:22:39 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:22:42 user cinder-volume[84696]: DEBUG cinder.utils [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.963(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:22:42 user cinder-volume[84696]: DEBUG cinder.utils [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_c9fa0a17-69d4-40d0-9d65-af6ae349cfd8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:22:42 user cinder-volume[84696]: DEBUG cinder.utils [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.744(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:22:42 user cinder-volume[84696]: DEBUG cinder.utils [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:22:45 user cinder-volume[84696]: DEBUG cinder.utils [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.031(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:22:45 user cinder-volume[84696]: DEBUG cinder.utils [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_c9fa0a17-69d4-40d0-9d65-af6ae349cfd8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:22:45 user cinder-volume[84696]: DEBUG cinder.utils [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.156(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:22:45 user cinder-volume[84696]: DEBUG cinder.utils [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_4725fbeb-e869-48db-a487-33998b955f83/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:22:45 user cinder-volume[84696]: DEBUG cinder.utils [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 140.454(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:22:45 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:22:45 user cinder-volume[84696]: INFO cinder.message.api [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Creating message record for request_id = req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 Apr 17 16:22:45 user cinder-volume[84696]: DEBUG cinder.utils [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 140.630(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:22:45 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_7f7b3b2e-e41d-466a-a3c2-e64e3d63b949/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:22:45 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (d7828182-b79b-4795-957b-3881571a5bc7) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 16:22:45 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (c96ba169-e55d-4bf5-90e7-5e8c5bbb9d4d) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:22:45 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (c96ba169-e55d-4bf5-90e7-5e8c5bbb9d4d) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:22:45 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Flow 'volume_create_manager' (75c8bceb-8d3c-46e1-8de8-c50a87b028f8) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 16:22:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 16:22:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 16:22:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 16:22:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 16:22:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 16:22:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 16:22:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 16:22:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 16:22:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 16:22:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 16:22:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 16:22:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 16:22:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 16:22:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 16:22:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 16:22:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 16:22:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 16:22:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 16:22:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 16:22:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 16:22:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 16:22:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 16:22:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 16:22:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 16:22:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:22:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 16:22:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:22:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 16:22:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:22:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 16:22:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:22:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 16:22:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:22:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 16:22:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:22:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 16:22:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:22:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 16:22:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:22:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:22:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 16:22:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 16:22:45 user cinder-volume[84696]: DEBUG cinder.utils [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.835(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:22:45 user cinder-volume[84696]: DEBUG cinder.utils [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:22:45 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 16:22:45 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (679460c5-9bf5-4df2-99d1-0df6c3fa1060) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 16:22:45 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 16:22:45 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:18:08Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-volume-769265242',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=f28ccf81-6553-4a98-83d7-400266578051,host='user@joviandss-2#joviandss-2',id=7f7b3b2e-e41d-466a-a3c2-e64e3d63b949,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='93c135fbb6f14928a9167adb3724faa3',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:20:25Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:20:25Z,use_quota=True,user_id='90edc8ffb805460794445bc433aa473f',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(be5f492b-b41f-4ad5-bffd-42eaafb454b4),volume_type_id=be5f492b-b41f-4ad5-bffd-42eaafb454b4)}, 'provides': None} Apr 17 16:22:45 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:18:08Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-volume-769265242',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=f28ccf81-6553-4a98-83d7-400266578051,host='user@joviandss-2#joviandss-2',id=7f7b3b2e-e41d-466a-a3c2-e64e3d63b949,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='93c135fbb6f14928a9167adb3724faa3',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:20:25Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:20:25Z,use_quota=True,user_id='90edc8ffb805460794445bc433aa473f',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(be5f492b-b41f-4ad5-bffd-42eaafb454b4),volume_type_id=be5f492b-b41f-4ad5-bffd-42eaafb454b4), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=f28ccf81-6553-4a98-83d7-400266578051,image_id=None,operation=,resource_backend='user@joviandss-2',snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(7f7b3b2e-e41d-466a-a3c2-e64e3d63b949),volume_id=7f7b3b2e-e41d-466a-a3c2-e64e3d63b949,volume_properties=VolumeProperties,volume_type=VolumeType(be5f492b-b41f-4ad5-bffd-42eaafb454b4))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': '7f7b3b2e-e41d-466a-a3c2-e64e3d63b949', 'volume_name': 'volume-7f7b3b2e-e41d-466a-a3c2-e64e3d63b949', 'volume_size': 1}} Apr 17 16:22:45 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'request_spec': {'consistencygroup_id': None, 'group_id': 'f28ccf81-6553-4a98-83d7-400266578051', 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': '7f7b3b2e-e41d-466a-a3c2-e64e3d63b949', 'volume': {'id': '7f7b3b2e-e41d-466a-a3c2-e64e3d63b949', '_name_id': None, 'ec2_id': None, 'user_id': '90edc8ffb805460794445bc433aa473f', 'project_id': '93c135fbb6f14928a9167adb3724faa3', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-volume-769265242', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': 'be5f492b-b41f-4ad5-bffd-42eaafb454b4', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': 'f28ccf81-6553-4a98-83d7-400266578051', 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'admin_metadata': {}, 'glance_metadata': {}, 'volume_type': {'id': 'be5f492b-b41f-4ad5-bffd-42eaafb454b4', 'name': 'tempest-GroupsV314Test-volume-type-1100522366', 'description': None, 'is_public': True, 'extra_specs': {}, 'qos_specs_id': None, 'created_at': '2023-04-17T16:18:06.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T16:18:07.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-7f7b3b2e-e41d-466a-a3c2-e64e3d63b949', 'name_id': '7f7b3b2e-e41d-466a-a3c2-e64e3d63b949', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': 'be5f492b-b41f-4ad5-bffd-42eaafb454b4', 'name': 'tempest-GroupsV314Test-volume-type-1100522366', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {}, 'qos_specs_id': None, 'created_at': '2023-04-17T16:18:06.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': 'f28ccf81-6553-4a98-83d7-400266578051', 'display_description': None, 'display_name': 'tempest-volume-769265242', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '93c135fbb6f14928a9167adb3724faa3', 'qos_specs': None, 'reservations': ['16413670-7ac2-4f17-a10d-ae77a197fc22', '05fb658c-9b25-41f9-b551-6c9fc7ca8983', 'bdd8bc3b-5d24-4a01-90e5-fca884142434', '6c4f66ea-ee22-430e-89df-e1bdd93b9f58'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': '90edc8ffb805460794445bc433aa473f', 'volume_type_id': 'be5f492b-b41f-4ad5-bffd-42eaafb454b4'}, 'resource_backend': 'user@joviandss-2', 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': 'f28ccf81-6553-4a98-83d7-400266578051', 'display_description': None, 'display_name': 'tempest-volume-769265242', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '93c135fbb6f14928a9167adb3724faa3', 'qos_specs': None, 'reservations': ['16413670-7ac2-4f17-a10d-ae77a197fc22', '05fb658c-9b25-41f9-b551-6c9fc7ca8983', 'bdd8bc3b-5d24-4a01-90e5-fca884142434', '6c4f66ea-ee22-430e-89df-e1bdd93b9f58'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': '90edc8ffb805460794445bc433aa473f', 'volume_type_id': 'be5f492b-b41f-4ad5-bffd-42eaafb454b4'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T16:18:06Z,deleted=False,deleted_at=None,description=None,extra_specs={},id=be5f492b-b41f-4ad5-bffd-42eaafb454b4,is_public=True,name='tempest-GroupsV314Test-volume-type-1100522366',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T16:18:06Z,deleted=False,deleted_at=None,description=None,extra_specs={},id=be5f492b-b41f-4ad5-bffd-42eaafb454b4,is_public=True,name='tempest-GroupsV314Test-volume-type-1100522366',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': '90edc8ffb805460794445bc433aa473f', 'metadata': {}, 'qos_specs': None, 'retry': {'num_attempts': 2, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']}}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=f28ccf81-6553-4a98-83d7-400266578051,image_id=None,operation=,resource_backend='user@joviandss-2',snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(7f7b3b2e-e41d-466a-a3c2-e64e3d63b949),volume_id=7f7b3b2e-e41d-466a-a3c2-e64e3d63b949,volume_properties=VolumeProperties,volume_type=VolumeType(be5f492b-b41f-4ad5-bffd-42eaafb454b4)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:18:08Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-volume-769265242',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=f28ccf81-6553-4a98-83d7-400266578051,host='user@joviandss-2#joviandss-2',id=7f7b3b2e-e41d-466a-a3c2-e64e3d63b949,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='93c135fbb6f14928a9167adb3724faa3',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:20:25Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:20:25Z,use_quota=True,user_id='90edc8ffb805460794445bc433aa473f',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(be5f492b-b41f-4ad5-bffd-42eaafb454b4),volume_type_id=be5f492b-b41f-4ad5-bffd-42eaafb454b4), 'context': }, 'provides': None} Apr 17 16:22:45 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:18:08Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-volume-769265242',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=f28ccf81-6553-4a98-83d7-400266578051,host='user@joviandss-2#joviandss-2',id=7f7b3b2e-e41d-466a-a3c2-e64e3d63b949,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='93c135fbb6f14928a9167adb3724faa3',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:20:25Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:20:25Z,use_quota=True,user_id='90edc8ffb805460794445bc433aa473f',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(be5f492b-b41f-4ad5-bffd-42eaafb454b4),volume_type_id=be5f492b-b41f-4ad5-bffd-42eaafb454b4)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:18:08Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-volume-769265242',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=f28ccf81-6553-4a98-83d7-400266578051,host='user@joviandss-2#joviandss-2',id=7f7b3b2e-e41d-466a-a3c2-e64e3d63b949,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='93c135fbb6f14928a9167adb3724faa3',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:20:25Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:20:25Z,use_quota=True,user_id='90edc8ffb805460794445bc433aa473f',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(be5f492b-b41f-4ad5-bffd-42eaafb454b4),volume_type_id=be5f492b-b41f-4ad5-bffd-42eaafb454b4)} Apr 17 16:22:45 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 16:22:45 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 16:22:45 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:22:45 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 16:22:45 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:22:45 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 16:22:45 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:22:45 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 16:22:45 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:22:45 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 16:22:45 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:22:45 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 16:22:45 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:22:45 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 16:22:45 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:22:45 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 16:22:45 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:22:45 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:22:45 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 16:22:45 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 16:22:45 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (679460c5-9bf5-4df2-99d1-0df6c3fa1060) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:22:45 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (679460c5-9bf5-4df2-99d1-0df6c3fa1060) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:22:45 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (44898e4d-1d99-453e-9276-3228f547ddf8) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:22:45 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (44898e4d-1d99-453e-9276-3228f547ddf8) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:22:45 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (518b98fa-7d20-487d-8ea7-bdf1f37f4c72) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:22:45 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (518b98fa-7d20-487d-8ea7-bdf1f37f4c72) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:22:45 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (ed1e4b67-153e-4602-81cf-2a65404dafbc) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:22:45 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Updating volume 7f7b3b2e-e41d-466a-a3c2-e64e3d63b949 with {'scheduled_at': datetime.datetime(2023, 4, 17, 16, 22, 45, 834852), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 16:22:45 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Volume 7f7b3b2e-e41d-466a-a3c2-e64e3d63b949: re-scheduling SchedulerAPI.create_volume attempt 2 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 16:22:45 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Volume 7f7b3b2e-e41d-466a-a3c2-e64e3d63b949: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 16:22:45 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] deleating volume v_7f7b3b2e-e41d-466a-a3c2-e64e3d63b949 {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 16:22:45 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] get snapshots for volume v_7f7b3b2e-e41d-466a-a3c2-e64e3d63b949 {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 16:22:45 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_7f7b3b2e-e41d-466a-a3c2-e64e3d63b949/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:22:45 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_7f7b3b2e-e41d-466a-a3c2-e64e3d63b949/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:22:45 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Volume reschedule parameters: True retry: {'num_attempts': 3, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 16:22:45 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Flow 'volume_create_manager' (5ada42b5-b063-40de-922e-364cee3d0293) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 16:22:45 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (c36dbe83-2e1c-4e7f-8e68-752878e3b897) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:22:45 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (c36dbe83-2e1c-4e7f-8e68-752878e3b897) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:18:08Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-volume-769265242',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=f28ccf81-6553-4a98-83d7-400266578051,host='user@joviandss-2#joviandss-2',id=7f7b3b2e-e41d-466a-a3c2-e64e3d63b949,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='93c135fbb6f14928a9167adb3724faa3',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:22:46Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='deleting',terminated_at=None,updated_at=2023-04-17T16:22:46Z,use_quota=True,user_id='90edc8ffb805460794445bc433aa473f',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(be5f492b-b41f-4ad5-bffd-42eaafb454b4),volume_type_id=be5f492b-b41f-4ad5-bffd-42eaafb454b4)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:22:45 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (c9f579d5-df6c-4c85-8f28-d3a84908979d) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:22:45 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (c9f579d5-df6c-4c85-8f28-d3a84908979d) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:22:45 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (de1fb827-2c8e-4b64-b262-95dcd9e034b6) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:22:46 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (de1fb827-2c8e-4b64-b262-95dcd9e034b6) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'deleting', 'type': 'raw', 'volume_id': '7f7b3b2e-e41d-466a-a3c2-e64e3d63b949', 'volume_name': 'volume-7f7b3b2e-e41d-466a-a3c2-e64e3d63b949', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:22:46 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (156c1695-ab20-4c94-98ad-035e893f892f) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:22:46 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (156c1695-ab20-4c94-98ad-035e893f892f) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:22:46 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (0088e165-b524-4b7c-b42d-8853f71b8293) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:22:46 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Volume 7f7b3b2e-e41d-466a-a3c2-e64e3d63b949: being created as raw with specification: {'status': 'deleting', 'volume_name': 'volume-7f7b3b2e-e41d-466a-a3c2-e64e3d63b949', 'volume_size': 1} Apr 17 16:22:46 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] creating volume v_7f7b3b2e-e41d-466a-a3c2-e64e3d63b949. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 16:22:46 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] create volume {'name': 'v_7f7b3b2e-e41d-466a-a3c2-e64e3d63b949', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 16:22:46 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:22:46 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:22:48 user cinder-volume[84696]: DEBUG cinder.utils [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.968(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:22:48 user cinder-volume[84696]: DEBUG cinder.utils [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_7f7b3b2e-e41d-466a-a3c2-e64e3d63b949/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:22:48 user cinder-volume[84696]: DEBUG cinder.utils [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 37.302(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:22:48 user cinder-volume[84696]: DEBUG cinder.utils [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:22:48 user cinder-volume[84696]: DEBUG cinder.utils [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.815(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:22:48 user cinder-volume[84696]: DEBUG cinder.utils [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:22:51 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.927(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:22:51 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:22:51 user cinder-volume[84696]: DEBUG cinder.utils [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.887(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:22:51 user cinder-volume[84696]: DEBUG cinder.utils [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:22:51 user cinder-volume[84696]: DEBUG cinder.utils [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.044(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:22:51 user cinder-volume[84696]: DEBUG cinder.utils [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_7f7b3b2e-e41d-466a-a3c2-e64e3d63b949/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:22:51 user cinder-volume[84696]: DEBUG cinder.utils [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 11.959(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:22:51 user cinder-volume[84696]: DEBUG cinder.utils [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:22:51 user cinder-volume[84696]: DEBUG cinder.utils [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.182(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:22:51 user cinder-volume[84696]: DEBUG cinder.utils [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_c9fa0a17-69d4-40d0-9d65-af6ae349cfd8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:22:54 user cinder-volume[84696]: DEBUG cinder.utils [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.370(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:22:54 user cinder-volume[84696]: DEBUG cinder.utils [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_4725fbeb-e869-48db-a487-33998b955f83/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:22:58 user cinder-volume[84696]: DEBUG cinder.utils [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.029(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:22:58 user cinder-volume[84696]: DEBUG cinder.utils [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:22:58 user cinder-volume[84696]: DEBUG cinder.utils [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.185(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:22:58 user cinder-volume[84696]: DEBUG cinder.utils [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_7f7b3b2e-e41d-466a-a3c2-e64e3d63b949/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:23:01 user cinder-volume[84696]: DEBUG cinder.utils [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 75.412(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:23:01 user cinder-volume[84696]: DEBUG cinder.utils [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_ae1eb1a7-8e56-418f-9032-a464645060fe/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:23:01 user cinder-volume[84696]: DEBUG cinder.utils [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.171(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:23:01 user cinder-volume[84696]: DEBUG cinder.utils [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:23:01 user cinder-volume[84696]: DEBUG cinder.utils [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.393(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:23:01 user cinder-volume[84696]: DEBUG cinder.utils [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_c9fa0a17-69d4-40d0-9d65-af6ae349cfd8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:23:04 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=17,cinder:UPDATE=7,cinder:INSERT=5 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:23:04 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:INSERT=1,cinder:UPDATE=5,cinder:SELECT=19 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:23:07 user cinder-volume[84696]: DEBUG cinder.utils [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 22.008(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:23:07 user cinder-volume[84696]: DEBUG cinder.utils [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_7f7b3b2e-e41d-466a-a3c2-e64e3d63b949/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:23:07 user cinder-volume[84696]: DEBUG cinder.utils [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.855(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:23:07 user cinder-volume[84696]: DEBUG cinder.utils [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:23:14 user cinder-volume[84696]: DEBUG cinder.utils [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 40.443(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:23:14 user cinder-volume[84696]: DEBUG cinder.utils [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_4725fbeb-e869-48db-a487-33998b955f83/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:23:14 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=1,cinder:UPDATE=1 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:23:20 user cinder-volume[84696]: DEBUG cinder.utils [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 40.463(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:23:20 user cinder-volume[84696]: DEBUG cinder.utils [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_c9fa0a17-69d4-40d0-9d65-af6ae349cfd8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:23:20 user cinder-volume[84696]: DEBUG cinder.utils [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 40.243(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:23:20 user cinder-volume[84696]: DEBUG cinder.utils [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:23:23 user cinder-volume[84696]: DEBUG cinder.utils [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 71.732(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:23:23 user cinder-volume[84696]: DEBUG cinder.utils [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:23:26 user cinder-volume[84696]: DEBUG cinder.utils [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 40.472(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:23:26 user cinder-volume[84696]: DEBUG cinder.utils [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_7f7b3b2e-e41d-466a-a3c2-e64e3d63b949/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:23:26 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 74.361(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:23:26 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:23:26 user cinder-volume[84696]: DEBUG cinder.utils [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 40.321(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:23:26 user cinder-volume[84696]: DEBUG cinder.utils [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:23:26 user cinder-volume[84696]: DEBUG cinder.utils [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 137.980(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:23:26 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:23:26 user cinder-volume[84696]: INFO cinder.message.api [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Creating message record for request_id = req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf Apr 17 16:23:26 user cinder-volume[84696]: DEBUG cinder.utils [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 138.162(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:23:26 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_b900713d-b991-43a5-b403-483dfded7e11/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:23:26 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (b7ab0fe7-a3c6-4101-9e8e-1d3ad5b552da) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 16:23:26 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (05bc28ea-f300-483a-bd08-a7ff9c91d60f) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:23:26 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (05bc28ea-f300-483a-bd08-a7ff9c91d60f) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:23:26 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Flow 'volume_create_manager' (7ab40436-e8df-4d1d-8861-91491ac5dbc2) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 16:23:26 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 16:23:26 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 16:23:26 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 16:23:26 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 16:23:26 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 16:23:26 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 16:23:26 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 16:23:26 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 16:23:26 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 16:23:26 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 16:23:26 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 16:23:26 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 16:23:26 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 16:23:26 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 16:23:26 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 16:23:26 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 16:23:26 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 16:23:26 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 16:23:26 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 16:23:26 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 16:23:26 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 16:23:26 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 16:23:26 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 16:23:26 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 16:23:26 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:23:26 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 16:23:26 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:23:26 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 16:23:26 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:23:26 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 16:23:26 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:23:26 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 16:23:26 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:23:26 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 16:23:26 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:23:26 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 16:23:26 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:23:26 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 16:23:26 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:23:26 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:23:26 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 16:23:26 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 16:23:26 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 16:23:26 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (0049b7bf-3755-4315-9906-42130ce19b91) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 16:23:26 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 16:23:26 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:18:51Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesTransfersV357Test-Volume-1424534055',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=b900713d-b991-43a5-b403-483dfded7e11,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='1db59dbe212347b8a44e27ebed9e22fb',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:21:08Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:21:08Z,use_quota=True,user_id='ac9545120d3a48bbb11e4cd5e706b53b',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 16:23:26 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:18:51Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesTransfersV357Test-Volume-1424534055',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=b900713d-b991-43a5-b403-483dfded7e11,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='1db59dbe212347b8a44e27ebed9e22fb',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:21:08Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:21:08Z,use_quota=True,user_id='ac9545120d3a48bbb11e4cd5e706b53b',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(b900713d-b991-43a5-b403-483dfded7e11),volume_id=b900713d-b991-43a5-b403-483dfded7e11,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': 'b900713d-b991-43a5-b403-483dfded7e11', 'volume_name': 'volume-b900713d-b991-43a5-b403-483dfded7e11', 'volume_size': 1}} Apr 17 16:23:26 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': 'b900713d-b991-43a5-b403-483dfded7e11', 'volume': {'id': 'b900713d-b991-43a5-b403-483dfded7e11', '_name_id': None, 'ec2_id': None, 'user_id': 'ac9545120d3a48bbb11e4cd5e706b53b', 'project_id': '1db59dbe212347b8a44e27ebed9e22fb', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-VolumesTransfersV357Test-Volume-1424534055', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'admin_metadata': {}, 'glance_metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T16:18:51.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-b900713d-b991-43a5-b403-483dfded7e11', 'name_id': 'b900713d-b991-43a5-b403-483dfded7e11', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumesTransfersV357Test-Volume-1424534055', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '1db59dbe212347b8a44e27ebed9e22fb', 'qos_specs': None, 'reservations': ['b40cb061-d9e6-4c02-bdc0-5d0cb5271144', '8ba69a37-4e7c-4eef-b721-634a940f66f4', 'a799c29a-0d31-4b62-9e2b-b056b451bb3c', 'e510dee0-051c-4a24-a14a-7acbc9a22867'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'ac9545120d3a48bbb11e4cd5e706b53b', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumesTransfersV357Test-Volume-1424534055', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '1db59dbe212347b8a44e27ebed9e22fb', 'qos_specs': None, 'reservations': ['b40cb061-d9e6-4c02-bdc0-5d0cb5271144', '8ba69a37-4e7c-4eef-b721-634a940f66f4', 'a799c29a-0d31-4b62-9e2b-b056b451bb3c', 'e510dee0-051c-4a24-a14a-7acbc9a22867'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'ac9545120d3a48bbb11e4cd5e706b53b', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': 'ac9545120d3a48bbb11e4cd5e706b53b', 'metadata': {}, 'qos_specs': None, 'retry': {'num_attempts': 2, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']}}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(b900713d-b991-43a5-b403-483dfded7e11),volume_id=b900713d-b991-43a5-b403-483dfded7e11,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:18:51Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesTransfersV357Test-Volume-1424534055',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=b900713d-b991-43a5-b403-483dfded7e11,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='1db59dbe212347b8a44e27ebed9e22fb',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:21:08Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:21:08Z,use_quota=True,user_id='ac9545120d3a48bbb11e4cd5e706b53b',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 16:23:26 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:18:51Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesTransfersV357Test-Volume-1424534055',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=b900713d-b991-43a5-b403-483dfded7e11,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='1db59dbe212347b8a44e27ebed9e22fb',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:21:08Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:21:08Z,use_quota=True,user_id='ac9545120d3a48bbb11e4cd5e706b53b',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:18:51Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesTransfersV357Test-Volume-1424534055',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=b900713d-b991-43a5-b403-483dfded7e11,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='1db59dbe212347b8a44e27ebed9e22fb',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:21:08Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:21:08Z,use_quota=True,user_id='ac9545120d3a48bbb11e4cd5e706b53b',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 16:23:26 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 16:23:26 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 16:23:26 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:23:26 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 16:23:26 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:23:26 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 16:23:26 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:23:26 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 16:23:26 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:23:26 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 16:23:26 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:23:26 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 16:23:26 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:23:26 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 16:23:26 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:23:26 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 16:23:26 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:23:26 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:23:26 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 16:23:26 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 16:23:26 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (0049b7bf-3755-4315-9906-42130ce19b91) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:23:26 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (0049b7bf-3755-4315-9906-42130ce19b91) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:23:26 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (45ea267e-7e55-4f48-9d6c-7a692e92f91f) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:23:26 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (45ea267e-7e55-4f48-9d6c-7a692e92f91f) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:23:26 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (cb00741d-99d9-404e-b7a5-683616a3f865) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:23:26 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (cb00741d-99d9-404e-b7a5-683616a3f865) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:23:26 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (ac6e72c4-80c1-44d3-8c37-428687e574ca) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:23:26 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Updating volume b900713d-b991-43a5-b403-483dfded7e11 with {'scheduled_at': datetime.datetime(2023, 4, 17, 16, 23, 26, 416043), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 16:23:26 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Volume b900713d-b991-43a5-b403-483dfded7e11: re-scheduling SchedulerAPI.create_volume attempt 2 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 16:23:26 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Volume b900713d-b991-43a5-b403-483dfded7e11: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 16:23:26 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] deleating volume v_b900713d-b991-43a5-b403-483dfded7e11 {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 16:23:26 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] get snapshots for volume v_b900713d-b991-43a5-b403-483dfded7e11 {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 16:23:26 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_b900713d-b991-43a5-b403-483dfded7e11/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:23:26 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_b900713d-b991-43a5-b403-483dfded7e11/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:23:26 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Volume reschedule parameters: True retry: {'num_attempts': 3, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 16:23:26 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Flow 'volume_create_manager' (c2749752-3942-4459-a666-55ad469f1e72) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 16:23:26 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (0ffb3fa7-e336-4c13-a8cb-decd6d3449a8) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:23:26 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (0ffb3fa7-e336-4c13-a8cb-decd6d3449a8) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:18:51Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesTransfersV357Test-Volume-1424534055',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=b900713d-b991-43a5-b403-483dfded7e11,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='1db59dbe212347b8a44e27ebed9e22fb',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:23:27Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:23:27Z,use_quota=True,user_id='ac9545120d3a48bbb11e4cd5e706b53b',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:23:26 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (9ca078d3-8942-4a79-9ad3-de0bd37971ce) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:23:26 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (9ca078d3-8942-4a79-9ad3-de0bd37971ce) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:23:26 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (db8eb656-182c-49f5-ba09-31b899fd5ada) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:23:26 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (db8eb656-182c-49f5-ba09-31b899fd5ada) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': 'b900713d-b991-43a5-b403-483dfded7e11', 'volume_name': 'volume-b900713d-b991-43a5-b403-483dfded7e11', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:23:26 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (c84d59c4-9a9e-4a00-bc27-e083ef01af0d) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:23:26 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (c84d59c4-9a9e-4a00-bc27-e083ef01af0d) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:23:26 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (3090157d-b623-4033-8887-eebeeebd838e) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:23:26 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Volume b900713d-b991-43a5-b403-483dfded7e11: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-b900713d-b991-43a5-b403-483dfded7e11', 'volume_size': 1} Apr 17 16:23:26 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] creating volume v_b900713d-b991-43a5-b403-483dfded7e11. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 16:23:26 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] create volume {'name': 'v_b900713d-b991-43a5-b403-483dfded7e11', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 16:23:26 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:23:26 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:23:29 user cinder-volume[84696]: DEBUG cinder.utils [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.959(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:23:29 user cinder-volume[84696]: DEBUG cinder.utils [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_b900713d-b991-43a5-b403-483dfded7e11/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:23:29 user cinder-volume[84696]: DEBUG cinder.utils [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.743(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:23:29 user cinder-volume[84696]: DEBUG cinder.utils [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:23:32 user cinder-volume[84696]: DEBUG cinder.utils [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.814(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:23:32 user cinder-volume[84696]: DEBUG cinder.utils [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:23:32 user cinder-volume[84696]: DEBUG cinder.utils [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.033(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:23:32 user cinder-volume[84696]: DEBUG cinder.utils [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_b900713d-b991-43a5-b403-483dfded7e11/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:23:38 user cinder-volume[84696]: DEBUG cinder.utils [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.463(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:23:38 user cinder-volume[84696]: DEBUG cinder.utils [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_b900713d-b991-43a5-b403-483dfded7e11/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:23:38 user cinder-volume[84696]: DEBUG cinder.utils [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.248(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:23:38 user cinder-volume[84696]: DEBUG cinder.utils [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:23:41 user cinder-volume[84696]: DEBUG cinder.utils [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 137.073(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:23:41 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:23:42 user cinder-volume[84696]: INFO cinder.message.api [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Creating message record for request_id = req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 Apr 17 16:23:42 user cinder-volume[84696]: DEBUG cinder.utils [None req-684f3c97-1d86-45f4-a649-1a4efd9f0051 tempest-GroupsV314Test-467284027 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 136.916(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:23:42 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-684f3c97-1d86-45f4-a649-1a4efd9f0051 tempest-GroupsV314Test-467284027 None] Geting None from DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:7f7b3b2e-e41d-466a-a3c2-e64e3d63b949/luns/v_7f7b3b2e-e41d-466a-a3c2-e64e3d63b949 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:23:42 user cinder-volume[84696]: DEBUG cinder.utils [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 137.895(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:23:42 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:23:42 user cinder-volume[84696]: INFO cinder.message.api [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Creating message record for request_id = req-d3245848-3d3a-4130-8413-3f811df2f47d Apr 17 16:23:42 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-684f3c97-1d86-45f4-a649-1a4efd9f0051 tempest-GroupsV314Test-467284027 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 16:23:42 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 16:23:42 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=10,cinder:UPDATE=4,cinder:INSERT=1 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:23:42 user cinder-volume[84696]: ERROR cinder.volume.manager [None req-684f3c97-1d86-45f4-a649-1a4efd9f0051 tempest-GroupsV314Test-467284027 None] Delete group failed. Apr 17 16:23:42 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (8b9bca99-a013-4bf7-86d0-049714fd79dc) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 16:23:42 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 16:23:42 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:21:25Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-volume-1452364054',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=fe50cd00-dcd4-4b21-a4e9-6f226f8c6ac1,host='user@joviandss-2#joviandss-2',id=02aba338-fa31-4e46-8fcf-7f86191e44d1,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='e10e34b8774b42d8a4cb1f78d72b411d',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:21:25Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:21:25Z,use_quota=True,user_id='e7392ed432ad41279967273f16b4a4c8',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(400bec8d-bd01-4057-8fb8-49a786cd39f1),volume_type_id=400bec8d-bd01-4057-8fb8-49a786cd39f1)}, 'provides': None} Apr 17 16:23:42 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:21:25Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-volume-1452364054',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=fe50cd00-dcd4-4b21-a4e9-6f226f8c6ac1,host='user@joviandss-2#joviandss-2',id=02aba338-fa31-4e46-8fcf-7f86191e44d1,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='e10e34b8774b42d8a4cb1f78d72b411d',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:21:25Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:21:25Z,use_quota=True,user_id='e7392ed432ad41279967273f16b4a4c8',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(400bec8d-bd01-4057-8fb8-49a786cd39f1),volume_type_id=400bec8d-bd01-4057-8fb8-49a786cd39f1), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=fe50cd00-dcd4-4b21-a4e9-6f226f8c6ac1,image_id=None,operation=,resource_backend='user@joviandss-2',snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(02aba338-fa31-4e46-8fcf-7f86191e44d1),volume_id=02aba338-fa31-4e46-8fcf-7f86191e44d1,volume_properties=VolumeProperties,volume_type=VolumeType(400bec8d-bd01-4057-8fb8-49a786cd39f1))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': '02aba338-fa31-4e46-8fcf-7f86191e44d1', 'volume_name': 'volume-02aba338-fa31-4e46-8fcf-7f86191e44d1', 'volume_size': 1}} Apr 17 16:23:42 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'retry': {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']}, 'request_spec': {'consistencygroup_id': None, 'group_id': 'fe50cd00-dcd4-4b21-a4e9-6f226f8c6ac1', 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': '02aba338-fa31-4e46-8fcf-7f86191e44d1', 'volume': {'id': '02aba338-fa31-4e46-8fcf-7f86191e44d1', '_name_id': None, 'ec2_id': None, 'user_id': 'e7392ed432ad41279967273f16b4a4c8', 'project_id': 'e10e34b8774b42d8a4cb1f78d72b411d', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-volume-1452364054', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '400bec8d-bd01-4057-8fb8-49a786cd39f1', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': 'fe50cd00-dcd4-4b21-a4e9-6f226f8c6ac1', 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'volume_type': {'id': '400bec8d-bd01-4057-8fb8-49a786cd39f1', 'name': 'tempest-GroupsTest-volume-type-1895690014', 'description': None, 'is_public': True, 'extra_specs': {}, 'qos_specs_id': None, 'created_at': '2023-04-17T16:21:23.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T16:21:24.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-02aba338-fa31-4e46-8fcf-7f86191e44d1', 'name_id': '02aba338-fa31-4e46-8fcf-7f86191e44d1', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '400bec8d-bd01-4057-8fb8-49a786cd39f1', 'name': 'tempest-GroupsTest-volume-type-1895690014', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {}, 'qos_specs_id': None, 'created_at': '2023-04-17T16:21:23.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': 'fe50cd00-dcd4-4b21-a4e9-6f226f8c6ac1', 'display_description': None, 'display_name': 'tempest-volume-1452364054', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': 'e10e34b8774b42d8a4cb1f78d72b411d', 'qos_specs': None, 'reservations': ['f5b0b08e-834b-48d7-bb1a-7efd1002da9b', 'ac966cb4-51d5-4799-b0b6-b3453a218bd0', '77612ba8-fbd6-4ff5-97c8-7ddceb3a7c7a', '101ca44e-fd59-494f-9f9a-2df7868625ff'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'e7392ed432ad41279967273f16b4a4c8', 'volume_type_id': '400bec8d-bd01-4057-8fb8-49a786cd39f1'}, 'resource_backend': 'user@joviandss-2', 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': 'fe50cd00-dcd4-4b21-a4e9-6f226f8c6ac1', 'display_description': None, 'display_name': 'tempest-volume-1452364054', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': 'e10e34b8774b42d8a4cb1f78d72b411d', 'qos_specs': None, 'reservations': ['f5b0b08e-834b-48d7-bb1a-7efd1002da9b', 'ac966cb4-51d5-4799-b0b6-b3453a218bd0', '77612ba8-fbd6-4ff5-97c8-7ddceb3a7c7a', '101ca44e-fd59-494f-9f9a-2df7868625ff'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'e7392ed432ad41279967273f16b4a4c8', 'volume_type_id': '400bec8d-bd01-4057-8fb8-49a786cd39f1'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T16:21:23Z,deleted=False,deleted_at=None,description=None,extra_specs={},id=400bec8d-bd01-4057-8fb8-49a786cd39f1,is_public=True,name='tempest-GroupsTest-volume-type-1895690014',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T16:21:23Z,deleted=False,deleted_at=None,description=None,extra_specs={},id=400bec8d-bd01-4057-8fb8-49a786cd39f1,is_public=True,name='tempest-GroupsTest-volume-type-1895690014',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': 'e7392ed432ad41279967273f16b4a4c8', 'metadata': {}, 'qos_specs': None}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=fe50cd00-dcd4-4b21-a4e9-6f226f8c6ac1,image_id=None,operation=,resource_backend='user@joviandss-2',snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(02aba338-fa31-4e46-8fcf-7f86191e44d1),volume_id=02aba338-fa31-4e46-8fcf-7f86191e44d1,volume_properties=VolumeProperties,volume_type=VolumeType(400bec8d-bd01-4057-8fb8-49a786cd39f1)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:21:25Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-volume-1452364054',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=fe50cd00-dcd4-4b21-a4e9-6f226f8c6ac1,host='user@joviandss-2#joviandss-2',id=02aba338-fa31-4e46-8fcf-7f86191e44d1,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='e10e34b8774b42d8a4cb1f78d72b411d',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:21:25Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:21:25Z,use_quota=True,user_id='e7392ed432ad41279967273f16b4a4c8',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(400bec8d-bd01-4057-8fb8-49a786cd39f1),volume_type_id=400bec8d-bd01-4057-8fb8-49a786cd39f1), 'context': }, 'provides': None} Apr 17 16:23:42 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:21:25Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-volume-1452364054',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=fe50cd00-dcd4-4b21-a4e9-6f226f8c6ac1,host='user@joviandss-2#joviandss-2',id=02aba338-fa31-4e46-8fcf-7f86191e44d1,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='e10e34b8774b42d8a4cb1f78d72b411d',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:21:25Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:21:25Z,use_quota=True,user_id='e7392ed432ad41279967273f16b4a4c8',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(400bec8d-bd01-4057-8fb8-49a786cd39f1),volume_type_id=400bec8d-bd01-4057-8fb8-49a786cd39f1)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:21:25Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-volume-1452364054',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=fe50cd00-dcd4-4b21-a4e9-6f226f8c6ac1,host='user@joviandss-2#joviandss-2',id=02aba338-fa31-4e46-8fcf-7f86191e44d1,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='e10e34b8774b42d8a4cb1f78d72b411d',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:21:25Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:21:25Z,use_quota=True,user_id='e7392ed432ad41279967273f16b4a4c8',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(400bec8d-bd01-4057-8fb8-49a786cd39f1),volume_type_id=400bec8d-bd01-4057-8fb8-49a786cd39f1)} Apr 17 16:23:42 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 16:23:42 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 16:23:42 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:23:42 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 16:23:42 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:23:42 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 16:23:42 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:23:42 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 16:23:42 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:23:42 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 16:23:42 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:23:42 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 16:23:42 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:23:42 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 16:23:42 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:23:42 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 16:23:42 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:23:42 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:23:42 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 16:23:42 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 16:23:42 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (8b9bca99-a013-4bf7-86d0-049714fd79dc) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:23:42 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (8b9bca99-a013-4bf7-86d0-049714fd79dc) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:23:42 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (18ad6f8d-a5b5-42ca-b79f-9ef4a14cb402) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:23:42 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (18ad6f8d-a5b5-42ca-b79f-9ef4a14cb402) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:23:42 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (e8bb6e08-bb48-464c-9d05-1378354a8f6b) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:23:42 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (e8bb6e08-bb48-464c-9d05-1378354a8f6b) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:23:42 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (d38aba9e-fa93-4bd1-9c3a-5884f9099207) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:23:42 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Updating volume 02aba338-fa31-4e46-8fcf-7f86191e44d1 with {'scheduled_at': datetime.datetime(2023, 4, 17, 16, 23, 42, 77008), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 16:23:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-684f3c97-1d86-45f4-a649-1a4efd9f0051 tempest-GroupsV314Test-467284027 None] Exception during message handling: cinder.exception.VolumeDriverException: Volume driver reported an error: Delete group failed. Apr 17 16:23:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 16:23:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 16:23:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 16:23:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 16:23:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 16:23:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 16:23:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 16:23:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 3805, in delete_group Apr 17 16:23:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 16:23:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:23:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 16:23:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:23:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 16:23:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 3799, in delete_group Apr 17 16:23:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise exception.VolumeDriverException(message=msg) Apr 17 16:23:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server cinder.exception.VolumeDriverException: Volume driver reported an error: Delete group failed. Apr 17 16:23:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 16:23:42 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (4985f125-71bc-4dbb-ae8b-c135918369bd) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 16:23:42 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 16:23:42 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:21:24Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-AbsoluteLimitsTests-Volume-1339022265',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=94610127-c8b8-4df4-9e39-a1f41b5a9b98,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='2d881d9502184c74b36e1c50acad1f80',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:21:24Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:21:24Z,use_quota=True,user_id='e460a7fb86b4454788ec1950b15434a4',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 16:23:42 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:21:24Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-AbsoluteLimitsTests-Volume-1339022265',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=94610127-c8b8-4df4-9e39-a1f41b5a9b98,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='2d881d9502184c74b36e1c50acad1f80',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:21:24Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:21:24Z,use_quota=True,user_id='e460a7fb86b4454788ec1950b15434a4',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(94610127-c8b8-4df4-9e39-a1f41b5a9b98),volume_id=94610127-c8b8-4df4-9e39-a1f41b5a9b98,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': '94610127-c8b8-4df4-9e39-a1f41b5a9b98', 'volume_name': 'volume-94610127-c8b8-4df4-9e39-a1f41b5a9b98', 'volume_size': 1}} Apr 17 16:23:42 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'retry': {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']}, 'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': '94610127-c8b8-4df4-9e39-a1f41b5a9b98', 'volume': {'id': '94610127-c8b8-4df4-9e39-a1f41b5a9b98', '_name_id': None, 'ec2_id': None, 'user_id': 'e460a7fb86b4454788ec1950b15434a4', 'project_id': '2d881d9502184c74b36e1c50acad1f80', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-AbsoluteLimitsTests-Volume-1339022265', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T16:21:23.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-94610127-c8b8-4df4-9e39-a1f41b5a9b98', 'name_id': '94610127-c8b8-4df4-9e39-a1f41b5a9b98', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-AbsoluteLimitsTests-Volume-1339022265', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '2d881d9502184c74b36e1c50acad1f80', 'qos_specs': None, 'reservations': ['7f441e01-433c-4826-9a44-ec0d9a586e2a', '62cb3d03-7543-4f33-9fda-df3d7d67cb94', 'a069f7b2-c2bc-47d4-85ba-479a14bdc4a4', 'e95f992f-6c15-431a-9f34-693f405be424'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'e460a7fb86b4454788ec1950b15434a4', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-AbsoluteLimitsTests-Volume-1339022265', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '2d881d9502184c74b36e1c50acad1f80', 'qos_specs': None, 'reservations': ['7f441e01-433c-4826-9a44-ec0d9a586e2a', '62cb3d03-7543-4f33-9fda-df3d7d67cb94', 'a069f7b2-c2bc-47d4-85ba-479a14bdc4a4', 'e95f992f-6c15-431a-9f34-693f405be424'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'e460a7fb86b4454788ec1950b15434a4', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': 'e460a7fb86b4454788ec1950b15434a4', 'metadata': {}, 'qos_specs': None}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(94610127-c8b8-4df4-9e39-a1f41b5a9b98),volume_id=94610127-c8b8-4df4-9e39-a1f41b5a9b98,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:21:24Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-AbsoluteLimitsTests-Volume-1339022265',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=94610127-c8b8-4df4-9e39-a1f41b5a9b98,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='2d881d9502184c74b36e1c50acad1f80',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:21:24Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:21:24Z,use_quota=True,user_id='e460a7fb86b4454788ec1950b15434a4',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 16:23:42 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:21:24Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-AbsoluteLimitsTests-Volume-1339022265',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=94610127-c8b8-4df4-9e39-a1f41b5a9b98,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='2d881d9502184c74b36e1c50acad1f80',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:21:24Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:21:24Z,use_quota=True,user_id='e460a7fb86b4454788ec1950b15434a4',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:21:24Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-AbsoluteLimitsTests-Volume-1339022265',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=94610127-c8b8-4df4-9e39-a1f41b5a9b98,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='2d881d9502184c74b36e1c50acad1f80',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:21:24Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:21:24Z,use_quota=True,user_id='e460a7fb86b4454788ec1950b15434a4',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 16:23:42 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 16:23:42 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 16:23:42 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:23:42 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 16:23:42 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:23:42 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 16:23:42 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:23:42 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 16:23:42 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:23:42 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 16:23:42 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:23:42 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 16:23:42 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:23:42 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 16:23:42 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:23:42 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 16:23:42 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:23:42 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:23:42 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 16:23:42 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 16:23:42 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (4985f125-71bc-4dbb-ae8b-c135918369bd) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:23:42 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (4985f125-71bc-4dbb-ae8b-c135918369bd) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:23:42 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (59d3448b-fe78-4b00-89bb-bc7abcca6374) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:23:42 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (59d3448b-fe78-4b00-89bb-bc7abcca6374) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:23:42 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (45fc99b4-3004-4757-add8-c527531a59a7) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:23:42 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (45fc99b4-3004-4757-add8-c527531a59a7) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:23:42 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (b549076a-a06e-48df-a37e-acd89eb0ce7d) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:23:42 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Updating volume 94610127-c8b8-4df4-9e39-a1f41b5a9b98 with {'scheduled_at': datetime.datetime(2023, 4, 17, 16, 23, 42, 123449), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 16:23:42 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Volume 94610127-c8b8-4df4-9e39-a1f41b5a9b98: re-scheduling SchedulerAPI.create_volume attempt 1 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 16:23:42 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Volume 02aba338-fa31-4e46-8fcf-7f86191e44d1: re-scheduling SchedulerAPI.create_volume attempt 1 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 16:23:42 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Volume 94610127-c8b8-4df4-9e39-a1f41b5a9b98: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 16:23:42 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] deleating volume v_94610127-c8b8-4df4-9e39-a1f41b5a9b98 {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 16:23:42 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] get snapshots for volume v_94610127-c8b8-4df4-9e39-a1f41b5a9b98 {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 16:23:42 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_94610127-c8b8-4df4-9e39-a1f41b5a9b98/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:23:42 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_94610127-c8b8-4df4-9e39-a1f41b5a9b98/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:23:42 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Volume 02aba338-fa31-4e46-8fcf-7f86191e44d1: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 16:23:42 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] deleating volume v_02aba338-fa31-4e46-8fcf-7f86191e44d1 {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 16:23:42 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] get snapshots for volume v_02aba338-fa31-4e46-8fcf-7f86191e44d1 {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 16:23:42 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_02aba338-fa31-4e46-8fcf-7f86191e44d1/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:23:42 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_02aba338-fa31-4e46-8fcf-7f86191e44d1/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:23:42 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Volume reschedule parameters: True retry: {'num_attempts': 2, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 16:23:42 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Flow 'volume_create_manager' (08b3d2ad-b078-465b-82ae-7d05f7e28f48) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 16:23:42 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (7a282bc8-9447-4b11-be85-006463b73164) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:23:42 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (7a282bc8-9447-4b11-be85-006463b73164) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:21:24Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-AbsoluteLimitsTests-Volume-1339022265',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=94610127-c8b8-4df4-9e39-a1f41b5a9b98,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='2d881d9502184c74b36e1c50acad1f80',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:23:42Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:23:42Z,use_quota=True,user_id='e460a7fb86b4454788ec1950b15434a4',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:23:42 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (fd9eb00d-6c37-455d-a77f-5284073a5df9) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:23:42 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (fd9eb00d-6c37-455d-a77f-5284073a5df9) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:23:42 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (2680ff9a-2c2b-4a73-abe7-a4509e8d26b4) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:23:42 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (2680ff9a-2c2b-4a73-abe7-a4509e8d26b4) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': '94610127-c8b8-4df4-9e39-a1f41b5a9b98', 'volume_name': 'volume-94610127-c8b8-4df4-9e39-a1f41b5a9b98', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:23:42 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (3f9305bf-d9f1-4be2-b50e-79d427b57764) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:23:42 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (3f9305bf-d9f1-4be2-b50e-79d427b57764) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:23:42 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (9032af0e-850d-4cc9-a380-151fd0dbc25f) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:23:42 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Volume 94610127-c8b8-4df4-9e39-a1f41b5a9b98: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-94610127-c8b8-4df4-9e39-a1f41b5a9b98', 'volume_size': 1} Apr 17 16:23:42 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] creating volume v_94610127-c8b8-4df4-9e39-a1f41b5a9b98. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 16:23:42 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] create volume {'name': 'v_94610127-c8b8-4df4-9e39-a1f41b5a9b98', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 16:23:42 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:23:42 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:23:42 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Volume reschedule parameters: True retry: {'num_attempts': 2, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 16:23:42 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Flow 'volume_create_manager' (1d6096aa-378e-4f49-8198-aca639c7cbe1) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 16:23:42 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (57c8bc6a-ea3d-4181-951a-bfa0a30743e4) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:23:42 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (57c8bc6a-ea3d-4181-951a-bfa0a30743e4) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:21:25Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-volume-1452364054',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=fe50cd00-dcd4-4b21-a4e9-6f226f8c6ac1,host='user@joviandss-2#joviandss-2',id=02aba338-fa31-4e46-8fcf-7f86191e44d1,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='e10e34b8774b42d8a4cb1f78d72b411d',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:23:42Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:23:42Z,use_quota=True,user_id='e7392ed432ad41279967273f16b4a4c8',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(400bec8d-bd01-4057-8fb8-49a786cd39f1),volume_type_id=400bec8d-bd01-4057-8fb8-49a786cd39f1)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:23:42 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (176d0698-e62f-4f58-b9a1-b2c5a154dbdc) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:23:42 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (176d0698-e62f-4f58-b9a1-b2c5a154dbdc) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:23:42 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (3bee15e4-cb30-4668-b13f-3da57b102ca6) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:23:42 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (3bee15e4-cb30-4668-b13f-3da57b102ca6) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': '02aba338-fa31-4e46-8fcf-7f86191e44d1', 'volume_name': 'volume-02aba338-fa31-4e46-8fcf-7f86191e44d1', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:23:42 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (38284650-61ae-4b47-8247-5ffc6530f319) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:23:42 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (38284650-61ae-4b47-8247-5ffc6530f319) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:23:42 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (9c89fe55-1bda-427e-b8c2-de5da7f4c4cf) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:23:42 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Volume 02aba338-fa31-4e46-8fcf-7f86191e44d1: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-02aba338-fa31-4e46-8fcf-7f86191e44d1', 'volume_size': 1} Apr 17 16:23:42 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] creating volume v_02aba338-fa31-4e46-8fcf-7f86191e44d1. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 16:23:42 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] create volume {'name': 'v_02aba338-fa31-4e46-8fcf-7f86191e44d1', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 16:23:42 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:23:42 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:23:45 user cinder-volume[84696]: DEBUG cinder.utils [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.647(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:23:45 user cinder-volume[84696]: DEBUG cinder.utils [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:23:45 user cinder-volume[84696]: DEBUG cinder.utils [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.574(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:23:45 user cinder-volume[84696]: DEBUG cinder.utils [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:23:45 user cinder-volume[84696]: DEBUG cinder.utils [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.880(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:23:45 user cinder-volume[84696]: DEBUG cinder.utils [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_94610127-c8b8-4df4-9e39-a1f41b5a9b98/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:23:45 user cinder-volume[84696]: DEBUG cinder.utils [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.871(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:23:45 user cinder-volume[84696]: DEBUG cinder.utils [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_02aba338-fa31-4e46-8fcf-7f86191e44d1/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:23:48 user cinder-volume[84696]: DEBUG cinder.utils [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.644(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:23:48 user cinder-volume[84696]: DEBUG cinder.utils [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:23:48 user cinder-volume[84696]: DEBUG cinder.utils [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.463(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:23:48 user cinder-volume[84696]: DEBUG cinder.utils [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:23:48 user cinder-volume[84696]: DEBUG cinder.utils [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 74.526(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:23:48 user cinder-volume[84696]: DEBUG cinder.utils [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_4725fbeb-e869-48db-a487-33998b955f83/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:23:48 user cinder-volume[84696]: DEBUG cinder.utils [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.684(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:23:48 user cinder-volume[84696]: DEBUG cinder.utils [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_b900713d-b991-43a5-b403-483dfded7e11/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:23:48 user cinder-volume[84696]: DEBUG cinder.utils [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.726(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:23:48 user cinder-volume[84696]: DEBUG cinder.utils [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:23:48 user cinder-volume[84696]: DEBUG cinder.utils [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.957(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:23:48 user cinder-volume[84696]: DEBUG cinder.utils [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_94610127-c8b8-4df4-9e39-a1f41b5a9b98/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:23:48 user cinder-volume[84696]: DEBUG cinder.utils [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.949(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:23:48 user cinder-volume[84696]: DEBUG cinder.utils [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_02aba338-fa31-4e46-8fcf-7f86191e44d1/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:23:54 user cinder-volume[84696]: DEBUG cinder.utils [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 74.322(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:23:54 user cinder-volume[84696]: DEBUG cinder.utils [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:23:54 user cinder-volume[84696]: DEBUG cinder.utils [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 11.791(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:23:54 user cinder-volume[84696]: DEBUG cinder.utils [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:23:54 user cinder-volume[84696]: DEBUG cinder.utils [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 11.867(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:23:54 user cinder-volume[84696]: DEBUG cinder.utils [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:23:54 user cinder-volume[84696]: DEBUG cinder.utils [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.089(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:23:54 user cinder-volume[84696]: DEBUG cinder.utils [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_02aba338-fa31-4e46-8fcf-7f86191e44d1/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:23:54 user cinder-volume[84696]: DEBUG cinder.utils [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.099(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:23:54 user cinder-volume[84696]: DEBUG cinder.utils [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_94610127-c8b8-4df4-9e39-a1f41b5a9b98/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:23:54 user cinder-volume[84696]: DEBUG cinder.utils [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 74.551(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:23:54 user cinder-volume[84696]: DEBUG cinder.utils [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_c9fa0a17-69d4-40d0-9d65-af6ae349cfd8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:23:54 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=26,cinder:UPDATE=8,cinder:INSERT=4 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:24:00 user cinder-volume[84696]: DEBUG cinder.utils [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 74.397(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:24:00 user cinder-volume[84696]: DEBUG cinder.utils [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:24:00 user cinder-volume[84696]: DEBUG cinder.utils [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 74.553(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:24:00 user cinder-volume[84696]: DEBUG cinder.utils [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_7f7b3b2e-e41d-466a-a3c2-e64e3d63b949/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:24:03 user cinder-volume[84696]: DEBUG cinder.utils [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.004(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:24:03 user cinder-volume[84696]: DEBUG cinder.utils [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:24:03 user cinder-volume[84696]: DEBUG cinder.utils [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.310(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:24:03 user cinder-volume[84696]: DEBUG cinder.utils [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_94610127-c8b8-4df4-9e39-a1f41b5a9b98/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:24:03 user cinder-volume[84696]: DEBUG cinder.utils [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.302(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:24:03 user cinder-volume[84696]: DEBUG cinder.utils [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_02aba338-fa31-4e46-8fcf-7f86191e44d1/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:24:03 user cinder-volume[84696]: DEBUG cinder.utils [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.083(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:24:03 user cinder-volume[84696]: DEBUG cinder.utils [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:24:06 user cinder-volume[84696]: DEBUG cinder.utils [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.893(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:24:06 user cinder-volume[84696]: DEBUG cinder.utils [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:24:06 user cinder-volume[84696]: DEBUG cinder.utils [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 40.112(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:24:06 user cinder-volume[84696]: DEBUG cinder.utils [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_b900713d-b991-43a5-b403-483dfded7e11/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:24:06 user cinder-volume[84696]: DEBUG cinder.utils [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 140.855(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:24:06 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_ae1eb1a7-8e56-418f-9032-a464645060fe/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:24:06 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (a83887a1-4d90-481a-a257-ad22ea100a28) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 16:24:06 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (2af8e7ff-4433-40a6-929a-d743fbd72ce9) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:24:06 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (2af8e7ff-4433-40a6-929a-d743fbd72ce9) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:24:06 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Flow 'volume_create_manager' (ae657be4-a652-4d1d-b136-310bcdde538f) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 16:24:06 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-5a436bf3-8843-4b13-8b6b-92ee59a23185 tempest-GroupsTest-789764105 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 16:24:06 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 16:24:06 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 16:24:06 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 16:24:06 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 16:24:06 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 16:24:06 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 16:24:06 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 16:24:06 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 16:24:06 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 16:24:06 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 16:24:06 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 16:24:06 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 16:24:06 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 16:24:06 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 16:24:06 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 16:24:06 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 16:24:06 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 16:24:06 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 16:24:06 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 16:24:06 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 16:24:06 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 16:24:06 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 16:24:06 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 16:24:06 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:24:06 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 16:24:06 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:24:06 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 16:24:06 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:24:06 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 16:24:06 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:24:06 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 16:24:06 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:24:06 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 16:24:06 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:24:06 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 16:24:06 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:24:06 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 16:24:06 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:24:06 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:24:06 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 16:24:06 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 16:24:14 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=1 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:24:14 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:UPDATE=5,cinder:SELECT=12,cinder:INSERT=1 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:24:22 user cinder-volume[84696]: DEBUG cinder.utils [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 40.385(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:24:22 user cinder-volume[84696]: DEBUG cinder.utils [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_94610127-c8b8-4df4-9e39-a1f41b5a9b98/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:24:22 user cinder-volume[84696]: DEBUG cinder.utils [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 40.156(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:24:22 user cinder-volume[84696]: DEBUG cinder.utils [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:24:22 user cinder-volume[84696]: DEBUG cinder.utils [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 40.377(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:24:22 user cinder-volume[84696]: DEBUG cinder.utils [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_02aba338-fa31-4e46-8fcf-7f86191e44d1/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:24:22 user cinder-volume[84696]: DEBUG cinder.utils [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 40.083(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:24:22 user cinder-volume[84696]: DEBUG cinder.utils [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:24:28 user cinder-volume[84696]: DEBUG cinder.utils [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 137.396(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:24:28 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:24:28 user cinder-volume[84696]: INFO cinder.message.api [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Creating message record for request_id = req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 Apr 17 16:24:28 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 16:24:28 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (987bfcf0-d50d-414e-819e-cbf51ab05454) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 16:24:28 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 16:24:28 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:22:11Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesNegativeTest-Volume-375402817',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=2e8580be-c7f2-4d3a-9007-d192a5b029b3,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='935190a703294690b79c182d789f31e4',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:22:11Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:22:11Z,use_quota=True,user_id='9296aa0c28024371bac0c53928bff421',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 16:24:28 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:22:11Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesNegativeTest-Volume-375402817',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=2e8580be-c7f2-4d3a-9007-d192a5b029b3,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='935190a703294690b79c182d789f31e4',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:22:11Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:22:11Z,use_quota=True,user_id='9296aa0c28024371bac0c53928bff421',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(2e8580be-c7f2-4d3a-9007-d192a5b029b3),volume_id=2e8580be-c7f2-4d3a-9007-d192a5b029b3,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': '2e8580be-c7f2-4d3a-9007-d192a5b029b3', 'volume_name': 'volume-2e8580be-c7f2-4d3a-9007-d192a5b029b3', 'volume_size': 1}} Apr 17 16:24:28 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'retry': {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']}, 'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': '2e8580be-c7f2-4d3a-9007-d192a5b029b3', 'volume': {'id': '2e8580be-c7f2-4d3a-9007-d192a5b029b3', '_name_id': None, 'ec2_id': None, 'user_id': '9296aa0c28024371bac0c53928bff421', 'project_id': '935190a703294690b79c182d789f31e4', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-VolumesNegativeTest-Volume-375402817', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T16:22:11.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-2e8580be-c7f2-4d3a-9007-d192a5b029b3', 'name_id': '2e8580be-c7f2-4d3a-9007-d192a5b029b3', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumesNegativeTest-Volume-375402817', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '935190a703294690b79c182d789f31e4', 'qos_specs': None, 'reservations': ['8302a15a-cac9-4777-aeda-3cd0f18e59f0', '4f81147c-1b8e-4140-ae06-353eac50d673', '0d992e26-ac75-42fe-82b4-74e2842f6518', 'ce106309-de8e-43ad-89d6-aaedb5c859b5'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': '9296aa0c28024371bac0c53928bff421', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumesNegativeTest-Volume-375402817', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '935190a703294690b79c182d789f31e4', 'qos_specs': None, 'reservations': ['8302a15a-cac9-4777-aeda-3cd0f18e59f0', '4f81147c-1b8e-4140-ae06-353eac50d673', '0d992e26-ac75-42fe-82b4-74e2842f6518', 'ce106309-de8e-43ad-89d6-aaedb5c859b5'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': '9296aa0c28024371bac0c53928bff421', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': '9296aa0c28024371bac0c53928bff421', 'metadata': {}, 'qos_specs': None}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(2e8580be-c7f2-4d3a-9007-d192a5b029b3),volume_id=2e8580be-c7f2-4d3a-9007-d192a5b029b3,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:22:11Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesNegativeTest-Volume-375402817',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=2e8580be-c7f2-4d3a-9007-d192a5b029b3,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='935190a703294690b79c182d789f31e4',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:22:11Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:22:11Z,use_quota=True,user_id='9296aa0c28024371bac0c53928bff421',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 16:24:28 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:22:11Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesNegativeTest-Volume-375402817',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=2e8580be-c7f2-4d3a-9007-d192a5b029b3,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='935190a703294690b79c182d789f31e4',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:22:11Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:22:11Z,use_quota=True,user_id='9296aa0c28024371bac0c53928bff421',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:22:11Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesNegativeTest-Volume-375402817',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=2e8580be-c7f2-4d3a-9007-d192a5b029b3,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='935190a703294690b79c182d789f31e4',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:22:11Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:22:11Z,use_quota=True,user_id='9296aa0c28024371bac0c53928bff421',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 16:24:28 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 16:24:28 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 16:24:28 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:24:28 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 16:24:28 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:24:28 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 16:24:28 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:24:28 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 16:24:28 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:24:28 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 16:24:28 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:24:28 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 16:24:28 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:24:28 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 16:24:28 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:24:28 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 16:24:28 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:24:28 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:24:28 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 16:24:28 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 16:24:28 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (987bfcf0-d50d-414e-819e-cbf51ab05454) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:24:28 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (987bfcf0-d50d-414e-819e-cbf51ab05454) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:24:28 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (3408d748-cf33-44b9-9c31-4ef573d30a9a) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:24:28 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (3408d748-cf33-44b9-9c31-4ef573d30a9a) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:24:28 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (b12c230c-770d-447e-a211-66cab96aa07a) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:24:28 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (b12c230c-770d-447e-a211-66cab96aa07a) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:24:28 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (242c0d4b-5c14-4989-a5d0-47d010c8ba81) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:24:28 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Updating volume 2e8580be-c7f2-4d3a-9007-d192a5b029b3 with {'scheduled_at': datetime.datetime(2023, 4, 17, 16, 24, 28, 994246), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 16:24:29 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Volume 2e8580be-c7f2-4d3a-9007-d192a5b029b3: re-scheduling SchedulerAPI.create_volume attempt 1 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 16:24:29 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Volume 2e8580be-c7f2-4d3a-9007-d192a5b029b3: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 16:24:29 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] deleating volume v_2e8580be-c7f2-4d3a-9007-d192a5b029b3 {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 16:24:29 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] get snapshots for volume v_2e8580be-c7f2-4d3a-9007-d192a5b029b3 {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 16:24:29 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_2e8580be-c7f2-4d3a-9007-d192a5b029b3/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:24:29 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_2e8580be-c7f2-4d3a-9007-d192a5b029b3/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:24:29 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Volume reschedule parameters: True retry: {'num_attempts': 2, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 16:24:29 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Flow 'volume_create_manager' (c075b31d-6ee9-4251-a364-5c1f818b4541) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 16:24:29 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (a26a2d6c-d88b-4ea5-abfa-8928c59bd5e6) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:24:29 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (a26a2d6c-d88b-4ea5-abfa-8928c59bd5e6) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:22:11Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesNegativeTest-Volume-375402817',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=2e8580be-c7f2-4d3a-9007-d192a5b029b3,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='935190a703294690b79c182d789f31e4',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:24:29Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:24:29Z,use_quota=True,user_id='9296aa0c28024371bac0c53928bff421',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:24:29 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (0c39da14-5af1-4383-938b-31d3f459502d) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:24:29 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (0c39da14-5af1-4383-938b-31d3f459502d) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:24:29 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (3ef9372d-af18-41ff-b9ba-6ab11ff2ec8a) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:24:29 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (3ef9372d-af18-41ff-b9ba-6ab11ff2ec8a) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': '2e8580be-c7f2-4d3a-9007-d192a5b029b3', 'volume_name': 'volume-2e8580be-c7f2-4d3a-9007-d192a5b029b3', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:24:29 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (926e76f2-7bac-451f-af08-c4f3134c3842) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:24:29 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (926e76f2-7bac-451f-af08-c4f3134c3842) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:24:29 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (a933103a-5b24-45a6-8021-a6eb82c39cfc) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:24:29 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Volume 2e8580be-c7f2-4d3a-9007-d192a5b029b3: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-2e8580be-c7f2-4d3a-9007-d192a5b029b3', 'volume_size': 1} Apr 17 16:24:29 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] creating volume v_2e8580be-c7f2-4d3a-9007-d192a5b029b3. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 16:24:29 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] create volume {'name': 'v_2e8580be-c7f2-4d3a-9007-d192a5b029b3', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 16:24:29 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:24:29 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:24:31 user cinder-volume[84696]: DEBUG cinder.utils [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.967(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:24:31 user cinder-volume[84696]: DEBUG cinder.utils [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_2e8580be-c7f2-4d3a-9007-d192a5b029b3/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:24:31 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 140.024(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:24:31 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:24:31 user cinder-volume[84696]: ERROR oslo_service.periodic_task [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Error during VolumeManager.publish_service_capabilities: TypeError: 'NoneType' object is not subscriptable Apr 17 16:24:31 user cinder-volume[84696]: ERROR oslo_service.periodic_task Traceback (most recent call last): Apr 17 16:24:31 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/usr/local/lib/python3.10/dist-packages/oslo_service/periodic_task.py", line 216, in run_periodic_tasks Apr 17 16:24:31 user cinder-volume[84696]: ERROR oslo_service.periodic_task task(self, context) Apr 17 16:24:31 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/manager.py", line 2876, in publish_service_capabilities Apr 17 16:24:31 user cinder-volume[84696]: ERROR oslo_service.periodic_task self._report_driver_status(context) Apr 17 16:24:31 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/manager.py", line 2755, in _report_driver_status Apr 17 16:24:31 user cinder-volume[84696]: ERROR oslo_service.periodic_task volume_stats = get_stats() Apr 17 16:24:31 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/usr/local/lib/python3.10/dist-packages/oslo_utils/timeutils.py", line 319, in wrapper Apr 17 16:24:31 user cinder-volume[84696]: ERROR oslo_service.periodic_task result = func(*args, **kwargs) Apr 17 16:24:31 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/manager.py", line 2753, in get_stats Apr 17 16:24:31 user cinder-volume[84696]: ERROR oslo_service.periodic_task return self.driver.get_volume_stats(refresh=True) Apr 17 16:24:31 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/driver.py", line 666, in get_volume_stats Apr 17 16:24:31 user cinder-volume[84696]: ERROR oslo_service.periodic_task self._update_volume_stats() Apr 17 16:24:31 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 726, in _update_volume_stats Apr 17 16:24:31 user cinder-volume[84696]: ERROR oslo_service.periodic_task pool_stats = self.ra.get_pool_stats() Apr 17 16:24:31 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 897, in get_pool_stats Apr 17 16:24:31 user cinder-volume[84696]: ERROR oslo_service.periodic_task if not resp["error"] and resp["code"] == 200: Apr 17 16:24:31 user cinder-volume[84696]: ERROR oslo_service.periodic_task TypeError: 'NoneType' object is not subscriptable Apr 17 16:24:31 user cinder-volume[84696]: ERROR oslo_service.periodic_task Apr 17 16:24:32 user cinder-volume[84696]: DEBUG cinder.utils [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.796(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:24:32 user cinder-volume[84696]: DEBUG cinder.utils [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:24:32 user cinder-volume[84696]: WARNING oslo.service.loopingcall [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Function 'cinder.service.Service.periodic_tasks' run outlasted interval by 80.03 sec Apr 17 16:24:32 user cinder-volume[84696]: DEBUG oslo_service.periodic_task [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Running periodic task VolumeManager.publish_service_capabilities {{(pid=84696) run_periodic_tasks /usr/local/lib/python3.10/dist-packages/oslo_service/periodic_task.py:210}} Apr 17 16:24:32 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Updating volume stats {{(pid=84696) _update_volume_stats /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:724}} Apr 17 16:24:32 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Get pool Pool-0 fsprops {{(pid=84696) get_pool_stats /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:894}} Apr 17 16:24:32 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:24:32 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:24:35 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 3.066(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:24:35 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:24:35 user cinder-volume[84696]: DEBUG cinder.utils [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.041(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:24:35 user cinder-volume[84696]: DEBUG cinder.utils [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_2e8580be-c7f2-4d3a-9007-d192a5b029b3/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:24:35 user cinder-volume[84696]: DEBUG cinder.utils [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.868(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:24:35 user cinder-volume[84696]: DEBUG cinder.utils [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:24:38 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.137(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:24:38 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:24:41 user cinder-volume[84696]: DEBUG cinder.utils [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.010(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:24:41 user cinder-volume[84696]: DEBUG cinder.utils [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:24:41 user cinder-volume[84696]: DEBUG cinder.utils [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.187(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:24:41 user cinder-volume[84696]: DEBUG cinder.utils [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_2e8580be-c7f2-4d3a-9007-d192a5b029b3/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:24:41 user cinder-volume[84696]: DEBUG cinder.utils [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 74.554(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:24:41 user cinder-volume[84696]: DEBUG cinder.utils [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:24:41 user cinder-volume[84696]: DEBUG cinder.utils [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 74.772(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:24:41 user cinder-volume[84696]: DEBUG cinder.utils [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_b900713d-b991-43a5-b403-483dfded7e11/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:24:41 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-d65d2053-22d1-4404-95e3-97728f5ffd95 tempest-GroupsTest-789764105 None] remove_export for volume: 02aba338-fa31-4e46-8fcf-7f86191e44d1. {{(pid=84696) remove_export /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:718}} Apr 17 16:24:41 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-d65d2053-22d1-4404-95e3-97728f5ffd95 tempest-GroupsTest-789764105 None] remove export {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:912}} Apr 17 16:24:41 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-d65d2053-22d1-4404-95e3-97728f5ffd95 tempest-GroupsTest-789764105 None] detach volume:Volume(_name_id=None,admin_metadata={},attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:21:25Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-volume-1452364054',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=fe50cd00-dcd4-4b21-a4e9-6f226f8c6ac1,host='user@joviandss-2#joviandss-2',id=02aba338-fa31-4e46-8fcf-7f86191e44d1,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='e10e34b8774b42d8a4cb1f78d72b411d',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:23:42Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='deleting',terminated_at=None,updated_at=2023-04-17T16:24:41Z,use_quota=True,user_id='e7392ed432ad41279967273f16b4a4c8',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(400bec8d-bd01-4057-8fb8-49a786cd39f1),volume_type_id=400bec8d-bd01-4057-8fb8-49a786cd39f1) from target:iqn.2010-10.org.openstack:02aba338-fa31-4e46-8fcf-7f86191e44d1. {{(pid=84696) _remove_target_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:913}} Apr 17 16:24:41 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-d65d2053-22d1-4404-95e3-97728f5ffd95 tempest-GroupsTest-789764105 None] detach volume v_02aba338-fa31-4e46-8fcf-7f86191e44d1 from target iqn.2010-10.org.openstack:02aba338-fa31-4e46-8fcf-7f86191e44d1 {{(pid=84696) detach_target_vol /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:606}} Apr 17 16:24:41 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-d65d2053-22d1-4404-95e3-97728f5ffd95 tempest-GroupsTest-789764105 None] Sending pool request DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:02aba338-fa31-4e46-8fcf-7f86191e44d1/luns/v_02aba338-fa31-4e46-8fcf-7f86191e44d1 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:24:41 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-d65d2053-22d1-4404-95e3-97728f5ffd95 tempest-GroupsTest-789764105 None] Sending DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:02aba338-fa31-4e46-8fcf-7f86191e44d1/luns/v_02aba338-fa31-4e46-8fcf-7f86191e44d1 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:24:42 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Volume reschedule parameters: True retry: {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 16:24:42 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Flow 'volume_create_manager' (8b4577f5-1014-4b6b-8e35-e6e3dea11c23) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 16:24:42 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (c1cc5b5b-0595-4919-a9be-081891bbc222) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:24:42 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (c1cc5b5b-0595-4919-a9be-081891bbc222) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:24:42Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesDeleteCascade-Volume-1549123617',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=53029faa-77b2-467d-8a64-eb5ef129e41c,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='abbfd52254f546b0bcfe289884d77f49',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:24:42Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:24:42Z,use_quota=True,user_id='6309fcbea41c471184457d276d32db73',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:24:42 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (d77b789b-f220-432a-8116-dd04f7251a2b) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:24:42 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (d77b789b-f220-432a-8116-dd04f7251a2b) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:24:42 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (917b9f19-733f-4891-979d-9d4adf0e89d2) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:24:42 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (917b9f19-733f-4891-979d-9d4adf0e89d2) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': '53029faa-77b2-467d-8a64-eb5ef129e41c', 'volume_name': 'volume-53029faa-77b2-467d-8a64-eb5ef129e41c', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:24:42 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (a8c72a42-2994-487a-b610-5d539ef74dae) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:24:42 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (a8c72a42-2994-487a-b610-5d539ef74dae) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:24:42 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (07a79a30-0bde-440f-8526-14c0dc2e9d5a) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:24:42 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Volume 53029faa-77b2-467d-8a64-eb5ef129e41c: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-53029faa-77b2-467d-8a64-eb5ef129e41c', 'volume_size': 1} Apr 17 16:24:42 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] creating volume v_53029faa-77b2-467d-8a64-eb5ef129e41c. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 16:24:42 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] create volume {'name': 'v_53029faa-77b2-467d-8a64-eb5ef129e41c', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 16:24:42 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:24:42 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:24:44 user cinder-volume[84696]: DEBUG cinder.utils [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 1.999(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:24:44 user cinder-volume[84696]: DEBUG cinder.utils [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:24:44 user cinder-volume[84696]: DEBUG cinder.utils [None req-d65d2053-22d1-4404-95e3-97728f5ffd95 tempest-GroupsTest-789764105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 3.062(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:24:44 user cinder-volume[84696]: DEBUG cinder.utils [None req-d65d2053-22d1-4404-95e3-97728f5ffd95 tempest-GroupsTest-789764105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:02aba338-fa31-4e46-8fcf-7f86191e44d1/luns/v_02aba338-fa31-4e46-8fcf-7f86191e44d1 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:24:44 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.287(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:24:44 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:24:45 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=13,cinder:UPDATE=4,cinder:INSERT=2 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:24:45 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=20,cinder:UPDATE=4 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:24:45 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Volume reschedule parameters: True retry: {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 16:24:45 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Flow 'volume_create_manager' (08b4dd37-7b56-4cd7-9be7-a82dd3ecb693) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 16:24:45 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (17a8ddec-f38d-4551-97a8-17d948ff30f2) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:24:45 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (17a8ddec-f38d-4551-97a8-17d948ff30f2) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:24:45Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-SnapshotsActionsTest-Volume-1453311494',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=35a67ca3-0f17-4a6c-a359-9173ce5f1aca,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='5b416855df3047d79fd4d27b2b1695b8',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:24:45Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:24:45Z,use_quota=True,user_id='535feccd53b44a24a33905f2f2a4795d',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:24:45 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (a86b9e02-957b-463f-8dc1-903e6e35d6d6) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:24:45 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (a86b9e02-957b-463f-8dc1-903e6e35d6d6) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:24:45 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (8308a4a0-3bd7-4b66-8ef0-d4ee3d506e8a) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:24:45 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (8308a4a0-3bd7-4b66-8ef0-d4ee3d506e8a) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': '35a67ca3-0f17-4a6c-a359-9173ce5f1aca', 'volume_name': 'volume-35a67ca3-0f17-4a6c-a359-9173ce5f1aca', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:24:45 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (e710bb08-39ca-4436-bac1-ad77c9ac3fe4) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:24:45 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (e710bb08-39ca-4436-bac1-ad77c9ac3fe4) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:24:45 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (1fe06e39-907e-44ec-a6b9-166ddc0adc1f) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:24:45 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Volume 35a67ca3-0f17-4a6c-a359-9173ce5f1aca: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-35a67ca3-0f17-4a6c-a359-9173ce5f1aca', 'volume_size': 1} Apr 17 16:24:45 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] creating volume v_35a67ca3-0f17-4a6c-a359-9173ce5f1aca. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 16:24:45 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] create volume {'name': 'v_35a67ca3-0f17-4a6c-a359-9173ce5f1aca', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 16:24:45 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:24:45 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:24:47 user cinder-volume[84696]: DEBUG cinder.utils [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 1.887(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:24:47 user cinder-volume[84696]: DEBUG cinder.utils [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:24:47 user cinder-volume[84696]: DEBUG cinder.utils [None req-d65d2053-22d1-4404-95e3-97728f5ffd95 tempest-GroupsTest-789764105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.133(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:24:47 user cinder-volume[84696]: DEBUG cinder.utils [None req-d65d2053-22d1-4404-95e3-97728f5ffd95 tempest-GroupsTest-789764105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:02aba338-fa31-4e46-8fcf-7f86191e44d1/luns/v_02aba338-fa31-4e46-8fcf-7f86191e44d1 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:24:47 user cinder-volume[84696]: DEBUG cinder.utils [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.073(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:24:47 user cinder-volume[84696]: DEBUG cinder.utils [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:24:50 user cinder-volume[84696]: DEBUG cinder.utils [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.225(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:24:50 user cinder-volume[84696]: DEBUG cinder.utils [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:24:50 user cinder-volume[84696]: DEBUG cinder.utils [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.401(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:24:50 user cinder-volume[84696]: DEBUG cinder.utils [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_2e8580be-c7f2-4d3a-9007-d192a5b029b3/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:24:50 user cinder-volume[84696]: DEBUG cinder.utils [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 4.961(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:24:50 user cinder-volume[84696]: DEBUG cinder.utils [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:24:53 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.498(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:24:53 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:24:53 user cinder-volume[84696]: DEBUG cinder.utils [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 139.900(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:24:53 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_4725fbeb-e869-48db-a487-33998b955f83/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:24:53 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (0ea2e582-cdc9-48a0-a628-5209cfb9fbf9) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 16:24:53 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (5aa2bdf6-3a17-4601-943e-e453086434ef) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:24:53 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (5aa2bdf6-3a17-4601-943e-e453086434ef) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:24:53 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Flow 'volume_create_manager' (2f4b7229-a766-417a-884d-bb3f4475738e) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 16:24:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-86e82c7b-2d41-4c71-a488-5361f4ac7ada tempest-VolumesTransfersV357Test-333017112 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 16:24:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 16:24:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 16:24:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 16:24:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 16:24:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 16:24:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 16:24:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 16:24:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 16:24:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 16:24:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 16:24:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 16:24:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 16:24:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 16:24:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 16:24:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 16:24:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 16:24:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 16:24:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 16:24:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 16:24:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 16:24:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 16:24:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 16:24:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 16:24:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:24:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 16:24:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:24:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 16:24:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:24:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 16:24:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:24:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 16:24:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:24:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 16:24:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:24:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 16:24:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:24:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 16:24:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:24:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:24:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 16:24:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 16:24:53 user cinder-volume[84696]: DEBUG cinder.utils [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 11.236(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:24:53 user cinder-volume[84696]: DEBUG cinder.utils [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:24:53 user cinder-volume[84696]: DEBUG cinder.utils [None req-d65d2053-22d1-4404-95e3-97728f5ffd95 tempest-GroupsTest-789764105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.298(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:24:53 user cinder-volume[84696]: DEBUG cinder.utils [None req-d65d2053-22d1-4404-95e3-97728f5ffd95 tempest-GroupsTest-789764105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:02aba338-fa31-4e46-8fcf-7f86191e44d1/luns/v_02aba338-fa31-4e46-8fcf-7f86191e44d1 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:24:56 user cinder-volume[84696]: DEBUG cinder.utils [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 11.104(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:24:56 user cinder-volume[84696]: DEBUG cinder.utils [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:24:56 user cinder-volume[84696]: DEBUG cinder.utils [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 74.400(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:24:56 user cinder-volume[84696]: DEBUG cinder.utils [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_94610127-c8b8-4df4-9e39-a1f41b5a9b98/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:24:56 user cinder-volume[84696]: DEBUG cinder.utils [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 74.096(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:24:56 user cinder-volume[84696]: DEBUG cinder.utils [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:24:56 user cinder-volume[84696]: DEBUG cinder.utils [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 74.393(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:24:56 user cinder-volume[84696]: DEBUG cinder.utils [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_02aba338-fa31-4e46-8fcf-7f86191e44d1/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:24:56 user cinder-volume[84696]: DEBUG cinder.utils [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 74.174(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:24:56 user cinder-volume[84696]: DEBUG cinder.utils [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:24:59 user cinder-volume[84696]: DEBUG cinder.utils [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 139.698(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:24:59 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:24:59 user cinder-volume[84696]: INFO cinder.message.api [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Creating message record for request_id = req-f436a31c-1175-4e56-a839-11ba99c1cd60 Apr 17 16:24:59 user cinder-volume[84696]: DEBUG cinder.utils [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 139.928(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:24:59 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_c9fa0a17-69d4-40d0-9d65-af6ae349cfd8/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:24:59 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (85d4bcaf-ef93-4bed-8fe3-77dd128ef3ae) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 16:24:59 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (b6c8e53c-bd02-45b1-aa4e-f9c9016d7a0e) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:24:59 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (b6c8e53c-bd02-45b1-aa4e-f9c9016d7a0e) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:24:59 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Flow 'volume_create_manager' (ab0bf919-e871-4d35-a73a-16c12db71ae4) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 16:24:59 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 16:24:59 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 16:24:59 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 16:24:59 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 16:24:59 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 16:24:59 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 16:24:59 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 16:24:59 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 16:24:59 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 16:24:59 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 16:24:59 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 16:24:59 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 16:24:59 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 16:24:59 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 16:24:59 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 16:24:59 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 16:24:59 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 16:24:59 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 16:24:59 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 16:24:59 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 16:24:59 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 16:24:59 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 16:24:59 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 16:24:59 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 16:24:59 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:24:59 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 16:24:59 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:24:59 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 16:24:59 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:24:59 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 16:24:59 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:24:59 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 16:24:59 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:24:59 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 16:24:59 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:24:59 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 16:24:59 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:24:59 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 16:24:59 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:24:59 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:24:59 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 16:24:59 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 16:24:59 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 16:24:59 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (a388ee1d-0d27-46cd-a4cf-0ab732a282d7) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 16:24:59 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 16:24:59 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:18:00Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumeSnapshotQuotasNegativeTestJSON-Volume-1211070769',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=c9fa0a17-69d4-40d0-9d65-af6ae349cfd8,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='46b47eb1ecf7468abe1458b30e249be8',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:22:40Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:22:40Z,use_quota=True,user_id='48df3e98011f41a7938b3da3e07615c2',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 16:24:59 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:18:00Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumeSnapshotQuotasNegativeTestJSON-Volume-1211070769',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=c9fa0a17-69d4-40d0-9d65-af6ae349cfd8,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='46b47eb1ecf7468abe1458b30e249be8',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:22:40Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:22:40Z,use_quota=True,user_id='48df3e98011f41a7938b3da3e07615c2',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(c9fa0a17-69d4-40d0-9d65-af6ae349cfd8),volume_id=c9fa0a17-69d4-40d0-9d65-af6ae349cfd8,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': 'c9fa0a17-69d4-40d0-9d65-af6ae349cfd8', 'volume_name': 'volume-c9fa0a17-69d4-40d0-9d65-af6ae349cfd8', 'volume_size': 1}} Apr 17 16:24:59 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': 'c9fa0a17-69d4-40d0-9d65-af6ae349cfd8', 'volume': {'id': 'c9fa0a17-69d4-40d0-9d65-af6ae349cfd8', '_name_id': None, 'ec2_id': None, 'user_id': '48df3e98011f41a7938b3da3e07615c2', 'project_id': '46b47eb1ecf7468abe1458b30e249be8', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-VolumeSnapshotQuotasNegativeTestJSON-Volume-1211070769', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'admin_metadata': {}, 'glance_metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T16:17:59.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-c9fa0a17-69d4-40d0-9d65-af6ae349cfd8', 'name_id': 'c9fa0a17-69d4-40d0-9d65-af6ae349cfd8', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumeSnapshotQuotasNegativeTestJSON-Volume-1211070769', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '46b47eb1ecf7468abe1458b30e249be8', 'qos_specs': None, 'reservations': ['019a7782-5155-4c89-a83b-235283c575e4', '0bfbe139-e9e6-4454-941c-cc4e8c814eb3', 'df625689-4086-45a1-a606-f6541e8045a3', '80b6f310-1127-40c3-bb09-d4d6b83f85c6'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': '48df3e98011f41a7938b3da3e07615c2', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumeSnapshotQuotasNegativeTestJSON-Volume-1211070769', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '46b47eb1ecf7468abe1458b30e249be8', 'qos_specs': None, 'reservations': ['019a7782-5155-4c89-a83b-235283c575e4', '0bfbe139-e9e6-4454-941c-cc4e8c814eb3', 'df625689-4086-45a1-a606-f6541e8045a3', '80b6f310-1127-40c3-bb09-d4d6b83f85c6'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': '48df3e98011f41a7938b3da3e07615c2', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': '48df3e98011f41a7938b3da3e07615c2', 'metadata': {}, 'qos_specs': None, 'retry': {'num_attempts': 3, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']}}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(c9fa0a17-69d4-40d0-9d65-af6ae349cfd8),volume_id=c9fa0a17-69d4-40d0-9d65-af6ae349cfd8,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:18:00Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumeSnapshotQuotasNegativeTestJSON-Volume-1211070769',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=c9fa0a17-69d4-40d0-9d65-af6ae349cfd8,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='46b47eb1ecf7468abe1458b30e249be8',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:22:40Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:22:40Z,use_quota=True,user_id='48df3e98011f41a7938b3da3e07615c2',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 16:24:59 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:18:00Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumeSnapshotQuotasNegativeTestJSON-Volume-1211070769',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=c9fa0a17-69d4-40d0-9d65-af6ae349cfd8,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='46b47eb1ecf7468abe1458b30e249be8',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:22:40Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:22:40Z,use_quota=True,user_id='48df3e98011f41a7938b3da3e07615c2',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:18:00Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumeSnapshotQuotasNegativeTestJSON-Volume-1211070769',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=c9fa0a17-69d4-40d0-9d65-af6ae349cfd8,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='46b47eb1ecf7468abe1458b30e249be8',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:22:40Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:22:40Z,use_quota=True,user_id='48df3e98011f41a7938b3da3e07615c2',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 16:24:59 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 16:24:59 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 16:24:59 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:24:59 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 16:24:59 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:24:59 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 16:24:59 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:24:59 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 16:24:59 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:24:59 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 16:24:59 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:24:59 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 16:24:59 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:24:59 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 16:24:59 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:24:59 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 16:24:59 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:24:59 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:24:59 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 16:24:59 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 16:24:59 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (a388ee1d-0d27-46cd-a4cf-0ab732a282d7) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:24:59 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (a388ee1d-0d27-46cd-a4cf-0ab732a282d7) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:24:59 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (86f3be07-58ed-409e-bd03-d75ee4497c5c) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:24:59 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (86f3be07-58ed-409e-bd03-d75ee4497c5c) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:24:59 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (1f63cb8c-4036-4247-8a6b-493a2cda3fa9) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:24:59 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (1f63cb8c-4036-4247-8a6b-493a2cda3fa9) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:24:59 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (28e9bdbf-3834-419e-8596-3394b9b72ead) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:24:59 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Updating volume c9fa0a17-69d4-40d0-9d65-af6ae349cfd8 with {'scheduled_at': datetime.datetime(2023, 4, 17, 16, 24, 59, 731844), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 16:24:59 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Volume c9fa0a17-69d4-40d0-9d65-af6ae349cfd8: re-scheduling SchedulerAPI.create_volume attempt 3 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 16:24:59 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Volume c9fa0a17-69d4-40d0-9d65-af6ae349cfd8: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 16:24:59 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] deleating volume v_c9fa0a17-69d4-40d0-9d65-af6ae349cfd8 {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 16:24:59 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] get snapshots for volume v_c9fa0a17-69d4-40d0-9d65-af6ae349cfd8 {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 16:24:59 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_c9fa0a17-69d4-40d0-9d65-af6ae349cfd8/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:24:59 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_c9fa0a17-69d4-40d0-9d65-af6ae349cfd8/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:25:02 user cinder-volume[84696]: DEBUG cinder.utils [None req-d65d2053-22d1-4404-95e3-97728f5ffd95 tempest-GroupsTest-789764105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.491(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:25:02 user cinder-volume[84696]: DEBUG cinder.utils [None req-d65d2053-22d1-4404-95e3-97728f5ffd95 tempest-GroupsTest-789764105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:02aba338-fa31-4e46-8fcf-7f86191e44d1/luns/v_02aba338-fa31-4e46-8fcf-7f86191e44d1 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:25:02 user cinder-volume[84696]: DEBUG cinder.utils [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 20.432(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:25:02 user cinder-volume[84696]: DEBUG cinder.utils [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:25:02 user cinder-volume[84696]: DEBUG cinder.utils [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.959(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:25:02 user cinder-volume[84696]: DEBUG cinder.utils [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_c9fa0a17-69d4-40d0-9d65-af6ae349cfd8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:25:05 user cinder-volume[84696]: DEBUG cinder.utils [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 139.928(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:25:05 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_7f7b3b2e-e41d-466a-a3c2-e64e3d63b949/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:25:05 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (ed1e4b67-153e-4602-81cf-2a65404dafbc) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 16:25:05 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (c41a31ea-c2c1-4456-a0cd-6a71f46175b9) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:25:05 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (c41a31ea-c2c1-4456-a0cd-6a71f46175b9) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:25:05 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Flow 'volume_create_manager' (2f24858c-65c0-4dd3-8b07-1964f233c23d) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 16:25:05 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 16:25:05 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 16:25:05 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 16:25:05 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 16:25:05 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 16:25:05 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 16:25:05 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 16:25:05 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 16:25:05 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 16:25:05 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 16:25:05 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 16:25:05 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 16:25:05 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 16:25:05 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 16:25:05 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 16:25:05 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 16:25:05 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 16:25:05 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 16:25:05 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 16:25:05 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 16:25:05 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 16:25:05 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 16:25:05 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 16:25:05 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 16:25:05 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:25:05 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 16:25:05 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:25:05 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 16:25:05 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:25:05 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 16:25:05 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:25:05 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 16:25:05 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:25:05 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 16:25:05 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:25:05 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 16:25:05 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:25:05 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 16:25:05 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:25:05 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:25:05 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 16:25:05 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 16:25:05 user cinder-volume[84696]: DEBUG cinder.utils [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 139.791(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:25:05 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:25:05 user cinder-volume[84696]: INFO cinder.message.api [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Creating message record for request_id = req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 Apr 17 16:25:05 user cinder-volume[84696]: DEBUG cinder.utils [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.049(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:25:05 user cinder-volume[84696]: DEBUG cinder.utils [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_c9fa0a17-69d4-40d0-9d65-af6ae349cfd8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:25:05 user cinder-volume[84696]: DEBUG cinder.utils [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 20.341(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:25:05 user cinder-volume[84696]: DEBUG cinder.utils [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:25:05 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 16:25:05 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (0088e165-b524-4b7c-b42d-8853f71b8293) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 16:25:05 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 16:25:05 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:18:08Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-volume-769265242',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=f28ccf81-6553-4a98-83d7-400266578051,host='user@joviandss-2#joviandss-2',id=7f7b3b2e-e41d-466a-a3c2-e64e3d63b949,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='93c135fbb6f14928a9167adb3724faa3',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:22:46Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='deleting',terminated_at=None,updated_at=2023-04-17T16:22:46Z,use_quota=True,user_id='90edc8ffb805460794445bc433aa473f',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(be5f492b-b41f-4ad5-bffd-42eaafb454b4),volume_type_id=be5f492b-b41f-4ad5-bffd-42eaafb454b4)}, 'provides': None} Apr 17 16:25:05 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:18:08Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-volume-769265242',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=f28ccf81-6553-4a98-83d7-400266578051,host='user@joviandss-2#joviandss-2',id=7f7b3b2e-e41d-466a-a3c2-e64e3d63b949,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='93c135fbb6f14928a9167adb3724faa3',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:22:46Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='deleting',terminated_at=None,updated_at=2023-04-17T16:22:46Z,use_quota=True,user_id='90edc8ffb805460794445bc433aa473f',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(be5f492b-b41f-4ad5-bffd-42eaafb454b4),volume_type_id=be5f492b-b41f-4ad5-bffd-42eaafb454b4), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=f28ccf81-6553-4a98-83d7-400266578051,image_id=None,operation=,resource_backend='user@joviandss-2',snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(7f7b3b2e-e41d-466a-a3c2-e64e3d63b949),volume_id=7f7b3b2e-e41d-466a-a3c2-e64e3d63b949,volume_properties=VolumeProperties,volume_type=VolumeType(be5f492b-b41f-4ad5-bffd-42eaafb454b4))}, 'provides': {'status': 'deleting', 'type': 'raw', 'volume_id': '7f7b3b2e-e41d-466a-a3c2-e64e3d63b949', 'volume_name': 'volume-7f7b3b2e-e41d-466a-a3c2-e64e3d63b949', 'volume_size': 1}} Apr 17 16:25:05 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'request_spec': {'consistencygroup_id': None, 'group_id': 'f28ccf81-6553-4a98-83d7-400266578051', 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': '7f7b3b2e-e41d-466a-a3c2-e64e3d63b949', 'volume': {'id': '7f7b3b2e-e41d-466a-a3c2-e64e3d63b949', '_name_id': None, 'ec2_id': None, 'user_id': '90edc8ffb805460794445bc433aa473f', 'project_id': '93c135fbb6f14928a9167adb3724faa3', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-volume-769265242', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': 'be5f492b-b41f-4ad5-bffd-42eaafb454b4', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': 'f28ccf81-6553-4a98-83d7-400266578051', 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'admin_metadata': {}, 'glance_metadata': {}, 'volume_type': {'id': 'be5f492b-b41f-4ad5-bffd-42eaafb454b4', 'name': 'tempest-GroupsV314Test-volume-type-1100522366', 'description': None, 'is_public': True, 'extra_specs': {}, 'qos_specs_id': None, 'created_at': '2023-04-17T16:18:06.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T16:18:07.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-7f7b3b2e-e41d-466a-a3c2-e64e3d63b949', 'name_id': '7f7b3b2e-e41d-466a-a3c2-e64e3d63b949', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': 'be5f492b-b41f-4ad5-bffd-42eaafb454b4', 'name': 'tempest-GroupsV314Test-volume-type-1100522366', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {}, 'qos_specs_id': None, 'created_at': '2023-04-17T16:18:06.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': 'f28ccf81-6553-4a98-83d7-400266578051', 'display_description': None, 'display_name': 'tempest-volume-769265242', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '93c135fbb6f14928a9167adb3724faa3', 'qos_specs': None, 'reservations': ['16413670-7ac2-4f17-a10d-ae77a197fc22', '05fb658c-9b25-41f9-b551-6c9fc7ca8983', 'bdd8bc3b-5d24-4a01-90e5-fca884142434', '6c4f66ea-ee22-430e-89df-e1bdd93b9f58'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': '90edc8ffb805460794445bc433aa473f', 'volume_type_id': 'be5f492b-b41f-4ad5-bffd-42eaafb454b4'}, 'resource_backend': 'user@joviandss-2', 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': 'f28ccf81-6553-4a98-83d7-400266578051', 'display_description': None, 'display_name': 'tempest-volume-769265242', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '93c135fbb6f14928a9167adb3724faa3', 'qos_specs': None, 'reservations': ['16413670-7ac2-4f17-a10d-ae77a197fc22', '05fb658c-9b25-41f9-b551-6c9fc7ca8983', 'bdd8bc3b-5d24-4a01-90e5-fca884142434', '6c4f66ea-ee22-430e-89df-e1bdd93b9f58'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': '90edc8ffb805460794445bc433aa473f', 'volume_type_id': 'be5f492b-b41f-4ad5-bffd-42eaafb454b4'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T16:18:06Z,deleted=False,deleted_at=None,description=None,extra_specs={},id=be5f492b-b41f-4ad5-bffd-42eaafb454b4,is_public=True,name='tempest-GroupsV314Test-volume-type-1100522366',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T16:18:06Z,deleted=False,deleted_at=None,description=None,extra_specs={},id=be5f492b-b41f-4ad5-bffd-42eaafb454b4,is_public=True,name='tempest-GroupsV314Test-volume-type-1100522366',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': '90edc8ffb805460794445bc433aa473f', 'metadata': {}, 'qos_specs': None, 'retry': {'num_attempts': 3, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']}}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=f28ccf81-6553-4a98-83d7-400266578051,image_id=None,operation=,resource_backend='user@joviandss-2',snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(7f7b3b2e-e41d-466a-a3c2-e64e3d63b949),volume_id=7f7b3b2e-e41d-466a-a3c2-e64e3d63b949,volume_properties=VolumeProperties,volume_type=VolumeType(be5f492b-b41f-4ad5-bffd-42eaafb454b4)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:18:08Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-volume-769265242',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=f28ccf81-6553-4a98-83d7-400266578051,host='user@joviandss-2#joviandss-2',id=7f7b3b2e-e41d-466a-a3c2-e64e3d63b949,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='93c135fbb6f14928a9167adb3724faa3',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:22:46Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='deleting',terminated_at=None,updated_at=2023-04-17T16:22:46Z,use_quota=True,user_id='90edc8ffb805460794445bc433aa473f',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(be5f492b-b41f-4ad5-bffd-42eaafb454b4),volume_type_id=be5f492b-b41f-4ad5-bffd-42eaafb454b4), 'context': }, 'provides': None} Apr 17 16:25:05 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:18:08Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-volume-769265242',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=f28ccf81-6553-4a98-83d7-400266578051,host='user@joviandss-2#joviandss-2',id=7f7b3b2e-e41d-466a-a3c2-e64e3d63b949,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='93c135fbb6f14928a9167adb3724faa3',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:22:46Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='deleting',terminated_at=None,updated_at=2023-04-17T16:22:46Z,use_quota=True,user_id='90edc8ffb805460794445bc433aa473f',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(be5f492b-b41f-4ad5-bffd-42eaafb454b4),volume_type_id=be5f492b-b41f-4ad5-bffd-42eaafb454b4)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:18:08Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-volume-769265242',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=f28ccf81-6553-4a98-83d7-400266578051,host='user@joviandss-2#joviandss-2',id=7f7b3b2e-e41d-466a-a3c2-e64e3d63b949,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='93c135fbb6f14928a9167adb3724faa3',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:22:46Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='deleting',terminated_at=None,updated_at=2023-04-17T16:22:46Z,use_quota=True,user_id='90edc8ffb805460794445bc433aa473f',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(be5f492b-b41f-4ad5-bffd-42eaafb454b4),volume_type_id=be5f492b-b41f-4ad5-bffd-42eaafb454b4)} Apr 17 16:25:05 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 16:25:05 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 16:25:05 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:25:05 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 16:25:05 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:25:05 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 16:25:05 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:25:05 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 16:25:05 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:25:05 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 16:25:05 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:25:05 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 16:25:05 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:25:05 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 16:25:05 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:25:05 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 16:25:05 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:25:05 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:25:05 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 16:25:05 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 16:25:05 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (0088e165-b524-4b7c-b42d-8853f71b8293) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:25:05 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (0088e165-b524-4b7c-b42d-8853f71b8293) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:25:05 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (156c1695-ab20-4c94-98ad-035e893f892f) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:25:05 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (156c1695-ab20-4c94-98ad-035e893f892f) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:25:05 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (de1fb827-2c8e-4b64-b262-95dcd9e034b6) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:25:05 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (de1fb827-2c8e-4b64-b262-95dcd9e034b6) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:25:05 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (c9f579d5-df6c-4c85-8f28-d3a84908979d) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:25:05 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Updating volume 7f7b3b2e-e41d-466a-a3c2-e64e3d63b949 with {'scheduled_at': datetime.datetime(2023, 4, 17, 16, 25, 5, 862729), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 16:25:05 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Volume 7f7b3b2e-e41d-466a-a3c2-e64e3d63b949: re-scheduling SchedulerAPI.create_volume attempt 3 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 16:25:05 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Volume 7f7b3b2e-e41d-466a-a3c2-e64e3d63b949: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 16:25:05 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] deleating volume v_7f7b3b2e-e41d-466a-a3c2-e64e3d63b949 {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 16:25:05 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] get snapshots for volume v_7f7b3b2e-e41d-466a-a3c2-e64e3d63b949 {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 16:25:05 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_7f7b3b2e-e41d-466a-a3c2-e64e3d63b949/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:25:05 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_7f7b3b2e-e41d-466a-a3c2-e64e3d63b949/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:25:08 user cinder-volume[84696]: DEBUG cinder.utils [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.974(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:25:08 user cinder-volume[84696]: DEBUG cinder.utils [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_7f7b3b2e-e41d-466a-a3c2-e64e3d63b949/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:25:08 user cinder-volume[84696]: DEBUG cinder.utils [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.834(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:25:08 user cinder-volume[84696]: DEBUG cinder.utils [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_2e8580be-c7f2-4d3a-9007-d192a5b029b3/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:25:08 user cinder-volume[84696]: DEBUG cinder.utils [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.662(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:25:08 user cinder-volume[84696]: DEBUG cinder.utils [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:25:11 user cinder-volume[84696]: DEBUG cinder.utils [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.172(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:25:11 user cinder-volume[84696]: DEBUG cinder.utils [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_c9fa0a17-69d4-40d0-9d65-af6ae349cfd8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:25:11 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.931(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:25:11 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:25:11 user cinder-volume[84696]: DEBUG cinder.utils [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.048(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:25:11 user cinder-volume[84696]: DEBUG cinder.utils [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_7f7b3b2e-e41d-466a-a3c2-e64e3d63b949/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:25:18 user cinder-volume[84696]: DEBUG cinder.utils [None req-d65d2053-22d1-4404-95e3-97728f5ffd95 tempest-GroupsTest-789764105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 37.751(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:25:18 user cinder-volume[84696]: DEBUG cinder.utils [None req-d65d2053-22d1-4404-95e3-97728f5ffd95 tempest-GroupsTest-789764105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:02aba338-fa31-4e46-8fcf-7f86191e44d1/luns/v_02aba338-fa31-4e46-8fcf-7f86191e44d1 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:25:18 user cinder-volume[84696]: DEBUG cinder.utils [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 13.091(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:25:18 user cinder-volume[84696]: DEBUG cinder.utils [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_7f7b3b2e-e41d-466a-a3c2-e64e3d63b949/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:25:18 user cinder-volume[84696]: DEBUG cinder.utils [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 36.692(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:25:18 user cinder-volume[84696]: DEBUG cinder.utils [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:25:22 user cinder-volume[84696]: DEBUG cinder.utils [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 22.284(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:25:22 user cinder-volume[84696]: DEBUG cinder.utils [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_c9fa0a17-69d4-40d0-9d65-af6ae349cfd8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:25:22 user cinder-volume[84696]: DEBUG cinder.utils [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 36.577(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:25:22 user cinder-volume[84696]: DEBUG cinder.utils [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:25:24 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=10,cinder:UPDATE=4,cinder:INSERT=2 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:25:24 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=12,cinder:INSERT=2,cinder:UPDATE=4 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:25:28 user cinder-volume[84696]: DEBUG cinder.utils [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 22.334(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:25:28 user cinder-volume[84696]: DEBUG cinder.utils [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_7f7b3b2e-e41d-466a-a3c2-e64e3d63b949/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:25:34 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=1,cinder:UPDATE=1 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:25:41 user cinder-volume[84696]: DEBUG cinder.utils [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 72.028(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:25:41 user cinder-volume[84696]: DEBUG cinder.utils [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_2e8580be-c7f2-4d3a-9007-d192a5b029b3/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:25:41 user cinder-volume[84696]: DEBUG cinder.utils [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 71.855(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:25:41 user cinder-volume[84696]: DEBUG cinder.utils [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:25:41 user cinder-volume[84696]: DEBUG cinder.utils [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 41.300(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:25:41 user cinder-volume[84696]: DEBUG cinder.utils [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_c9fa0a17-69d4-40d0-9d65-af6ae349cfd8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:25:44 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 72.122(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:25:44 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:25:44 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=3,cinder:UPDATE=1 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:25:47 user cinder-volume[84696]: DEBUG cinder.utils [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 140.533(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:25:47 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:25:47 user cinder-volume[84696]: INFO cinder.message.api [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Creating message record for request_id = req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf Apr 17 16:25:47 user cinder-volume[84696]: DEBUG cinder.utils [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 140.754(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:25:47 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_b900713d-b991-43a5-b403-483dfded7e11/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:25:47 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (ac6e72c4-80c1-44d3-8c37-428687e574ca) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 16:25:47 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (f2d11083-eeae-4c90-b3b0-925cc0f83d7e) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:25:47 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (f2d11083-eeae-4c90-b3b0-925cc0f83d7e) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:25:47 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Flow 'volume_create_manager' (f05b4bd3-9089-4b61-bdf1-73a44b40be8c) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 16:25:47 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 16:25:47 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 16:25:47 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 16:25:47 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 16:25:47 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 16:25:47 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 16:25:47 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 16:25:47 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 16:25:47 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 16:25:47 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 16:25:47 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 16:25:47 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 16:25:47 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 16:25:47 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 16:25:47 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 16:25:47 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 16:25:47 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 16:25:47 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 16:25:47 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 16:25:47 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 16:25:47 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 16:25:47 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 16:25:47 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 16:25:47 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 16:25:47 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:25:47 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 16:25:47 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:25:47 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 16:25:47 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:25:47 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 16:25:47 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:25:47 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 16:25:47 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:25:47 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 16:25:47 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:25:47 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 16:25:47 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:25:47 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 16:25:47 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:25:47 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:25:47 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 16:25:47 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 16:25:47 user cinder-volume[84696]: DEBUG cinder.utils [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 41.332(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:25:47 user cinder-volume[84696]: DEBUG cinder.utils [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_7f7b3b2e-e41d-466a-a3c2-e64e3d63b949/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:25:47 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 16:25:47 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (3090157d-b623-4033-8887-eebeeebd838e) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 16:25:47 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 16:25:47 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:18:51Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesTransfersV357Test-Volume-1424534055',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=b900713d-b991-43a5-b403-483dfded7e11,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='1db59dbe212347b8a44e27ebed9e22fb',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:23:27Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:23:27Z,use_quota=True,user_id='ac9545120d3a48bbb11e4cd5e706b53b',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 16:25:47 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:18:51Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesTransfersV357Test-Volume-1424534055',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=b900713d-b991-43a5-b403-483dfded7e11,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='1db59dbe212347b8a44e27ebed9e22fb',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:23:27Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:23:27Z,use_quota=True,user_id='ac9545120d3a48bbb11e4cd5e706b53b',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(b900713d-b991-43a5-b403-483dfded7e11),volume_id=b900713d-b991-43a5-b403-483dfded7e11,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': 'b900713d-b991-43a5-b403-483dfded7e11', 'volume_name': 'volume-b900713d-b991-43a5-b403-483dfded7e11', 'volume_size': 1}} Apr 17 16:25:47 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': 'b900713d-b991-43a5-b403-483dfded7e11', 'volume': {'id': 'b900713d-b991-43a5-b403-483dfded7e11', '_name_id': None, 'ec2_id': None, 'user_id': 'ac9545120d3a48bbb11e4cd5e706b53b', 'project_id': '1db59dbe212347b8a44e27ebed9e22fb', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-VolumesTransfersV357Test-Volume-1424534055', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'admin_metadata': {}, 'glance_metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T16:18:51.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-b900713d-b991-43a5-b403-483dfded7e11', 'name_id': 'b900713d-b991-43a5-b403-483dfded7e11', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumesTransfersV357Test-Volume-1424534055', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '1db59dbe212347b8a44e27ebed9e22fb', 'qos_specs': None, 'reservations': ['b40cb061-d9e6-4c02-bdc0-5d0cb5271144', '8ba69a37-4e7c-4eef-b721-634a940f66f4', 'a799c29a-0d31-4b62-9e2b-b056b451bb3c', 'e510dee0-051c-4a24-a14a-7acbc9a22867'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'ac9545120d3a48bbb11e4cd5e706b53b', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumesTransfersV357Test-Volume-1424534055', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '1db59dbe212347b8a44e27ebed9e22fb', 'qos_specs': None, 'reservations': ['b40cb061-d9e6-4c02-bdc0-5d0cb5271144', '8ba69a37-4e7c-4eef-b721-634a940f66f4', 'a799c29a-0d31-4b62-9e2b-b056b451bb3c', 'e510dee0-051c-4a24-a14a-7acbc9a22867'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'ac9545120d3a48bbb11e4cd5e706b53b', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': 'ac9545120d3a48bbb11e4cd5e706b53b', 'metadata': {}, 'qos_specs': None, 'retry': {'num_attempts': 3, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']}}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(b900713d-b991-43a5-b403-483dfded7e11),volume_id=b900713d-b991-43a5-b403-483dfded7e11,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:18:51Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesTransfersV357Test-Volume-1424534055',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=b900713d-b991-43a5-b403-483dfded7e11,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='1db59dbe212347b8a44e27ebed9e22fb',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:23:27Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:23:27Z,use_quota=True,user_id='ac9545120d3a48bbb11e4cd5e706b53b',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 16:25:47 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:18:51Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesTransfersV357Test-Volume-1424534055',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=b900713d-b991-43a5-b403-483dfded7e11,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='1db59dbe212347b8a44e27ebed9e22fb',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:23:27Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:23:27Z,use_quota=True,user_id='ac9545120d3a48bbb11e4cd5e706b53b',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:18:51Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesTransfersV357Test-Volume-1424534055',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=b900713d-b991-43a5-b403-483dfded7e11,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='1db59dbe212347b8a44e27ebed9e22fb',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:23:27Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:23:27Z,use_quota=True,user_id='ac9545120d3a48bbb11e4cd5e706b53b',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 16:25:47 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 16:25:47 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 16:25:47 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:25:47 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 16:25:47 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:25:47 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 16:25:47 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:25:47 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 16:25:47 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:25:47 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 16:25:47 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:25:47 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 16:25:47 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:25:47 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 16:25:47 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:25:47 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 16:25:47 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:25:47 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:25:47 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 16:25:47 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 16:25:47 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (3090157d-b623-4033-8887-eebeeebd838e) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:25:47 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (3090157d-b623-4033-8887-eebeeebd838e) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:25:47 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (c84d59c4-9a9e-4a00-bc27-e083ef01af0d) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:25:47 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (c84d59c4-9a9e-4a00-bc27-e083ef01af0d) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:25:47 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (db8eb656-182c-49f5-ba09-31b899fd5ada) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:25:47 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (db8eb656-182c-49f5-ba09-31b899fd5ada) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:25:47 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (9ca078d3-8942-4a79-9ad3-de0bd37971ce) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:25:47 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Updating volume b900713d-b991-43a5-b403-483dfded7e11 with {'scheduled_at': datetime.datetime(2023, 4, 17, 16, 25, 47, 273309), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 16:25:47 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Volume b900713d-b991-43a5-b403-483dfded7e11: re-scheduling SchedulerAPI.create_volume attempt 3 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 16:25:47 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Volume b900713d-b991-43a5-b403-483dfded7e11: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 16:25:47 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] deleating volume v_b900713d-b991-43a5-b403-483dfded7e11 {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 16:25:47 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] get snapshots for volume v_b900713d-b991-43a5-b403-483dfded7e11 {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 16:25:47 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_b900713d-b991-43a5-b403-483dfded7e11/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:25:47 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_b900713d-b991-43a5-b403-483dfded7e11/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:25:50 user cinder-volume[84696]: DEBUG cinder.utils [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.974(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:25:50 user cinder-volume[84696]: DEBUG cinder.utils [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_b900713d-b991-43a5-b403-483dfded7e11/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:25:53 user cinder-volume[84696]: DEBUG cinder.utils [None req-d65d2053-22d1-4404-95e3-97728f5ffd95 tempest-GroupsTest-789764105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 72.115(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:25:53 user cinder-volume[84696]: DEBUG cinder.utils [None req-d65d2053-22d1-4404-95e3-97728f5ffd95 tempest-GroupsTest-789764105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:02aba338-fa31-4e46-8fcf-7f86191e44d1/luns/v_02aba338-fa31-4e46-8fcf-7f86191e44d1 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:25:53 user cinder-volume[84696]: DEBUG cinder.utils [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 71.055(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:25:53 user cinder-volume[84696]: DEBUG cinder.utils [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:25:53 user cinder-volume[84696]: DEBUG cinder.utils [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.044(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:25:53 user cinder-volume[84696]: DEBUG cinder.utils [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_b900713d-b991-43a5-b403-483dfded7e11/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:25:56 user cinder-volume[84696]: DEBUG cinder.utils [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 70.947(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:25:56 user cinder-volume[84696]: DEBUG cinder.utils [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:25:59 user cinder-volume[84696]: DEBUG cinder.utils [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.186(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:25:59 user cinder-volume[84696]: DEBUG cinder.utils [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_b900713d-b991-43a5-b403-483dfded7e11/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:26:02 user cinder-volume[84696]: DEBUG cinder.utils [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 140.371(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:26:02 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_02aba338-fa31-4e46-8fcf-7f86191e44d1/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:26:02 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (d38aba9e-fa93-4bd1-9c3a-5884f9099207) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 16:26:02 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (cf9fc825-9cbf-4a8f-8652-ab5cc09e6dfe) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:26:02 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (cf9fc825-9cbf-4a8f-8652-ab5cc09e6dfe) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:26:02 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Flow 'volume_create_manager' (a315cb75-5740-45a4-9e24-b7f1a9e6a279) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 16:26:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 16:26:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 16:26:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 16:26:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 16:26:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 16:26:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 16:26:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 16:26:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 16:26:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 16:26:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 16:26:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 16:26:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 16:26:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 16:26:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 16:26:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 16:26:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 16:26:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 16:26:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 16:26:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 16:26:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 16:26:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 16:26:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 16:26:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 16:26:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 16:26:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:26:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 16:26:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:26:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 16:26:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:26:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 16:26:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:26:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 16:26:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:26:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 16:26:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:26:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 16:26:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:26:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 16:26:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:26:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:26:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 16:26:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 16:26:02 user cinder-volume[84696]: DEBUG cinder.utils [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 140.095(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:26:02 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:26:02 user cinder-volume[84696]: INFO cinder.message.api [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Creating message record for request_id = req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 Apr 17 16:26:02 user cinder-volume[84696]: DEBUG cinder.utils [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 140.403(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:26:02 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_94610127-c8b8-4df4-9e39-a1f41b5a9b98/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:26:02 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (b549076a-a06e-48df-a37e-acd89eb0ce7d) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 16:26:02 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (9c018aa7-8849-4897-ab64-f67667e36734) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:26:02 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (9c018aa7-8849-4897-ab64-f67667e36734) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:26:02 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Flow 'volume_create_manager' (8fe5f406-9d74-4724-8669-269bd8374179) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 16:26:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 16:26:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 16:26:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 16:26:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 16:26:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 16:26:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 16:26:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 16:26:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 16:26:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 16:26:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 16:26:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 16:26:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 16:26:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 16:26:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 16:26:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 16:26:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 16:26:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 16:26:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 16:26:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 16:26:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 16:26:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 16:26:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 16:26:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 16:26:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 16:26:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:26:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 16:26:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:26:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 16:26:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:26:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 16:26:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:26:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 16:26:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:26:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 16:26:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:26:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 16:26:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:26:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 16:26:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:26:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:26:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 16:26:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 16:26:02 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 16:26:02 user cinder-volume[84696]: DEBUG cinder.utils [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 140.198(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:26:02 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:26:02 user cinder-volume[84696]: INFO cinder.message.api [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Creating message record for request_id = req-d3245848-3d3a-4130-8413-3f811df2f47d Apr 17 16:26:02 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (9c89fe55-1bda-427e-b8c2-de5da7f4c4cf) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 16:26:02 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 16:26:02 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:21:25Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-volume-1452364054',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=fe50cd00-dcd4-4b21-a4e9-6f226f8c6ac1,host='user@joviandss-2#joviandss-2',id=02aba338-fa31-4e46-8fcf-7f86191e44d1,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='e10e34b8774b42d8a4cb1f78d72b411d',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:23:42Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:23:42Z,use_quota=True,user_id='e7392ed432ad41279967273f16b4a4c8',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(400bec8d-bd01-4057-8fb8-49a786cd39f1),volume_type_id=400bec8d-bd01-4057-8fb8-49a786cd39f1)}, 'provides': None} Apr 17 16:26:02 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:21:25Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-volume-1452364054',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=fe50cd00-dcd4-4b21-a4e9-6f226f8c6ac1,host='user@joviandss-2#joviandss-2',id=02aba338-fa31-4e46-8fcf-7f86191e44d1,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='e10e34b8774b42d8a4cb1f78d72b411d',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:23:42Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:23:42Z,use_quota=True,user_id='e7392ed432ad41279967273f16b4a4c8',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(400bec8d-bd01-4057-8fb8-49a786cd39f1),volume_type_id=400bec8d-bd01-4057-8fb8-49a786cd39f1), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=fe50cd00-dcd4-4b21-a4e9-6f226f8c6ac1,image_id=None,operation=,resource_backend='user@joviandss-2',snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(02aba338-fa31-4e46-8fcf-7f86191e44d1),volume_id=02aba338-fa31-4e46-8fcf-7f86191e44d1,volume_properties=VolumeProperties,volume_type=VolumeType(400bec8d-bd01-4057-8fb8-49a786cd39f1))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': '02aba338-fa31-4e46-8fcf-7f86191e44d1', 'volume_name': 'volume-02aba338-fa31-4e46-8fcf-7f86191e44d1', 'volume_size': 1}} Apr 17 16:26:02 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'request_spec': {'consistencygroup_id': None, 'group_id': 'fe50cd00-dcd4-4b21-a4e9-6f226f8c6ac1', 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': '02aba338-fa31-4e46-8fcf-7f86191e44d1', 'volume': {'id': '02aba338-fa31-4e46-8fcf-7f86191e44d1', '_name_id': None, 'ec2_id': None, 'user_id': 'e7392ed432ad41279967273f16b4a4c8', 'project_id': 'e10e34b8774b42d8a4cb1f78d72b411d', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-volume-1452364054', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '400bec8d-bd01-4057-8fb8-49a786cd39f1', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': 'fe50cd00-dcd4-4b21-a4e9-6f226f8c6ac1', 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'admin_metadata': {}, 'glance_metadata': {}, 'volume_type': {'id': '400bec8d-bd01-4057-8fb8-49a786cd39f1', 'name': 'tempest-GroupsTest-volume-type-1895690014', 'description': None, 'is_public': True, 'extra_specs': {}, 'qos_specs_id': None, 'created_at': '2023-04-17T16:21:23.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T16:21:24.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-02aba338-fa31-4e46-8fcf-7f86191e44d1', 'name_id': '02aba338-fa31-4e46-8fcf-7f86191e44d1', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '400bec8d-bd01-4057-8fb8-49a786cd39f1', 'name': 'tempest-GroupsTest-volume-type-1895690014', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {}, 'qos_specs_id': None, 'created_at': '2023-04-17T16:21:23.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': 'fe50cd00-dcd4-4b21-a4e9-6f226f8c6ac1', 'display_description': None, 'display_name': 'tempest-volume-1452364054', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': 'e10e34b8774b42d8a4cb1f78d72b411d', 'qos_specs': None, 'reservations': ['f5b0b08e-834b-48d7-bb1a-7efd1002da9b', 'ac966cb4-51d5-4799-b0b6-b3453a218bd0', '77612ba8-fbd6-4ff5-97c8-7ddceb3a7c7a', '101ca44e-fd59-494f-9f9a-2df7868625ff'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'e7392ed432ad41279967273f16b4a4c8', 'volume_type_id': '400bec8d-bd01-4057-8fb8-49a786cd39f1'}, 'resource_backend': 'user@joviandss-2', 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': 'fe50cd00-dcd4-4b21-a4e9-6f226f8c6ac1', 'display_description': None, 'display_name': 'tempest-volume-1452364054', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': 'e10e34b8774b42d8a4cb1f78d72b411d', 'qos_specs': None, 'reservations': ['f5b0b08e-834b-48d7-bb1a-7efd1002da9b', 'ac966cb4-51d5-4799-b0b6-b3453a218bd0', '77612ba8-fbd6-4ff5-97c8-7ddceb3a7c7a', '101ca44e-fd59-494f-9f9a-2df7868625ff'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'e7392ed432ad41279967273f16b4a4c8', 'volume_type_id': '400bec8d-bd01-4057-8fb8-49a786cd39f1'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T16:21:23Z,deleted=False,deleted_at=None,description=None,extra_specs={},id=400bec8d-bd01-4057-8fb8-49a786cd39f1,is_public=True,name='tempest-GroupsTest-volume-type-1895690014',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T16:21:23Z,deleted=False,deleted_at=None,description=None,extra_specs={},id=400bec8d-bd01-4057-8fb8-49a786cd39f1,is_public=True,name='tempest-GroupsTest-volume-type-1895690014',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': 'e7392ed432ad41279967273f16b4a4c8', 'metadata': {}, 'qos_specs': None, 'retry': {'num_attempts': 2, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']}}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=fe50cd00-dcd4-4b21-a4e9-6f226f8c6ac1,image_id=None,operation=,resource_backend='user@joviandss-2',snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(02aba338-fa31-4e46-8fcf-7f86191e44d1),volume_id=02aba338-fa31-4e46-8fcf-7f86191e44d1,volume_properties=VolumeProperties,volume_type=VolumeType(400bec8d-bd01-4057-8fb8-49a786cd39f1)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:21:25Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-volume-1452364054',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=fe50cd00-dcd4-4b21-a4e9-6f226f8c6ac1,host='user@joviandss-2#joviandss-2',id=02aba338-fa31-4e46-8fcf-7f86191e44d1,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='e10e34b8774b42d8a4cb1f78d72b411d',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:23:42Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:23:42Z,use_quota=True,user_id='e7392ed432ad41279967273f16b4a4c8',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(400bec8d-bd01-4057-8fb8-49a786cd39f1),volume_type_id=400bec8d-bd01-4057-8fb8-49a786cd39f1), 'context': }, 'provides': None} Apr 17 16:26:02 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:21:25Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-volume-1452364054',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=fe50cd00-dcd4-4b21-a4e9-6f226f8c6ac1,host='user@joviandss-2#joviandss-2',id=02aba338-fa31-4e46-8fcf-7f86191e44d1,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='e10e34b8774b42d8a4cb1f78d72b411d',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:23:42Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:23:42Z,use_quota=True,user_id='e7392ed432ad41279967273f16b4a4c8',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(400bec8d-bd01-4057-8fb8-49a786cd39f1),volume_type_id=400bec8d-bd01-4057-8fb8-49a786cd39f1)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:21:25Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-volume-1452364054',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=fe50cd00-dcd4-4b21-a4e9-6f226f8c6ac1,host='user@joviandss-2#joviandss-2',id=02aba338-fa31-4e46-8fcf-7f86191e44d1,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='e10e34b8774b42d8a4cb1f78d72b411d',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:23:42Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:23:42Z,use_quota=True,user_id='e7392ed432ad41279967273f16b4a4c8',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(400bec8d-bd01-4057-8fb8-49a786cd39f1),volume_type_id=400bec8d-bd01-4057-8fb8-49a786cd39f1)} Apr 17 16:26:02 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 16:26:02 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 16:26:02 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:26:02 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 16:26:02 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:26:02 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 16:26:02 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:26:02 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 16:26:02 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:26:02 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 16:26:02 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:26:02 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 16:26:02 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:26:02 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 16:26:02 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:26:02 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 16:26:02 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:26:02 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:26:02 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 16:26:02 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 16:26:02 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (9c89fe55-1bda-427e-b8c2-de5da7f4c4cf) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:26:02 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (9c89fe55-1bda-427e-b8c2-de5da7f4c4cf) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:26:02 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (38284650-61ae-4b47-8247-5ffc6530f319) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:26:02 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (38284650-61ae-4b47-8247-5ffc6530f319) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:26:02 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (3bee15e4-cb30-4668-b13f-3da57b102ca6) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:26:02 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (3bee15e4-cb30-4668-b13f-3da57b102ca6) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:26:02 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (176d0698-e62f-4f58-b9a1-b2c5a154dbdc) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:26:02 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Updating volume 02aba338-fa31-4e46-8fcf-7f86191e44d1 with {'scheduled_at': datetime.datetime(2023, 4, 17, 16, 26, 2, 658309), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 16:26:02 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 16:26:02 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Volume 02aba338-fa31-4e46-8fcf-7f86191e44d1: re-scheduling SchedulerAPI.create_volume attempt 2 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 16:26:02 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (9032af0e-850d-4cc9-a380-151fd0dbc25f) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 16:26:02 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 16:26:02 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:21:24Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-AbsoluteLimitsTests-Volume-1339022265',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=94610127-c8b8-4df4-9e39-a1f41b5a9b98,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='2d881d9502184c74b36e1c50acad1f80',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:23:42Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:23:42Z,use_quota=True,user_id='e460a7fb86b4454788ec1950b15434a4',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 16:26:02 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:21:24Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-AbsoluteLimitsTests-Volume-1339022265',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=94610127-c8b8-4df4-9e39-a1f41b5a9b98,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='2d881d9502184c74b36e1c50acad1f80',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:23:42Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:23:42Z,use_quota=True,user_id='e460a7fb86b4454788ec1950b15434a4',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(94610127-c8b8-4df4-9e39-a1f41b5a9b98),volume_id=94610127-c8b8-4df4-9e39-a1f41b5a9b98,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': '94610127-c8b8-4df4-9e39-a1f41b5a9b98', 'volume_name': 'volume-94610127-c8b8-4df4-9e39-a1f41b5a9b98', 'volume_size': 1}} Apr 17 16:26:02 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': '94610127-c8b8-4df4-9e39-a1f41b5a9b98', 'volume': {'id': '94610127-c8b8-4df4-9e39-a1f41b5a9b98', '_name_id': None, 'ec2_id': None, 'user_id': 'e460a7fb86b4454788ec1950b15434a4', 'project_id': '2d881d9502184c74b36e1c50acad1f80', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-AbsoluteLimitsTests-Volume-1339022265', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'admin_metadata': {}, 'glance_metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T16:21:23.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-94610127-c8b8-4df4-9e39-a1f41b5a9b98', 'name_id': '94610127-c8b8-4df4-9e39-a1f41b5a9b98', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-AbsoluteLimitsTests-Volume-1339022265', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '2d881d9502184c74b36e1c50acad1f80', 'qos_specs': None, 'reservations': ['7f441e01-433c-4826-9a44-ec0d9a586e2a', '62cb3d03-7543-4f33-9fda-df3d7d67cb94', 'a069f7b2-c2bc-47d4-85ba-479a14bdc4a4', 'e95f992f-6c15-431a-9f34-693f405be424'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'e460a7fb86b4454788ec1950b15434a4', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-AbsoluteLimitsTests-Volume-1339022265', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '2d881d9502184c74b36e1c50acad1f80', 'qos_specs': None, 'reservations': ['7f441e01-433c-4826-9a44-ec0d9a586e2a', '62cb3d03-7543-4f33-9fda-df3d7d67cb94', 'a069f7b2-c2bc-47d4-85ba-479a14bdc4a4', 'e95f992f-6c15-431a-9f34-693f405be424'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'e460a7fb86b4454788ec1950b15434a4', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': 'e460a7fb86b4454788ec1950b15434a4', 'metadata': {}, 'qos_specs': None, 'retry': {'num_attempts': 2, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']}}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(94610127-c8b8-4df4-9e39-a1f41b5a9b98),volume_id=94610127-c8b8-4df4-9e39-a1f41b5a9b98,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:21:24Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-AbsoluteLimitsTests-Volume-1339022265',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=94610127-c8b8-4df4-9e39-a1f41b5a9b98,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='2d881d9502184c74b36e1c50acad1f80',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:23:42Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:23:42Z,use_quota=True,user_id='e460a7fb86b4454788ec1950b15434a4',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 16:26:02 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:21:24Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-AbsoluteLimitsTests-Volume-1339022265',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=94610127-c8b8-4df4-9e39-a1f41b5a9b98,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='2d881d9502184c74b36e1c50acad1f80',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:23:42Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:23:42Z,use_quota=True,user_id='e460a7fb86b4454788ec1950b15434a4',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:21:24Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-AbsoluteLimitsTests-Volume-1339022265',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=94610127-c8b8-4df4-9e39-a1f41b5a9b98,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='2d881d9502184c74b36e1c50acad1f80',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:23:42Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:23:42Z,use_quota=True,user_id='e460a7fb86b4454788ec1950b15434a4',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 16:26:02 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 16:26:02 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 16:26:02 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:26:02 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 16:26:02 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:26:02 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 16:26:02 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:26:02 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 16:26:02 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:26:02 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 16:26:02 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:26:02 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 16:26:02 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:26:02 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 16:26:02 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:26:02 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 16:26:02 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:26:02 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:26:02 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 16:26:02 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 16:26:02 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (9032af0e-850d-4cc9-a380-151fd0dbc25f) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:26:02 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (9032af0e-850d-4cc9-a380-151fd0dbc25f) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:26:02 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (3f9305bf-d9f1-4be2-b50e-79d427b57764) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:26:02 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (3f9305bf-d9f1-4be2-b50e-79d427b57764) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:26:02 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (2680ff9a-2c2b-4a73-abe7-a4509e8d26b4) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:26:02 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (2680ff9a-2c2b-4a73-abe7-a4509e8d26b4) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:26:02 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (fd9eb00d-6c37-455d-a77f-5284073a5df9) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:26:02 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Updating volume 94610127-c8b8-4df4-9e39-a1f41b5a9b98 with {'scheduled_at': datetime.datetime(2023, 4, 17, 16, 26, 2, 718293), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 16:26:02 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Volume 94610127-c8b8-4df4-9e39-a1f41b5a9b98: re-scheduling SchedulerAPI.create_volume attempt 2 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 16:26:02 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Volume 02aba338-fa31-4e46-8fcf-7f86191e44d1: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 16:26:02 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] deleating volume v_02aba338-fa31-4e46-8fcf-7f86191e44d1 {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 16:26:02 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] get snapshots for volume v_02aba338-fa31-4e46-8fcf-7f86191e44d1 {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 16:26:02 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_02aba338-fa31-4e46-8fcf-7f86191e44d1/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:26:02 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_02aba338-fa31-4e46-8fcf-7f86191e44d1/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:26:02 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Volume 94610127-c8b8-4df4-9e39-a1f41b5a9b98: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 16:26:02 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] deleating volume v_94610127-c8b8-4df4-9e39-a1f41b5a9b98 {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 16:26:02 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] get snapshots for volume v_94610127-c8b8-4df4-9e39-a1f41b5a9b98 {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 16:26:02 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_94610127-c8b8-4df4-9e39-a1f41b5a9b98/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:26:02 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_94610127-c8b8-4df4-9e39-a1f41b5a9b98/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:26:02 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Volume reschedule parameters: True retry: {'num_attempts': 3, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 16:26:02 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Flow 'volume_create_manager' (ae635d45-5956-4347-8615-a522c49f73fd) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 16:26:02 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (cac58a2b-56c0-4720-bcba-d46d30f9eb16) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:26:02 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Volume reschedule parameters: True retry: {'num_attempts': 3, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 16:26:02 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Flow 'volume_create_manager' (eee06777-deee-475c-a4c0-0678fb763ca9) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 16:26:03 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (f8f3a404-fc9a-498c-afaf-d9ab99c14c6c) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:26:03 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (cac58a2b-56c0-4720-bcba-d46d30f9eb16) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:21:25Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-volume-1452364054',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=fe50cd00-dcd4-4b21-a4e9-6f226f8c6ac1,host='user@joviandss-2#joviandss-2',id=02aba338-fa31-4e46-8fcf-7f86191e44d1,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='e10e34b8774b42d8a4cb1f78d72b411d',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:26:03Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='deleting',terminated_at=None,updated_at=2023-04-17T16:26:03Z,use_quota=True,user_id='e7392ed432ad41279967273f16b4a4c8',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(400bec8d-bd01-4057-8fb8-49a786cd39f1),volume_type_id=400bec8d-bd01-4057-8fb8-49a786cd39f1)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:26:03 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (64080c9c-bb96-4290-97cf-4bc092461a3e) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:26:03 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (64080c9c-bb96-4290-97cf-4bc092461a3e) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:26:03 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (ad539e83-7287-4a7b-927c-fe8b69e98df5) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:26:03 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (ad539e83-7287-4a7b-927c-fe8b69e98df5) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'deleting', 'type': 'raw', 'volume_id': '02aba338-fa31-4e46-8fcf-7f86191e44d1', 'volume_name': 'volume-02aba338-fa31-4e46-8fcf-7f86191e44d1', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:26:03 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (569fdc18-4a98-49fd-b912-cfe5e9f79489) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:26:03 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (569fdc18-4a98-49fd-b912-cfe5e9f79489) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:26:03 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (7ce8a6b4-0b75-4b62-b424-a4503fa7c4c9) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:26:03 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Volume 02aba338-fa31-4e46-8fcf-7f86191e44d1: being created as raw with specification: {'status': 'deleting', 'volume_name': 'volume-02aba338-fa31-4e46-8fcf-7f86191e44d1', 'volume_size': 1} Apr 17 16:26:03 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] creating volume v_02aba338-fa31-4e46-8fcf-7f86191e44d1. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 16:26:03 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] create volume {'name': 'v_02aba338-fa31-4e46-8fcf-7f86191e44d1', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 16:26:03 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:26:03 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:26:03 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (f8f3a404-fc9a-498c-afaf-d9ab99c14c6c) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:21:24Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-AbsoluteLimitsTests-Volume-1339022265',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=94610127-c8b8-4df4-9e39-a1f41b5a9b98,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='2d881d9502184c74b36e1c50acad1f80',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:26:03Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:26:03Z,use_quota=True,user_id='e460a7fb86b4454788ec1950b15434a4',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:26:03 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (66979d03-bf50-4ec4-a32e-e4b96a752e54) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:26:03 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (66979d03-bf50-4ec4-a32e-e4b96a752e54) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:26:03 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (71ccce28-3f6c-4370-b660-37e5ae648a21) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:26:03 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (71ccce28-3f6c-4370-b660-37e5ae648a21) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': '94610127-c8b8-4df4-9e39-a1f41b5a9b98', 'volume_name': 'volume-94610127-c8b8-4df4-9e39-a1f41b5a9b98', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:26:03 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (735b1cd2-114f-40a4-98a7-9ea866b1163d) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:26:03 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (735b1cd2-114f-40a4-98a7-9ea866b1163d) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:26:03 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (665c1787-9f99-4a5c-9b6c-74f1f587dfd8) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:26:03 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Volume 94610127-c8b8-4df4-9e39-a1f41b5a9b98: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-94610127-c8b8-4df4-9e39-a1f41b5a9b98', 'volume_size': 1} Apr 17 16:26:03 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] creating volume v_94610127-c8b8-4df4-9e39-a1f41b5a9b98. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 16:26:03 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] create volume {'name': 'v_94610127-c8b8-4df4-9e39-a1f41b5a9b98', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 16:26:03 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:26:03 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:26:05 user cinder-volume[84696]: DEBUG cinder.utils [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.891(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:26:05 user cinder-volume[84696]: DEBUG cinder.utils [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_02aba338-fa31-4e46-8fcf-7f86191e44d1/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:26:05 user cinder-volume[84696]: DEBUG cinder.utils [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.542(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:26:05 user cinder-volume[84696]: DEBUG cinder.utils [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:26:05 user cinder-volume[84696]: DEBUG cinder.utils [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.589(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:26:05 user cinder-volume[84696]: DEBUG cinder.utils [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:26:05 user cinder-volume[84696]: DEBUG cinder.utils [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.881(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:26:05 user cinder-volume[84696]: DEBUG cinder.utils [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_94610127-c8b8-4df4-9e39-a1f41b5a9b98/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:26:08 user cinder-volume[84696]: DEBUG cinder.utils [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.612(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:26:08 user cinder-volume[84696]: DEBUG cinder.utils [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:26:08 user cinder-volume[84696]: DEBUG cinder.utils [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.403(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:26:08 user cinder-volume[84696]: DEBUG cinder.utils [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_b900713d-b991-43a5-b403-483dfded7e11/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:26:08 user cinder-volume[84696]: DEBUG cinder.utils [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.950(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:26:08 user cinder-volume[84696]: DEBUG cinder.utils [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_94610127-c8b8-4df4-9e39-a1f41b5a9b98/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:26:08 user cinder-volume[84696]: DEBUG cinder.utils [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.966(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:26:08 user cinder-volume[84696]: DEBUG cinder.utils [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_02aba338-fa31-4e46-8fcf-7f86191e44d1/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:26:08 user cinder-volume[84696]: DEBUG cinder.utils [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.661(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:26:08 user cinder-volume[84696]: DEBUG cinder.utils [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:26:14 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=16,cinder:UPDATE=7,cinder:INSERT=3 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:26:15 user cinder-volume[84696]: DEBUG cinder.utils [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 11.916(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:26:15 user cinder-volume[84696]: DEBUG cinder.utils [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:26:15 user cinder-volume[84696]: DEBUG cinder.utils [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 11.962(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:26:15 user cinder-volume[84696]: DEBUG cinder.utils [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:26:15 user cinder-volume[84696]: DEBUG cinder.utils [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.254(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:26:15 user cinder-volume[84696]: DEBUG cinder.utils [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_94610127-c8b8-4df4-9e39-a1f41b5a9b98/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:26:15 user cinder-volume[84696]: DEBUG cinder.utils [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 75.247(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:26:15 user cinder-volume[84696]: DEBUG cinder.utils [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_c9fa0a17-69d4-40d0-9d65-af6ae349cfd8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:26:15 user cinder-volume[84696]: DEBUG cinder.utils [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.271(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:26:15 user cinder-volume[84696]: DEBUG cinder.utils [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_02aba338-fa31-4e46-8fcf-7f86191e44d1/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:26:22 user cinder-volume[84696]: DEBUG cinder.utils [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 76.127(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:26:22 user cinder-volume[84696]: DEBUG cinder.utils [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_7f7b3b2e-e41d-466a-a3c2-e64e3d63b949/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:26:24 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=19,cinder:UPDATE=5,cinder:INSERT=3 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:26:25 user cinder-volume[84696]: DEBUG cinder.utils [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.996(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:26:25 user cinder-volume[84696]: DEBUG cinder.utils [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:26:25 user cinder-volume[84696]: DEBUG cinder.utils [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 22.334(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:26:25 user cinder-volume[84696]: DEBUG cinder.utils [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_94610127-c8b8-4df4-9e39-a1f41b5a9b98/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:26:25 user cinder-volume[84696]: DEBUG cinder.utils [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 22.044(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:26:25 user cinder-volume[84696]: DEBUG cinder.utils [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:26:25 user cinder-volume[84696]: DEBUG cinder.utils [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 37.789(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:26:25 user cinder-volume[84696]: DEBUG cinder.utils [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_b900713d-b991-43a5-b403-483dfded7e11/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:26:25 user cinder-volume[84696]: DEBUG cinder.utils [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 22.351(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:26:25 user cinder-volume[84696]: DEBUG cinder.utils [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_02aba338-fa31-4e46-8fcf-7f86191e44d1/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:26:42 user cinder-volume[84696]: DEBUG cinder.utils [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 38.924(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:26:42 user cinder-volume[84696]: DEBUG cinder.utils [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:26:42 user cinder-volume[84696]: DEBUG cinder.utils [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 38.971(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:26:42 user cinder-volume[84696]: DEBUG cinder.utils [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:26:42 user cinder-volume[84696]: DEBUG cinder.utils [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.263(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:26:42 user cinder-volume[84696]: DEBUG cinder.utils [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_94610127-c8b8-4df4-9e39-a1f41b5a9b98/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:26:42 user cinder-volume[84696]: DEBUG cinder.utils [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.279(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:26:42 user cinder-volume[84696]: DEBUG cinder.utils [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_02aba338-fa31-4e46-8fcf-7f86191e44d1/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:26:44 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=5,cinder:UPDATE=2 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:26:45 user cinder-volume[84696]: DEBUG cinder.utils [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 136.055(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:26:45 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_2e8580be-c7f2-4d3a-9007-d192a5b029b3/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:26:45 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (242c0d4b-5c14-4989-a5d0-47d010c8ba81) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 16:26:45 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (08f1b9ad-1bbb-446c-85cd-796b0907142d) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:26:45 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (08f1b9ad-1bbb-446c-85cd-796b0907142d) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:26:45 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Flow 'volume_create_manager' (481e1710-a15b-499f-8369-3ea87288fff6) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 16:26:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 16:26:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 16:26:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 16:26:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 16:26:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 16:26:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 16:26:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 16:26:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 16:26:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 16:26:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 16:26:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 16:26:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 16:26:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 16:26:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 16:26:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 16:26:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 16:26:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 16:26:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 16:26:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 16:26:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 16:26:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 16:26:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 16:26:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 16:26:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 16:26:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:26:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 16:26:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:26:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 16:26:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:26:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 16:26:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:26:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 16:26:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:26:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 16:26:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:26:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 16:26:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:26:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 16:26:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:26:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:26:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 16:26:45 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 16:26:45 user cinder-volume[84696]: DEBUG cinder.utils [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 135.899(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:26:45 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:26:45 user cinder-volume[84696]: INFO cinder.message.api [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Creating message record for request_id = req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 Apr 17 16:26:45 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 16:26:45 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (a933103a-5b24-45a6-8021-a6eb82c39cfc) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 16:26:45 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 16:26:45 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:22:11Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesNegativeTest-Volume-375402817',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=2e8580be-c7f2-4d3a-9007-d192a5b029b3,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='935190a703294690b79c182d789f31e4',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:24:29Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:24:29Z,use_quota=True,user_id='9296aa0c28024371bac0c53928bff421',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 16:26:45 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:22:11Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesNegativeTest-Volume-375402817',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=2e8580be-c7f2-4d3a-9007-d192a5b029b3,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='935190a703294690b79c182d789f31e4',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:24:29Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:24:29Z,use_quota=True,user_id='9296aa0c28024371bac0c53928bff421',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(2e8580be-c7f2-4d3a-9007-d192a5b029b3),volume_id=2e8580be-c7f2-4d3a-9007-d192a5b029b3,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': '2e8580be-c7f2-4d3a-9007-d192a5b029b3', 'volume_name': 'volume-2e8580be-c7f2-4d3a-9007-d192a5b029b3', 'volume_size': 1}} Apr 17 16:26:45 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': '2e8580be-c7f2-4d3a-9007-d192a5b029b3', 'volume': {'id': '2e8580be-c7f2-4d3a-9007-d192a5b029b3', '_name_id': None, 'ec2_id': None, 'user_id': '9296aa0c28024371bac0c53928bff421', 'project_id': '935190a703294690b79c182d789f31e4', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-VolumesNegativeTest-Volume-375402817', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'admin_metadata': {}, 'glance_metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T16:22:11.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-2e8580be-c7f2-4d3a-9007-d192a5b029b3', 'name_id': '2e8580be-c7f2-4d3a-9007-d192a5b029b3', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumesNegativeTest-Volume-375402817', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '935190a703294690b79c182d789f31e4', 'qos_specs': None, 'reservations': ['8302a15a-cac9-4777-aeda-3cd0f18e59f0', '4f81147c-1b8e-4140-ae06-353eac50d673', '0d992e26-ac75-42fe-82b4-74e2842f6518', 'ce106309-de8e-43ad-89d6-aaedb5c859b5'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': '9296aa0c28024371bac0c53928bff421', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumesNegativeTest-Volume-375402817', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '935190a703294690b79c182d789f31e4', 'qos_specs': None, 'reservations': ['8302a15a-cac9-4777-aeda-3cd0f18e59f0', '4f81147c-1b8e-4140-ae06-353eac50d673', '0d992e26-ac75-42fe-82b4-74e2842f6518', 'ce106309-de8e-43ad-89d6-aaedb5c859b5'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': '9296aa0c28024371bac0c53928bff421', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': '9296aa0c28024371bac0c53928bff421', 'metadata': {}, 'qos_specs': None, 'retry': {'num_attempts': 2, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']}}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(2e8580be-c7f2-4d3a-9007-d192a5b029b3),volume_id=2e8580be-c7f2-4d3a-9007-d192a5b029b3,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:22:11Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesNegativeTest-Volume-375402817',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=2e8580be-c7f2-4d3a-9007-d192a5b029b3,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='935190a703294690b79c182d789f31e4',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:24:29Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:24:29Z,use_quota=True,user_id='9296aa0c28024371bac0c53928bff421',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 16:26:45 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:22:11Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesNegativeTest-Volume-375402817',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=2e8580be-c7f2-4d3a-9007-d192a5b029b3,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='935190a703294690b79c182d789f31e4',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:24:29Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:24:29Z,use_quota=True,user_id='9296aa0c28024371bac0c53928bff421',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:22:11Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesNegativeTest-Volume-375402817',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=2e8580be-c7f2-4d3a-9007-d192a5b029b3,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='935190a703294690b79c182d789f31e4',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:24:29Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:24:29Z,use_quota=True,user_id='9296aa0c28024371bac0c53928bff421',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 16:26:45 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 16:26:45 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 16:26:45 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:26:45 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 16:26:45 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:26:45 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 16:26:45 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:26:45 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 16:26:45 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:26:45 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 16:26:45 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:26:45 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 16:26:45 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:26:45 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 16:26:45 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:26:45 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 16:26:45 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:26:45 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:26:45 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 16:26:45 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 16:26:45 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (a933103a-5b24-45a6-8021-a6eb82c39cfc) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:26:45 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (a933103a-5b24-45a6-8021-a6eb82c39cfc) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:26:45 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (926e76f2-7bac-451f-af08-c4f3134c3842) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:26:45 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (926e76f2-7bac-451f-af08-c4f3134c3842) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:26:45 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (3ef9372d-af18-41ff-b9ba-6ab11ff2ec8a) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:26:45 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (3ef9372d-af18-41ff-b9ba-6ab11ff2ec8a) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:26:45 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (0c39da14-5af1-4383-938b-31d3f459502d) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:26:45 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Updating volume 2e8580be-c7f2-4d3a-9007-d192a5b029b3 with {'scheduled_at': datetime.datetime(2023, 4, 17, 16, 26, 45, 160134), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 16:26:45 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Volume 2e8580be-c7f2-4d3a-9007-d192a5b029b3: re-scheduling SchedulerAPI.create_volume attempt 2 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 16:26:45 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Volume 2e8580be-c7f2-4d3a-9007-d192a5b029b3: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 16:26:45 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] deleating volume v_2e8580be-c7f2-4d3a-9007-d192a5b029b3 {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 16:26:45 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] get snapshots for volume v_2e8580be-c7f2-4d3a-9007-d192a5b029b3 {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 16:26:45 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_2e8580be-c7f2-4d3a-9007-d192a5b029b3/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:26:45 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_2e8580be-c7f2-4d3a-9007-d192a5b029b3/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:26:45 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Volume reschedule parameters: True retry: {'num_attempts': 3, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 16:26:45 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Flow 'volume_create_manager' (b5c76848-fd44-4b33-80b0-8ddd572ad41b) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 16:26:45 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (f2c6c15e-04e0-4e2c-876e-9a3ee61201b2) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:26:45 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (f2c6c15e-04e0-4e2c-876e-9a3ee61201b2) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:22:11Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesNegativeTest-Volume-375402817',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=2e8580be-c7f2-4d3a-9007-d192a5b029b3,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='935190a703294690b79c182d789f31e4',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:26:45Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:26:45Z,use_quota=True,user_id='9296aa0c28024371bac0c53928bff421',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:26:45 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (ab9fc0d2-624c-4bd2-a9aa-de04049a75b9) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:26:45 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (ab9fc0d2-624c-4bd2-a9aa-de04049a75b9) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:26:45 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (bc2fffc6-0dc8-4335-98cc-5934cc0c66fc) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:26:45 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (bc2fffc6-0dc8-4335-98cc-5934cc0c66fc) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': '2e8580be-c7f2-4d3a-9007-d192a5b029b3', 'volume_name': 'volume-2e8580be-c7f2-4d3a-9007-d192a5b029b3', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:26:45 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (eea81c23-06ad-4cdf-a8d6-9fa401768843) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:26:45 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (eea81c23-06ad-4cdf-a8d6-9fa401768843) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:26:45 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (927390f1-640a-47a6-8f48-6587ffb094a4) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:26:45 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Volume 2e8580be-c7f2-4d3a-9007-d192a5b029b3: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-2e8580be-c7f2-4d3a-9007-d192a5b029b3', 'volume_size': 1} Apr 17 16:26:45 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] creating volume v_2e8580be-c7f2-4d3a-9007-d192a5b029b3. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 16:26:45 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] create volume {'name': 'v_2e8580be-c7f2-4d3a-9007-d192a5b029b3', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 16:26:45 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:26:45 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:26:48 user cinder-volume[84696]: DEBUG cinder.utils [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.908(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:26:48 user cinder-volume[84696]: DEBUG cinder.utils [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:26:48 user cinder-volume[84696]: DEBUG cinder.utils [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 3.067(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:26:48 user cinder-volume[84696]: DEBUG cinder.utils [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_2e8580be-c7f2-4d3a-9007-d192a5b029b3/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:26:48 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 136.254(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:26:48 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:26:48 user cinder-volume[84696]: ERROR oslo_service.periodic_task [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Error during VolumeManager.publish_service_capabilities: TypeError: 'NoneType' object is not subscriptable Apr 17 16:26:48 user cinder-volume[84696]: ERROR oslo_service.periodic_task Traceback (most recent call last): Apr 17 16:26:48 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/usr/local/lib/python3.10/dist-packages/oslo_service/periodic_task.py", line 216, in run_periodic_tasks Apr 17 16:26:48 user cinder-volume[84696]: ERROR oslo_service.periodic_task task(self, context) Apr 17 16:26:48 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/manager.py", line 2876, in publish_service_capabilities Apr 17 16:26:48 user cinder-volume[84696]: ERROR oslo_service.periodic_task self._report_driver_status(context) Apr 17 16:26:48 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/manager.py", line 2755, in _report_driver_status Apr 17 16:26:48 user cinder-volume[84696]: ERROR oslo_service.periodic_task volume_stats = get_stats() Apr 17 16:26:48 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/usr/local/lib/python3.10/dist-packages/oslo_utils/timeutils.py", line 319, in wrapper Apr 17 16:26:48 user cinder-volume[84696]: ERROR oslo_service.periodic_task result = func(*args, **kwargs) Apr 17 16:26:48 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/manager.py", line 2753, in get_stats Apr 17 16:26:48 user cinder-volume[84696]: ERROR oslo_service.periodic_task return self.driver.get_volume_stats(refresh=True) Apr 17 16:26:48 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/driver.py", line 666, in get_volume_stats Apr 17 16:26:48 user cinder-volume[84696]: ERROR oslo_service.periodic_task self._update_volume_stats() Apr 17 16:26:48 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 726, in _update_volume_stats Apr 17 16:26:48 user cinder-volume[84696]: ERROR oslo_service.periodic_task pool_stats = self.ra.get_pool_stats() Apr 17 16:26:48 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 897, in get_pool_stats Apr 17 16:26:48 user cinder-volume[84696]: ERROR oslo_service.periodic_task if not resp["error"] and resp["code"] == 200: Apr 17 16:26:48 user cinder-volume[84696]: ERROR oslo_service.periodic_task TypeError: 'NoneType' object is not subscriptable Apr 17 16:26:48 user cinder-volume[84696]: ERROR oslo_service.periodic_task Apr 17 16:26:48 user cinder-volume[84696]: WARNING oslo.service.loopingcall [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Function 'cinder.service.Service.periodic_tasks' run outlasted interval by 76.26 sec Apr 17 16:26:48 user cinder-volume[84696]: DEBUG oslo_service.periodic_task [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Running periodic task VolumeManager.publish_service_capabilities {{(pid=84696) run_periodic_tasks /usr/local/lib/python3.10/dist-packages/oslo_service/periodic_task.py:210}} Apr 17 16:26:48 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Updating volume stats {{(pid=84696) _update_volume_stats /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:724}} Apr 17 16:26:48 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Get pool Pool-0 fsprops {{(pid=84696) get_pool_stats /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:894}} Apr 17 16:26:48 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:26:48 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:26:51 user cinder-volume[84696]: DEBUG cinder.utils [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.137(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:26:51 user cinder-volume[84696]: DEBUG cinder.utils [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_2e8580be-c7f2-4d3a-9007-d192a5b029b3/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:26:51 user cinder-volume[84696]: DEBUG cinder.utils [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.981(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:26:51 user cinder-volume[84696]: DEBUG cinder.utils [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:26:51 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 3.064(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:26:51 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:26:54 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.134(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:26:54 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:26:54 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=11,cinder:UPDATE=3 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:26:57 user cinder-volume[84696]: DEBUG cinder.utils [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 135.186(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:26:57 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:26:57 user cinder-volume[84696]: INFO cinder.message.api [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Creating message record for request_id = req-8d765f93-bce0-4815-ac07-22b8b3bd2556 Apr 17 16:26:57 user cinder-volume[84696]: DEBUG cinder.utils [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.134(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:26:57 user cinder-volume[84696]: DEBUG cinder.utils [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:26:57 user cinder-volume[84696]: DEBUG cinder.utils [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.292(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:26:57 user cinder-volume[84696]: DEBUG cinder.utils [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_2e8580be-c7f2-4d3a-9007-d192a5b029b3/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:26:57 user cinder-volume[84696]: DEBUG cinder.utils [None req-d65d2053-22d1-4404-95e3-97728f5ffd95 tempest-GroupsTest-789764105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 136.256(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:26:57 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-d65d2053-22d1-4404-95e3-97728f5ffd95 tempest-GroupsTest-789764105 None] Geting None from DELETE to https://172.16.0.220:82/api/v3/pools/Pool-0/san/iscsi/targets/iqn.2010-10.org.openstack:02aba338-fa31-4e46-8fcf-7f86191e44d1/luns/v_02aba338-fa31-4e46-8fcf-7f86191e44d1 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:26:57 user cinder-volume[84696]: DEBUG cinder.utils [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 70.184(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:26:57 user cinder-volume[84696]: DEBUG cinder.utils [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_b900713d-b991-43a5-b403-483dfded7e11/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:26:57 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-d65d2053-22d1-4404-95e3-97728f5ffd95 tempest-GroupsTest-789764105 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 16:26:57 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (07a79a30-0bde-440f-8526-14c0dc2e9d5a) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 16:26:57 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 16:26:57 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:24:42Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesDeleteCascade-Volume-1549123617',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=53029faa-77b2-467d-8a64-eb5ef129e41c,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='abbfd52254f546b0bcfe289884d77f49',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:24:42Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:24:42Z,use_quota=True,user_id='6309fcbea41c471184457d276d32db73',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 16:26:57 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:24:42Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesDeleteCascade-Volume-1549123617',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=53029faa-77b2-467d-8a64-eb5ef129e41c,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='abbfd52254f546b0bcfe289884d77f49',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:24:42Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:24:42Z,use_quota=True,user_id='6309fcbea41c471184457d276d32db73',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(53029faa-77b2-467d-8a64-eb5ef129e41c),volume_id=53029faa-77b2-467d-8a64-eb5ef129e41c,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': '53029faa-77b2-467d-8a64-eb5ef129e41c', 'volume_name': 'volume-53029faa-77b2-467d-8a64-eb5ef129e41c', 'volume_size': 1}} Apr 17 16:26:57 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'retry': {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']}, 'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': '53029faa-77b2-467d-8a64-eb5ef129e41c', 'volume': {'id': '53029faa-77b2-467d-8a64-eb5ef129e41c', '_name_id': None, 'ec2_id': None, 'user_id': '6309fcbea41c471184457d276d32db73', 'project_id': 'abbfd52254f546b0bcfe289884d77f49', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-VolumesDeleteCascade-Volume-1549123617', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T16:24:42.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-53029faa-77b2-467d-8a64-eb5ef129e41c', 'name_id': '53029faa-77b2-467d-8a64-eb5ef129e41c', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumesDeleteCascade-Volume-1549123617', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': 'abbfd52254f546b0bcfe289884d77f49', 'qos_specs': None, 'reservations': ['15abf3ac-c420-4557-ab29-de463d495550', '4070e169-cef7-4670-80f8-a805062b3ed7', '598ac100-d775-4c42-82f1-6a2eb9c17061', 'c035551f-9406-4ac7-a6e8-e732d18bff88'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': '6309fcbea41c471184457d276d32db73', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumesDeleteCascade-Volume-1549123617', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': 'abbfd52254f546b0bcfe289884d77f49', 'qos_specs': None, 'reservations': ['15abf3ac-c420-4557-ab29-de463d495550', '4070e169-cef7-4670-80f8-a805062b3ed7', '598ac100-d775-4c42-82f1-6a2eb9c17061', 'c035551f-9406-4ac7-a6e8-e732d18bff88'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': '6309fcbea41c471184457d276d32db73', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': '6309fcbea41c471184457d276d32db73', 'metadata': {}, 'qos_specs': None}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(53029faa-77b2-467d-8a64-eb5ef129e41c),volume_id=53029faa-77b2-467d-8a64-eb5ef129e41c,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:24:42Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesDeleteCascade-Volume-1549123617',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=53029faa-77b2-467d-8a64-eb5ef129e41c,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='abbfd52254f546b0bcfe289884d77f49',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:24:42Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:24:42Z,use_quota=True,user_id='6309fcbea41c471184457d276d32db73',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 16:26:57 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:24:42Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesDeleteCascade-Volume-1549123617',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=53029faa-77b2-467d-8a64-eb5ef129e41c,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='abbfd52254f546b0bcfe289884d77f49',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:24:42Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:24:42Z,use_quota=True,user_id='6309fcbea41c471184457d276d32db73',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:24:42Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesDeleteCascade-Volume-1549123617',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=53029faa-77b2-467d-8a64-eb5ef129e41c,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='abbfd52254f546b0bcfe289884d77f49',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:24:42Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:24:42Z,use_quota=True,user_id='6309fcbea41c471184457d276d32db73',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 16:26:57 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 16:26:57 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 16:26:57 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:26:57 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 16:26:57 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:26:57 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 16:26:57 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:26:57 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 16:26:57 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:26:57 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 16:26:57 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:26:57 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 16:26:57 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:26:57 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 16:26:57 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:26:57 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 16:26:57 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:26:57 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:26:57 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 16:26:57 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 16:26:57 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (07a79a30-0bde-440f-8526-14c0dc2e9d5a) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:26:57 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (07a79a30-0bde-440f-8526-14c0dc2e9d5a) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:26:57 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (a8c72a42-2994-487a-b610-5d539ef74dae) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:26:57 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (a8c72a42-2994-487a-b610-5d539ef74dae) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:26:57 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (917b9f19-733f-4891-979d-9d4adf0e89d2) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:26:57 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (917b9f19-733f-4891-979d-9d4adf0e89d2) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:26:57 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (d77b789b-f220-432a-8116-dd04f7251a2b) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:26:57 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Updating volume 53029faa-77b2-467d-8a64-eb5ef129e41c with {'scheduled_at': datetime.datetime(2023, 4, 17, 16, 26, 57, 560761), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 16:26:57 user cinder-volume[84696]: ERROR cinder.volume.manager [None req-d65d2053-22d1-4404-95e3-97728f5ffd95 tempest-GroupsTest-789764105 None] Delete group failed. Apr 17 16:26:57 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Volume 53029faa-77b2-467d-8a64-eb5ef129e41c: re-scheduling SchedulerAPI.create_volume attempt 1 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 16:26:57 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-d65d2053-22d1-4404-95e3-97728f5ffd95 tempest-GroupsTest-789764105 None] Exception during message handling: cinder.exception.VolumeDriverException: Volume driver reported an error: Delete group failed. Apr 17 16:26:57 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 16:26:57 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 16:26:57 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 16:26:57 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 16:26:57 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 16:26:57 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 16:26:57 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 16:26:57 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 3805, in delete_group Apr 17 16:26:57 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 16:26:57 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:26:57 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 16:26:57 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:26:57 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 16:26:57 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 3799, in delete_group Apr 17 16:26:57 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise exception.VolumeDriverException(message=msg) Apr 17 16:26:57 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server cinder.exception.VolumeDriverException: Volume driver reported an error: Delete group failed. Apr 17 16:26:57 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 16:26:57 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Volume 53029faa-77b2-467d-8a64-eb5ef129e41c: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 16:26:57 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] deleating volume v_53029faa-77b2-467d-8a64-eb5ef129e41c {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 16:26:57 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] get snapshots for volume v_53029faa-77b2-467d-8a64-eb5ef129e41c {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 16:26:57 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_53029faa-77b2-467d-8a64-eb5ef129e41c/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:26:57 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_53029faa-77b2-467d-8a64-eb5ef129e41c/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:26:57 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Volume reschedule parameters: True retry: {'num_attempts': 2, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 16:26:57 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Flow 'volume_create_manager' (4e5cd157-86df-4a33-91e8-ec5f2f8fba8c) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 16:26:57 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (2a596980-4c2a-4f58-80d8-0259ceffa0aa) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:26:57 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (2a596980-4c2a-4f58-80d8-0259ceffa0aa) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:24:42Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesDeleteCascade-Volume-1549123617',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=53029faa-77b2-467d-8a64-eb5ef129e41c,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='abbfd52254f546b0bcfe289884d77f49',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:26:58Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:26:58Z,use_quota=True,user_id='6309fcbea41c471184457d276d32db73',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:26:57 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (33495df2-1373-45cd-b3ea-3fcb107e4d47) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:26:57 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (33495df2-1373-45cd-b3ea-3fcb107e4d47) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:26:57 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (d9c097c3-7f72-4214-baa4-1a8f17ee01e1) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:26:57 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (d9c097c3-7f72-4214-baa4-1a8f17ee01e1) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': '53029faa-77b2-467d-8a64-eb5ef129e41c', 'volume_name': 'volume-53029faa-77b2-467d-8a64-eb5ef129e41c', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:26:57 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (8235445a-ab69-4e76-b3ab-77ed3c2566ec) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:26:57 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (8235445a-ab69-4e76-b3ab-77ed3c2566ec) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:26:57 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (17e59cd1-25dc-4cf1-a81d-545228670599) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:26:57 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Volume 53029faa-77b2-467d-8a64-eb5ef129e41c: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-53029faa-77b2-467d-8a64-eb5ef129e41c', 'volume_size': 1} Apr 17 16:26:57 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] creating volume v_53029faa-77b2-467d-8a64-eb5ef129e41c. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 16:26:57 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] create volume {'name': 'v_53029faa-77b2-467d-8a64-eb5ef129e41c', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 16:26:57 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:26:57 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:27:00 user cinder-volume[84696]: DEBUG cinder.utils [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.949(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:27:00 user cinder-volume[84696]: DEBUG cinder.utils [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_53029faa-77b2-467d-8a64-eb5ef129e41c/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:27:00 user cinder-volume[84696]: DEBUG cinder.utils [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.793(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:27:00 user cinder-volume[84696]: DEBUG cinder.utils [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:27:00 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.285(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:27:00 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:27:00 user cinder-volume[84696]: DEBUG cinder.utils [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 135.079(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:27:00 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:27:00 user cinder-volume[84696]: INFO cinder.message.api [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Creating message record for request_id = req-61f60e22-802a-4068-b929-14f669a47260 Apr 17 16:27:00 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 16:27:00 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (1fe06e39-907e-44ec-a6b9-166ddc0adc1f) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 16:27:00 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 16:27:00 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:24:45Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-SnapshotsActionsTest-Volume-1453311494',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=35a67ca3-0f17-4a6c-a359-9173ce5f1aca,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='5b416855df3047d79fd4d27b2b1695b8',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:24:45Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:24:45Z,use_quota=True,user_id='535feccd53b44a24a33905f2f2a4795d',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 16:27:00 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:24:45Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-SnapshotsActionsTest-Volume-1453311494',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=35a67ca3-0f17-4a6c-a359-9173ce5f1aca,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='5b416855df3047d79fd4d27b2b1695b8',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:24:45Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:24:45Z,use_quota=True,user_id='535feccd53b44a24a33905f2f2a4795d',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(35a67ca3-0f17-4a6c-a359-9173ce5f1aca),volume_id=35a67ca3-0f17-4a6c-a359-9173ce5f1aca,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': '35a67ca3-0f17-4a6c-a359-9173ce5f1aca', 'volume_name': 'volume-35a67ca3-0f17-4a6c-a359-9173ce5f1aca', 'volume_size': 1}} Apr 17 16:27:00 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'retry': {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']}, 'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': '35a67ca3-0f17-4a6c-a359-9173ce5f1aca', 'volume': {'id': '35a67ca3-0f17-4a6c-a359-9173ce5f1aca', '_name_id': None, 'ec2_id': None, 'user_id': '535feccd53b44a24a33905f2f2a4795d', 'project_id': '5b416855df3047d79fd4d27b2b1695b8', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-SnapshotsActionsTest-Volume-1453311494', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T16:24:45.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-35a67ca3-0f17-4a6c-a359-9173ce5f1aca', 'name_id': '35a67ca3-0f17-4a6c-a359-9173ce5f1aca', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-SnapshotsActionsTest-Volume-1453311494', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '5b416855df3047d79fd4d27b2b1695b8', 'qos_specs': None, 'reservations': ['7d4b6592-57f9-4b52-b860-f07e41add803', 'f96f8a6a-f71d-4298-a8e4-362b7bcfdfe8', '39019dff-14a7-4a6d-8e21-9297b7666adc', '3459ea23-ef9a-4aec-b56b-db2ebc6e2a90'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': '535feccd53b44a24a33905f2f2a4795d', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-SnapshotsActionsTest-Volume-1453311494', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '5b416855df3047d79fd4d27b2b1695b8', 'qos_specs': None, 'reservations': ['7d4b6592-57f9-4b52-b860-f07e41add803', 'f96f8a6a-f71d-4298-a8e4-362b7bcfdfe8', '39019dff-14a7-4a6d-8e21-9297b7666adc', '3459ea23-ef9a-4aec-b56b-db2ebc6e2a90'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': '535feccd53b44a24a33905f2f2a4795d', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': '535feccd53b44a24a33905f2f2a4795d', 'metadata': {}, 'qos_specs': None}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(35a67ca3-0f17-4a6c-a359-9173ce5f1aca),volume_id=35a67ca3-0f17-4a6c-a359-9173ce5f1aca,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:24:45Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-SnapshotsActionsTest-Volume-1453311494',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=35a67ca3-0f17-4a6c-a359-9173ce5f1aca,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='5b416855df3047d79fd4d27b2b1695b8',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:24:45Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:24:45Z,use_quota=True,user_id='535feccd53b44a24a33905f2f2a4795d',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 16:27:00 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:24:45Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-SnapshotsActionsTest-Volume-1453311494',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=35a67ca3-0f17-4a6c-a359-9173ce5f1aca,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='5b416855df3047d79fd4d27b2b1695b8',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:24:45Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:24:45Z,use_quota=True,user_id='535feccd53b44a24a33905f2f2a4795d',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:24:45Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-SnapshotsActionsTest-Volume-1453311494',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=35a67ca3-0f17-4a6c-a359-9173ce5f1aca,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='5b416855df3047d79fd4d27b2b1695b8',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:24:45Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:24:45Z,use_quota=True,user_id='535feccd53b44a24a33905f2f2a4795d',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 16:27:00 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 16:27:00 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 16:27:00 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:27:00 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 16:27:00 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:27:00 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 16:27:00 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:27:00 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 16:27:00 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:27:00 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 16:27:00 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:27:00 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 16:27:00 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:27:00 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 16:27:00 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:27:00 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 16:27:00 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:27:00 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:27:00 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 16:27:00 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 16:27:00 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (1fe06e39-907e-44ec-a6b9-166ddc0adc1f) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:27:00 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (1fe06e39-907e-44ec-a6b9-166ddc0adc1f) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:27:00 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (e710bb08-39ca-4436-bac1-ad77c9ac3fe4) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:27:00 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (e710bb08-39ca-4436-bac1-ad77c9ac3fe4) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:27:00 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (8308a4a0-3bd7-4b66-8ef0-d4ee3d506e8a) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:27:00 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (8308a4a0-3bd7-4b66-8ef0-d4ee3d506e8a) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:27:00 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (a86b9e02-957b-463f-8dc1-903e6e35d6d6) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:27:00 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Updating volume 35a67ca3-0f17-4a6c-a359-9173ce5f1aca with {'scheduled_at': datetime.datetime(2023, 4, 17, 16, 27, 0, 611245), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 16:27:00 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Volume 35a67ca3-0f17-4a6c-a359-9173ce5f1aca: re-scheduling SchedulerAPI.create_volume attempt 1 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 16:27:00 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Volume 35a67ca3-0f17-4a6c-a359-9173ce5f1aca: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 16:27:00 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] deleating volume v_35a67ca3-0f17-4a6c-a359-9173ce5f1aca {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 16:27:00 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] get snapshots for volume v_35a67ca3-0f17-4a6c-a359-9173ce5f1aca {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 16:27:00 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_35a67ca3-0f17-4a6c-a359-9173ce5f1aca/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:27:00 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_35a67ca3-0f17-4a6c-a359-9173ce5f1aca/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:27:00 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Volume reschedule parameters: True retry: {'num_attempts': 2, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 16:27:00 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Flow 'volume_create_manager' (2d7a2364-63fa-4118-bdb1-5126c469be94) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 16:27:00 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (fa74d58f-3eab-46df-a605-41884a549be8) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:27:00 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (fa74d58f-3eab-46df-a605-41884a549be8) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:24:45Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-SnapshotsActionsTest-Volume-1453311494',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=35a67ca3-0f17-4a6c-a359-9173ce5f1aca,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='5b416855df3047d79fd4d27b2b1695b8',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:27:01Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:27:01Z,use_quota=True,user_id='535feccd53b44a24a33905f2f2a4795d',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:27:00 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (ace27a89-4176-490d-a4aa-fc8ee0980b04) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:27:00 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (ace27a89-4176-490d-a4aa-fc8ee0980b04) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:27:00 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (35a05545-70e2-44e8-b3d6-237b02ccb4da) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:27:00 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (35a05545-70e2-44e8-b3d6-237b02ccb4da) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': '35a67ca3-0f17-4a6c-a359-9173ce5f1aca', 'volume_name': 'volume-35a67ca3-0f17-4a6c-a359-9173ce5f1aca', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:27:00 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (bebd6ca5-f6cf-413e-8bcd-fdbdd2a73ac0) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:27:00 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (bebd6ca5-f6cf-413e-8bcd-fdbdd2a73ac0) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:27:00 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (d35eee88-b0be-4cea-b26c-e5539cb9bd12) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:27:00 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Volume 35a67ca3-0f17-4a6c-a359-9173ce5f1aca: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-35a67ca3-0f17-4a6c-a359-9173ce5f1aca', 'volume_size': 1} Apr 17 16:27:00 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] creating volume v_35a67ca3-0f17-4a6c-a359-9173ce5f1aca. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 16:27:00 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] create volume {'name': 'v_35a67ca3-0f17-4a6c-a359-9173ce5f1aca', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 16:27:00 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:27:00 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:27:03 user cinder-volume[84696]: DEBUG cinder.utils [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.976(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:27:03 user cinder-volume[84696]: DEBUG cinder.utils [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_35a67ca3-0f17-4a6c-a359-9173ce5f1aca/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:27:03 user cinder-volume[84696]: DEBUG cinder.utils [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.862(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:27:03 user cinder-volume[84696]: DEBUG cinder.utils [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:27:03 user cinder-volume[84696]: DEBUG cinder.utils [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.021(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:27:03 user cinder-volume[84696]: DEBUG cinder.utils [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_53029faa-77b2-467d-8a64-eb5ef129e41c/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:27:03 user cinder-volume[84696]: DEBUG cinder.utils [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.801(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:27:03 user cinder-volume[84696]: DEBUG cinder.utils [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:27:06 user cinder-volume[84696]: DEBUG cinder.utils [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.868(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:27:06 user cinder-volume[84696]: DEBUG cinder.utils [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:27:06 user cinder-volume[84696]: DEBUG cinder.utils [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.499(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:27:06 user cinder-volume[84696]: DEBUG cinder.utils [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_2e8580be-c7f2-4d3a-9007-d192a5b029b3/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:27:06 user cinder-volume[84696]: DEBUG cinder.utils [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.343(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:27:06 user cinder-volume[84696]: DEBUG cinder.utils [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:27:06 user cinder-volume[84696]: DEBUG cinder.utils [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.053(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:27:06 user cinder-volume[84696]: DEBUG cinder.utils [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_35a67ca3-0f17-4a6c-a359-9173ce5f1aca/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:27:09 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.495(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:27:09 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:27:09 user cinder-volume[84696]: DEBUG cinder.utils [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.163(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:27:09 user cinder-volume[84696]: DEBUG cinder.utils [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_53029faa-77b2-467d-8a64-eb5ef129e41c/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:27:09 user cinder-volume[84696]: DEBUG cinder.utils [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.006(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:27:09 user cinder-volume[84696]: DEBUG cinder.utils [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:27:10 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:UPDATE=7,cinder:SELECT=8,cinder:INSERT=1 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:27:12 user cinder-volume[84696]: DEBUG cinder.utils [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.012(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:27:12 user cinder-volume[84696]: DEBUG cinder.utils [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:27:12 user cinder-volume[84696]: DEBUG cinder.utils [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.193(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:27:12 user cinder-volume[84696]: DEBUG cinder.utils [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_35a67ca3-0f17-4a6c-a359-9173ce5f1aca/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:27:15 user cinder-volume[84696]: DEBUG cinder.utils [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 72.812(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:27:15 user cinder-volume[84696]: DEBUG cinder.utils [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:27:15 user cinder-volume[84696]: DEBUG cinder.utils [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 73.150(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:27:15 user cinder-volume[84696]: DEBUG cinder.utils [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_94610127-c8b8-4df4-9e39-a1f41b5a9b98/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:27:15 user cinder-volume[84696]: DEBUG cinder.utils [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 73.166(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:27:15 user cinder-volume[84696]: DEBUG cinder.utils [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_02aba338-fa31-4e46-8fcf-7f86191e44d1/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:27:15 user cinder-volume[84696]: DEBUG cinder.utils [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 72.862(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:27:15 user cinder-volume[84696]: DEBUG cinder.utils [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:27:18 user cinder-volume[84696]: DEBUG cinder.utils [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.220(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:27:18 user cinder-volume[84696]: DEBUG cinder.utils [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:27:18 user cinder-volume[84696]: DEBUG cinder.utils [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.378(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:27:18 user cinder-volume[84696]: DEBUG cinder.utils [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_53029faa-77b2-467d-8a64-eb5ef129e41c/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:27:22 user cinder-volume[84696]: DEBUG cinder.utils [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.228(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:27:22 user cinder-volume[84696]: DEBUG cinder.utils [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:27:22 user cinder-volume[84696]: DEBUG cinder.utils [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 142.286(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:27:22 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_c9fa0a17-69d4-40d0-9d65-af6ae349cfd8/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:27:22 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (28e9bdbf-3834-419e-8596-3394b9b72ead) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 16:27:22 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (1fb8c9f4-3ea6-4074-a3b7-6276b466d7e4) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:27:22 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (1fb8c9f4-3ea6-4074-a3b7-6276b466d7e4) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:27:22 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Flow 'volume_create_manager' (fe1e710e-d987-444a-80f3-30b4d3d04581) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 16:27:22 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-f436a31c-1175-4e56-a839-11ba99c1cd60 tempest-VolumeSnapshotQuotasNegativeTestJSON-101904959 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 16:27:22 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 16:27:22 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 16:27:22 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 16:27:22 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 16:27:22 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 16:27:22 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 16:27:22 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 16:27:22 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 16:27:22 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 16:27:22 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 16:27:22 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 16:27:22 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 16:27:22 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 16:27:22 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 16:27:22 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 16:27:22 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 16:27:22 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 16:27:22 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 16:27:22 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 16:27:22 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 16:27:22 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 16:27:22 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 16:27:22 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 16:27:22 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:27:22 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 16:27:22 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:27:22 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 16:27:22 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:27:22 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 16:27:22 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:27:22 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 16:27:22 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:27:22 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 16:27:22 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:27:22 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 16:27:22 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:27:22 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 16:27:22 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:27:22 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:27:22 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 16:27:22 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 16:27:22 user cinder-volume[84696]: DEBUG cinder.utils [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.431(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:27:22 user cinder-volume[84696]: DEBUG cinder.utils [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_35a67ca3-0f17-4a6c-a359-9173ce5f1aca/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:27:24 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=2 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:27:24 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:INSERT=5,cinder:UPDATE=5,cinder:SELECT=22 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:27:25 user cinder-volume[84696]: DEBUG cinder.utils [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.929(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:27:25 user cinder-volume[84696]: DEBUG cinder.utils [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_2e8580be-c7f2-4d3a-9007-d192a5b029b3/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:27:25 user cinder-volume[84696]: DEBUG cinder.utils [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.773(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:27:25 user cinder-volume[84696]: DEBUG cinder.utils [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:27:28 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.926(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:27:28 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:27:28 user cinder-volume[84696]: DEBUG cinder.utils [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 142.310(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:27:28 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_7f7b3b2e-e41d-466a-a3c2-e64e3d63b949/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:27:28 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (c9f579d5-df6c-4c85-8f28-d3a84908979d) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 16:27:28 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (c36dbe83-2e1c-4e7f-8e68-752878e3b897) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:27:28 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (c36dbe83-2e1c-4e7f-8e68-752878e3b897) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:27:28 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Flow 'volume_create_manager' (5ada42b5-b063-40de-922e-364cee3d0293) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 16:27:28 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-88db98a2-70e9-4d55-ae40-2f2cdaa2d051 tempest-GroupsV314Test-467284027 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 16:27:28 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 16:27:28 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 16:27:28 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 16:27:28 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 16:27:28 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 16:27:28 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 16:27:28 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 16:27:28 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 16:27:28 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 16:27:28 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 16:27:28 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 16:27:28 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 16:27:28 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 16:27:28 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 16:27:28 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 16:27:28 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 16:27:28 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 16:27:28 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 16:27:28 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 16:27:28 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 16:27:28 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 16:27:28 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 16:27:28 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 16:27:28 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:27:28 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 16:27:28 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:27:28 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 16:27:28 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:27:28 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 16:27:28 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:27:28 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 16:27:28 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:27:28 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 16:27:28 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:27:28 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 16:27:28 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:27:28 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 16:27:28 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:27:28 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:27:28 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 16:27:28 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 16:27:35 user cinder-volume[84696]: DEBUG cinder.utils [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 37.315(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:27:35 user cinder-volume[84696]: DEBUG cinder.utils [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:27:35 user cinder-volume[84696]: DEBUG cinder.utils [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 37.475(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:27:35 user cinder-volume[84696]: DEBUG cinder.utils [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_53029faa-77b2-467d-8a64-eb5ef129e41c/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:27:38 user cinder-volume[84696]: DEBUG cinder.utils [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 37.324(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:27:38 user cinder-volume[84696]: DEBUG cinder.utils [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:27:38 user cinder-volume[84696]: DEBUG cinder.utils [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 37.505(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:27:38 user cinder-volume[84696]: DEBUG cinder.utils [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_35a67ca3-0f17-4a6c-a359-9173ce5f1aca/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:27:44 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=1,cinder:UPDATE=1 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:27:59 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=9,cinder:UPDATE=4 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:27:59 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Volume reschedule parameters: True retry: {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 16:27:59 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Flow 'volume_create_manager' (02c1cb2a-e9f0-4d27-b9f3-3c239ebbae7f) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 16:27:59 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (e8eb09f7-e8c6-41cc-ad12-49dea48c296e) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:27:59 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (e8eb09f7-e8c6-41cc-ad12-49dea48c296e) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:27:59Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesDeleteCascade-Volume-2055115456',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=8b7f1136-3d20-4cf8-a521-6db1d3a0c8a8,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='abbfd52254f546b0bcfe289884d77f49',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:27:59Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:27:59Z,use_quota=True,user_id='6309fcbea41c471184457d276d32db73',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:27:59 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (1e87cdd6-a8de-472d-9153-6a29d1e99877) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:27:59 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (1e87cdd6-a8de-472d-9153-6a29d1e99877) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:27:59 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (968f20d9-3011-4ae0-8aeb-b95644443b93) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:27:59 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (968f20d9-3011-4ae0-8aeb-b95644443b93) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': '8b7f1136-3d20-4cf8-a521-6db1d3a0c8a8', 'volume_name': 'volume-8b7f1136-3d20-4cf8-a521-6db1d3a0c8a8', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:27:59 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (a688c952-58b5-4e72-b557-e3f6aa59fc9f) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:27:59 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (a688c952-58b5-4e72-b557-e3f6aa59fc9f) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:27:59 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (98e22037-912c-49d2-98ed-1476c4c337fc) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:27:59 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Volume 8b7f1136-3d20-4cf8-a521-6db1d3a0c8a8: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-8b7f1136-3d20-4cf8-a521-6db1d3a0c8a8', 'volume_size': 1} Apr 17 16:27:59 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] creating volume v_8b7f1136-3d20-4cf8-a521-6db1d3a0c8a8. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 16:27:59 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] create volume {'name': 'v_8b7f1136-3d20-4cf8-a521-6db1d3a0c8a8', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 16:27:59 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:27:59 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:27:59 user cinder-volume[84696]: DEBUG cinder.utils [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 73.827(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:27:59 user cinder-volume[84696]: DEBUG cinder.utils [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:27:59 user cinder-volume[84696]: DEBUG cinder.utils [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 73.985(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:27:59 user cinder-volume[84696]: DEBUG cinder.utils [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_2e8580be-c7f2-4d3a-9007-d192a5b029b3/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:28:02 user cinder-volume[84696]: DEBUG cinder.utils [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 3.066(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:28:02 user cinder-volume[84696]: DEBUG cinder.utils [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:28:02 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 73.977(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:28:02 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:28:02 user cinder-volume[84696]: DEBUG cinder.utils [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 134.942(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:28:02 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_b900713d-b991-43a5-b403-483dfded7e11/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:28:02 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (9ca078d3-8942-4a79-9ad3-de0bd37971ce) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 16:28:02 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (0ffb3fa7-e336-4c13-a8cb-decd6d3449a8) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:28:02 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (0ffb3fa7-e336-4c13-a8cb-decd6d3449a8) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:28:02 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Flow 'volume_create_manager' (c2749752-3942-4459-a666-55ad469f1e72) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 16:28:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-1f5f4be7-f5ce-44e1-9d40-d669e0a61ddf tempest-VolumesTransfersV357Test-333017112 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 16:28:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 16:28:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 16:28:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 16:28:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 16:28:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 16:28:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 16:28:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 16:28:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 16:28:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 16:28:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 16:28:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 16:28:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 16:28:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 16:28:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 16:28:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 16:28:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 16:28:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 16:28:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 16:28:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 16:28:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 16:28:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 16:28:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 16:28:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 16:28:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:28:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 16:28:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:28:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 16:28:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:28:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 16:28:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:28:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 16:28:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:28:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 16:28:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:28:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 16:28:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:28:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 16:28:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:28:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:28:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 16:28:02 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 16:28:03 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Volume reschedule parameters: True retry: {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 16:28:03 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Flow 'volume_create_manager' (bf763189-a29f-46d2-8264-0dc7ac20dcae) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 16:28:03 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (7e67e899-dc4f-4a7f-8e99-b3ef0885e2df) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:28:03 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (7e67e899-dc4f-4a7f-8e99-b3ef0885e2df) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:28:03Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesSnapshotListTestJSON-Volume-552939668',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=2159f1f1-2564-4d84-8110-c1f7d843edb1,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='c4b1a8a0074e4798bd385ee96e8ac8b7',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:28:03Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:28:03Z,use_quota=True,user_id='476228814d42467c880dce2b1c5c5113',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:28:03 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (95c2cf7a-1593-47ae-a6ac-d14d8e803005) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:28:03 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (95c2cf7a-1593-47ae-a6ac-d14d8e803005) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:28:03 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (09582591-6c95-4ae1-a204-94f972fa7c5a) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:28:03 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (09582591-6c95-4ae1-a204-94f972fa7c5a) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': '2159f1f1-2564-4d84-8110-c1f7d843edb1', 'volume_name': 'volume-2159f1f1-2564-4d84-8110-c1f7d843edb1', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:28:03 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (2c0ee00b-1090-4004-8623-c56a01c14570) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:28:03 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (2c0ee00b-1090-4004-8623-c56a01c14570) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:28:03 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (570e6f99-fa26-4ab4-9479-4c30e864117b) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:28:03 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Volume 2159f1f1-2564-4d84-8110-c1f7d843edb1: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-2159f1f1-2564-4d84-8110-c1f7d843edb1', 'volume_size': 1} Apr 17 16:28:03 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] creating volume v_2159f1f1-2564-4d84-8110-c1f7d843edb1. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 16:28:03 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] create volume {'name': 'v_2159f1f1-2564-4d84-8110-c1f7d843edb1', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 16:28:03 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:28:03 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:28:05 user cinder-volume[84696]: DEBUG cinder.utils [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.149(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:28:05 user cinder-volume[84696]: DEBUG cinder.utils [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:28:05 user cinder-volume[84696]: DEBUG cinder.utils [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.140(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:28:05 user cinder-volume[84696]: DEBUG cinder.utils [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:28:08 user cinder-volume[84696]: DEBUG cinder.utils [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 70.628(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:28:08 user cinder-volume[84696]: DEBUG cinder.utils [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:28:08 user cinder-volume[84696]: DEBUG cinder.utils [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 70.787(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:28:08 user cinder-volume[84696]: DEBUG cinder.utils [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_53029faa-77b2-467d-8a64-eb5ef129e41c/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:28:08 user cinder-volume[84696]: DEBUG cinder.utils [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.224(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:28:08 user cinder-volume[84696]: DEBUG cinder.utils [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:28:09 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Volume reschedule parameters: True retry: {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 16:28:09 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Flow 'volume_create_manager' (4bec709d-dabc-402a-b320-5c8918e025d7) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 16:28:09 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (70340d43-3477-43ac-b1f9-b7fbc29f47b0) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:28:09 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (70340d43-3477-43ac-b1f9-b7fbc29f47b0) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:28:10Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumeQuotasAdminTestJSON-Volume-332841798',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=f6c0e341-6f00-4e32-bd07-31f954c1ad8a,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='6e4c2ec9177349a08ce53e07b24ad79c',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:28:10Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:28:10Z,use_quota=True,user_id='ab5017da30ac4d47ba665ff135deb309',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:28:09 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (17697f70-2399-4d33-91c6-50d7e87f28f6) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:28:09 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (17697f70-2399-4d33-91c6-50d7e87f28f6) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:28:09 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (64f343b0-d0c4-416d-ba0c-c437b0cfcc5d) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:28:09 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (64f343b0-d0c4-416d-ba0c-c437b0cfcc5d) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': 'f6c0e341-6f00-4e32-bd07-31f954c1ad8a', 'volume_name': 'volume-f6c0e341-6f00-4e32-bd07-31f954c1ad8a', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:28:09 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (e84dbb19-68b3-46b9-a71c-47dfc92a01e4) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:28:09 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (e84dbb19-68b3-46b9-a71c-47dfc92a01e4) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:28:09 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (731f3428-59f2-4f05-be5a-40a8f39522a5) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:28:09 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Volume f6c0e341-6f00-4e32-bd07-31f954c1ad8a: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-f6c0e341-6f00-4e32-bd07-31f954c1ad8a', 'volume_size': 1} Apr 17 16:28:09 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] creating volume v_f6c0e341-6f00-4e32-bd07-31f954c1ad8a. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 16:28:09 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] create volume {'name': 'v_f6c0e341-6f00-4e32-bd07-31f954c1ad8a', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 16:28:09 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:28:09 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:28:11 user cinder-volume[84696]: DEBUG cinder.utils [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 1.464(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:28:11 user cinder-volume[84696]: DEBUG cinder.utils [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:28:11 user cinder-volume[84696]: DEBUG cinder.utils [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.284(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:28:11 user cinder-volume[84696]: DEBUG cinder.utils [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:28:11 user cinder-volume[84696]: DEBUG cinder.utils [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 70.819(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:28:11 user cinder-volume[84696]: DEBUG cinder.utils [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_35a67ca3-0f17-4a6c-a359-9173ce5f1aca/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:28:11 user cinder-volume[84696]: DEBUG cinder.utils [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 70.641(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:28:11 user cinder-volume[84696]: DEBUG cinder.utils [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:28:14 user cinder-volume[84696]: DEBUG cinder.utils [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 4.536(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:28:14 user cinder-volume[84696]: DEBUG cinder.utils [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:28:14 user cinder-volume[84696]: DEBUG cinder.utils [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 11.367(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:28:14 user cinder-volume[84696]: DEBUG cinder.utils [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:28:21 user cinder-volume[84696]: DEBUG cinder.utils [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 138.028(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:28:21 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:28:21 user cinder-volume[84696]: INFO cinder.message.api [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Creating message record for request_id = req-d3245848-3d3a-4130-8413-3f811df2f47d Apr 17 16:28:21 user cinder-volume[84696]: DEBUG cinder.utils [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.950(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:28:21 user cinder-volume[84696]: DEBUG cinder.utils [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:28:21 user cinder-volume[84696]: DEBUG cinder.utils [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 11.134(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:28:21 user cinder-volume[84696]: DEBUG cinder.utils [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:28:21 user cinder-volume[84696]: DEBUG cinder.utils [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 138.371(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:28:21 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_94610127-c8b8-4df4-9e39-a1f41b5a9b98/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:28:21 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (fd9eb00d-6c37-455d-a77f-5284073a5df9) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 16:28:21 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (7a282bc8-9447-4b11-be85-006463b73164) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:28:21 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (7a282bc8-9447-4b11-be85-006463b73164) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:28:21 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Flow 'volume_create_manager' (08b3d2ad-b078-465b-82ae-7d05f7e28f48) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 16:28:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 16:28:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 16:28:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 16:28:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 16:28:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 16:28:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 16:28:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 16:28:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 16:28:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 16:28:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 16:28:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 16:28:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 16:28:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 16:28:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 16:28:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 16:28:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 16:28:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 16:28:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 16:28:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 16:28:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 16:28:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 16:28:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 16:28:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 16:28:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 16:28:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:28:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 16:28:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:28:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 16:28:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:28:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 16:28:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:28:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 16:28:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:28:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 16:28:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:28:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 16:28:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:28:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 16:28:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:28:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:28:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 16:28:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 16:28:21 user cinder-volume[84696]: DEBUG cinder.utils [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 138.097(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:28:21 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:28:21 user cinder-volume[84696]: INFO cinder.message.api [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Creating message record for request_id = req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 Apr 17 16:28:21 user cinder-volume[84696]: DEBUG cinder.utils [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 138.407(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:28:21 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_02aba338-fa31-4e46-8fcf-7f86191e44d1/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:28:21 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (176d0698-e62f-4f58-b9a1-b2c5a154dbdc) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 16:28:21 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (57c8bc6a-ea3d-4181-951a-bfa0a30743e4) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:28:21 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (57c8bc6a-ea3d-4181-951a-bfa0a30743e4) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:28:21 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Flow 'volume_create_manager' (1d6096aa-378e-4f49-8198-aca639c7cbe1) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 16:28:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 16:28:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 16:28:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 16:28:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 16:28:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 16:28:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 16:28:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 16:28:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 16:28:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 16:28:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 16:28:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 16:28:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 16:28:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 16:28:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 16:28:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 16:28:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 16:28:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 16:28:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 16:28:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 16:28:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 16:28:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 16:28:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 16:28:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 16:28:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 16:28:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:28:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 16:28:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:28:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 16:28:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:28:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 16:28:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:28:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 16:28:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:28:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 16:28:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:28:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 16:28:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:28:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 16:28:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:28:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:28:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 16:28:21 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 16:28:21 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 16:28:21 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (665c1787-9f99-4a5c-9b6c-74f1f587dfd8) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 16:28:21 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 16:28:21 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:21:24Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-AbsoluteLimitsTests-Volume-1339022265',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=94610127-c8b8-4df4-9e39-a1f41b5a9b98,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='2d881d9502184c74b36e1c50acad1f80',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:26:03Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:26:03Z,use_quota=True,user_id='e460a7fb86b4454788ec1950b15434a4',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 16:28:21 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:21:24Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-AbsoluteLimitsTests-Volume-1339022265',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=94610127-c8b8-4df4-9e39-a1f41b5a9b98,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='2d881d9502184c74b36e1c50acad1f80',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:26:03Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:26:03Z,use_quota=True,user_id='e460a7fb86b4454788ec1950b15434a4',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(94610127-c8b8-4df4-9e39-a1f41b5a9b98),volume_id=94610127-c8b8-4df4-9e39-a1f41b5a9b98,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': '94610127-c8b8-4df4-9e39-a1f41b5a9b98', 'volume_name': 'volume-94610127-c8b8-4df4-9e39-a1f41b5a9b98', 'volume_size': 1}} Apr 17 16:28:21 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': '94610127-c8b8-4df4-9e39-a1f41b5a9b98', 'volume': {'id': '94610127-c8b8-4df4-9e39-a1f41b5a9b98', '_name_id': None, 'ec2_id': None, 'user_id': 'e460a7fb86b4454788ec1950b15434a4', 'project_id': '2d881d9502184c74b36e1c50acad1f80', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-AbsoluteLimitsTests-Volume-1339022265', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'admin_metadata': {}, 'glance_metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T16:21:23.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-94610127-c8b8-4df4-9e39-a1f41b5a9b98', 'name_id': '94610127-c8b8-4df4-9e39-a1f41b5a9b98', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-AbsoluteLimitsTests-Volume-1339022265', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '2d881d9502184c74b36e1c50acad1f80', 'qos_specs': None, 'reservations': ['7f441e01-433c-4826-9a44-ec0d9a586e2a', '62cb3d03-7543-4f33-9fda-df3d7d67cb94', 'a069f7b2-c2bc-47d4-85ba-479a14bdc4a4', 'e95f992f-6c15-431a-9f34-693f405be424'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'e460a7fb86b4454788ec1950b15434a4', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-AbsoluteLimitsTests-Volume-1339022265', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '2d881d9502184c74b36e1c50acad1f80', 'qos_specs': None, 'reservations': ['7f441e01-433c-4826-9a44-ec0d9a586e2a', '62cb3d03-7543-4f33-9fda-df3d7d67cb94', 'a069f7b2-c2bc-47d4-85ba-479a14bdc4a4', 'e95f992f-6c15-431a-9f34-693f405be424'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'e460a7fb86b4454788ec1950b15434a4', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': 'e460a7fb86b4454788ec1950b15434a4', 'metadata': {}, 'qos_specs': None, 'retry': {'num_attempts': 3, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']}}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(94610127-c8b8-4df4-9e39-a1f41b5a9b98),volume_id=94610127-c8b8-4df4-9e39-a1f41b5a9b98,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:21:24Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-AbsoluteLimitsTests-Volume-1339022265',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=94610127-c8b8-4df4-9e39-a1f41b5a9b98,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='2d881d9502184c74b36e1c50acad1f80',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:26:03Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:26:03Z,use_quota=True,user_id='e460a7fb86b4454788ec1950b15434a4',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 16:28:21 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:21:24Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-AbsoluteLimitsTests-Volume-1339022265',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=94610127-c8b8-4df4-9e39-a1f41b5a9b98,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='2d881d9502184c74b36e1c50acad1f80',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:26:03Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:26:03Z,use_quota=True,user_id='e460a7fb86b4454788ec1950b15434a4',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:21:24Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-AbsoluteLimitsTests-Volume-1339022265',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=94610127-c8b8-4df4-9e39-a1f41b5a9b98,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='2d881d9502184c74b36e1c50acad1f80',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:26:03Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:26:03Z,use_quota=True,user_id='e460a7fb86b4454788ec1950b15434a4',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 16:28:21 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 16:28:21 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 16:28:21 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:28:21 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 16:28:21 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:28:21 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 16:28:21 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:28:21 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 16:28:21 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:28:21 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 16:28:21 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:28:21 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 16:28:21 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:28:21 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 16:28:21 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:28:21 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 16:28:21 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:28:21 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:28:21 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 16:28:21 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 16:28:21 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (665c1787-9f99-4a5c-9b6c-74f1f587dfd8) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:28:21 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (665c1787-9f99-4a5c-9b6c-74f1f587dfd8) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:28:21 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (735b1cd2-114f-40a4-98a7-9ea866b1163d) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:28:21 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (735b1cd2-114f-40a4-98a7-9ea866b1163d) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:28:21 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (71ccce28-3f6c-4370-b660-37e5ae648a21) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:28:21 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (71ccce28-3f6c-4370-b660-37e5ae648a21) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:28:21 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (66979d03-bf50-4ec4-a32e-e4b96a752e54) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:28:21 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Updating volume 94610127-c8b8-4df4-9e39-a1f41b5a9b98 with {'scheduled_at': datetime.datetime(2023, 4, 17, 16, 28, 21, 209612), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 16:28:21 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 16:28:21 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=16,cinder:UPDATE=1 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:28:21 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (7ce8a6b4-0b75-4b62-b424-a4503fa7c4c9) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 16:28:21 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 16:28:21 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:21:25Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-volume-1452364054',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=fe50cd00-dcd4-4b21-a4e9-6f226f8c6ac1,host='user@joviandss-2#joviandss-2',id=02aba338-fa31-4e46-8fcf-7f86191e44d1,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='e10e34b8774b42d8a4cb1f78d72b411d',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:26:03Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='deleting',terminated_at=None,updated_at=2023-04-17T16:26:03Z,use_quota=True,user_id='e7392ed432ad41279967273f16b4a4c8',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(400bec8d-bd01-4057-8fb8-49a786cd39f1),volume_type_id=400bec8d-bd01-4057-8fb8-49a786cd39f1)}, 'provides': None} Apr 17 16:28:21 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:21:25Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-volume-1452364054',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=fe50cd00-dcd4-4b21-a4e9-6f226f8c6ac1,host='user@joviandss-2#joviandss-2',id=02aba338-fa31-4e46-8fcf-7f86191e44d1,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='e10e34b8774b42d8a4cb1f78d72b411d',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:26:03Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='deleting',terminated_at=None,updated_at=2023-04-17T16:26:03Z,use_quota=True,user_id='e7392ed432ad41279967273f16b4a4c8',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(400bec8d-bd01-4057-8fb8-49a786cd39f1),volume_type_id=400bec8d-bd01-4057-8fb8-49a786cd39f1), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=fe50cd00-dcd4-4b21-a4e9-6f226f8c6ac1,image_id=None,operation=,resource_backend='user@joviandss-2',snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(02aba338-fa31-4e46-8fcf-7f86191e44d1),volume_id=02aba338-fa31-4e46-8fcf-7f86191e44d1,volume_properties=VolumeProperties,volume_type=VolumeType(400bec8d-bd01-4057-8fb8-49a786cd39f1))}, 'provides': {'status': 'deleting', 'type': 'raw', 'volume_id': '02aba338-fa31-4e46-8fcf-7f86191e44d1', 'volume_name': 'volume-02aba338-fa31-4e46-8fcf-7f86191e44d1', 'volume_size': 1}} Apr 17 16:28:21 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'request_spec': {'consistencygroup_id': None, 'group_id': 'fe50cd00-dcd4-4b21-a4e9-6f226f8c6ac1', 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': '02aba338-fa31-4e46-8fcf-7f86191e44d1', 'volume': {'id': '02aba338-fa31-4e46-8fcf-7f86191e44d1', '_name_id': None, 'ec2_id': None, 'user_id': 'e7392ed432ad41279967273f16b4a4c8', 'project_id': 'e10e34b8774b42d8a4cb1f78d72b411d', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-volume-1452364054', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '400bec8d-bd01-4057-8fb8-49a786cd39f1', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': 'fe50cd00-dcd4-4b21-a4e9-6f226f8c6ac1', 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'admin_metadata': {}, 'glance_metadata': {}, 'volume_type': {'id': '400bec8d-bd01-4057-8fb8-49a786cd39f1', 'name': 'tempest-GroupsTest-volume-type-1895690014', 'description': None, 'is_public': True, 'extra_specs': {}, 'qos_specs_id': None, 'created_at': '2023-04-17T16:21:23.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T16:21:24.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-02aba338-fa31-4e46-8fcf-7f86191e44d1', 'name_id': '02aba338-fa31-4e46-8fcf-7f86191e44d1', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '400bec8d-bd01-4057-8fb8-49a786cd39f1', 'name': 'tempest-GroupsTest-volume-type-1895690014', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {}, 'qos_specs_id': None, 'created_at': '2023-04-17T16:21:23.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': 'fe50cd00-dcd4-4b21-a4e9-6f226f8c6ac1', 'display_description': None, 'display_name': 'tempest-volume-1452364054', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': 'e10e34b8774b42d8a4cb1f78d72b411d', 'qos_specs': None, 'reservations': ['f5b0b08e-834b-48d7-bb1a-7efd1002da9b', 'ac966cb4-51d5-4799-b0b6-b3453a218bd0', '77612ba8-fbd6-4ff5-97c8-7ddceb3a7c7a', '101ca44e-fd59-494f-9f9a-2df7868625ff'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'e7392ed432ad41279967273f16b4a4c8', 'volume_type_id': '400bec8d-bd01-4057-8fb8-49a786cd39f1'}, 'resource_backend': 'user@joviandss-2', 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': 'fe50cd00-dcd4-4b21-a4e9-6f226f8c6ac1', 'display_description': None, 'display_name': 'tempest-volume-1452364054', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': 'e10e34b8774b42d8a4cb1f78d72b411d', 'qos_specs': None, 'reservations': ['f5b0b08e-834b-48d7-bb1a-7efd1002da9b', 'ac966cb4-51d5-4799-b0b6-b3453a218bd0', '77612ba8-fbd6-4ff5-97c8-7ddceb3a7c7a', '101ca44e-fd59-494f-9f9a-2df7868625ff'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'e7392ed432ad41279967273f16b4a4c8', 'volume_type_id': '400bec8d-bd01-4057-8fb8-49a786cd39f1'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T16:21:23Z,deleted=False,deleted_at=None,description=None,extra_specs={},id=400bec8d-bd01-4057-8fb8-49a786cd39f1,is_public=True,name='tempest-GroupsTest-volume-type-1895690014',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T16:21:23Z,deleted=False,deleted_at=None,description=None,extra_specs={},id=400bec8d-bd01-4057-8fb8-49a786cd39f1,is_public=True,name='tempest-GroupsTest-volume-type-1895690014',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': 'e7392ed432ad41279967273f16b4a4c8', 'metadata': {}, 'qos_specs': None, 'retry': {'num_attempts': 3, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']}}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=fe50cd00-dcd4-4b21-a4e9-6f226f8c6ac1,image_id=None,operation=,resource_backend='user@joviandss-2',snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(02aba338-fa31-4e46-8fcf-7f86191e44d1),volume_id=02aba338-fa31-4e46-8fcf-7f86191e44d1,volume_properties=VolumeProperties,volume_type=VolumeType(400bec8d-bd01-4057-8fb8-49a786cd39f1)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:21:25Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-volume-1452364054',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=fe50cd00-dcd4-4b21-a4e9-6f226f8c6ac1,host='user@joviandss-2#joviandss-2',id=02aba338-fa31-4e46-8fcf-7f86191e44d1,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='e10e34b8774b42d8a4cb1f78d72b411d',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:26:03Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='deleting',terminated_at=None,updated_at=2023-04-17T16:26:03Z,use_quota=True,user_id='e7392ed432ad41279967273f16b4a4c8',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(400bec8d-bd01-4057-8fb8-49a786cd39f1),volume_type_id=400bec8d-bd01-4057-8fb8-49a786cd39f1), 'context': }, 'provides': None} Apr 17 16:28:21 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:21:25Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-volume-1452364054',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=fe50cd00-dcd4-4b21-a4e9-6f226f8c6ac1,host='user@joviandss-2#joviandss-2',id=02aba338-fa31-4e46-8fcf-7f86191e44d1,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='e10e34b8774b42d8a4cb1f78d72b411d',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:26:03Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='deleting',terminated_at=None,updated_at=2023-04-17T16:26:03Z,use_quota=True,user_id='e7392ed432ad41279967273f16b4a4c8',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(400bec8d-bd01-4057-8fb8-49a786cd39f1),volume_type_id=400bec8d-bd01-4057-8fb8-49a786cd39f1)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:21:25Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-volume-1452364054',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=fe50cd00-dcd4-4b21-a4e9-6f226f8c6ac1,host='user@joviandss-2#joviandss-2',id=02aba338-fa31-4e46-8fcf-7f86191e44d1,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='e10e34b8774b42d8a4cb1f78d72b411d',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:26:03Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='deleting',terminated_at=None,updated_at=2023-04-17T16:26:03Z,use_quota=True,user_id='e7392ed432ad41279967273f16b4a4c8',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(400bec8d-bd01-4057-8fb8-49a786cd39f1),volume_type_id=400bec8d-bd01-4057-8fb8-49a786cd39f1)} Apr 17 16:28:21 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 16:28:21 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 16:28:21 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:28:21 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 16:28:21 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:28:21 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 16:28:21 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:28:21 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 16:28:21 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:28:21 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 16:28:21 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:28:21 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 16:28:21 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:28:21 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 16:28:21 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:28:21 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 16:28:21 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:28:21 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:28:21 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 16:28:21 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 16:28:21 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (7ce8a6b4-0b75-4b62-b424-a4503fa7c4c9) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:28:21 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (7ce8a6b4-0b75-4b62-b424-a4503fa7c4c9) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:28:21 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (569fdc18-4a98-49fd-b912-cfe5e9f79489) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:28:21 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (569fdc18-4a98-49fd-b912-cfe5e9f79489) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:28:21 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (ad539e83-7287-4a7b-927c-fe8b69e98df5) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:28:21 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (ad539e83-7287-4a7b-927c-fe8b69e98df5) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:28:21 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (64080c9c-bb96-4290-97cf-4bc092461a3e) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:28:21 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Updating volume 02aba338-fa31-4e46-8fcf-7f86191e44d1 with {'scheduled_at': datetime.datetime(2023, 4, 17, 16, 28, 21, 261567), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 16:28:21 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Volume 94610127-c8b8-4df4-9e39-a1f41b5a9b98: re-scheduling SchedulerAPI.create_volume attempt 3 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 16:28:21 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Volume 02aba338-fa31-4e46-8fcf-7f86191e44d1: re-scheduling SchedulerAPI.create_volume attempt 3 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 16:28:21 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Volume 94610127-c8b8-4df4-9e39-a1f41b5a9b98: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 16:28:21 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] deleating volume v_94610127-c8b8-4df4-9e39-a1f41b5a9b98 {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 16:28:21 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] get snapshots for volume v_94610127-c8b8-4df4-9e39-a1f41b5a9b98 {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 16:28:21 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_94610127-c8b8-4df4-9e39-a1f41b5a9b98/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:28:21 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_94610127-c8b8-4df4-9e39-a1f41b5a9b98/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:28:21 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Volume 02aba338-fa31-4e46-8fcf-7f86191e44d1: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 16:28:21 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] deleating volume v_02aba338-fa31-4e46-8fcf-7f86191e44d1 {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 16:28:21 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] get snapshots for volume v_02aba338-fa31-4e46-8fcf-7f86191e44d1 {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 16:28:21 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_02aba338-fa31-4e46-8fcf-7f86191e44d1/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:28:21 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_02aba338-fa31-4e46-8fcf-7f86191e44d1/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:28:24 user cinder-volume[84696]: DEBUG cinder.utils [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.888(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:28:24 user cinder-volume[84696]: DEBUG cinder.utils [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_02aba338-fa31-4e46-8fcf-7f86191e44d1/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:28:24 user cinder-volume[84696]: DEBUG cinder.utils [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.031(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:28:24 user cinder-volume[84696]: DEBUG cinder.utils [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:28:24 user cinder-volume[84696]: DEBUG cinder.utils [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.898(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:28:24 user cinder-volume[84696]: DEBUG cinder.utils [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_94610127-c8b8-4df4-9e39-a1f41b5a9b98/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:28:27 user cinder-volume[84696]: DEBUG cinder.utils [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.961(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:28:27 user cinder-volume[84696]: DEBUG cinder.utils [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_02aba338-fa31-4e46-8fcf-7f86191e44d1/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:28:27 user cinder-volume[84696]: DEBUG cinder.utils [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.978(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:28:27 user cinder-volume[84696]: DEBUG cinder.utils [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_94610127-c8b8-4df4-9e39-a1f41b5a9b98/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:28:30 user cinder-volume[84696]: DEBUG cinder.utils [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 20.344(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:28:30 user cinder-volume[84696]: DEBUG cinder.utils [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:28:33 user cinder-volume[84696]: DEBUG cinder.utils [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.112(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:28:33 user cinder-volume[84696]: DEBUG cinder.utils [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_94610127-c8b8-4df4-9e39-a1f41b5a9b98/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:28:33 user cinder-volume[84696]: DEBUG cinder.utils [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.105(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:28:33 user cinder-volume[84696]: DEBUG cinder.utils [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_02aba338-fa31-4e46-8fcf-7f86191e44d1/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:28:34 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=14,cinder:UPDATE=5,cinder:INSERT=3 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:28:34 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:UPDATE=3,cinder:SELECT=3,cinder:INSERT=1 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:28:40 user cinder-volume[84696]: DEBUG cinder.utils [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 40.954(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:28:40 user cinder-volume[84696]: DEBUG cinder.utils [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:28:43 user cinder-volume[84696]: DEBUG cinder.utils [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 40.038(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:28:43 user cinder-volume[84696]: DEBUG cinder.utils [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:28:43 user cinder-volume[84696]: DEBUG cinder.utils [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.898(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:28:43 user cinder-volume[84696]: DEBUG cinder.utils [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_02aba338-fa31-4e46-8fcf-7f86191e44d1/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:28:43 user cinder-volume[84696]: DEBUG cinder.utils [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.908(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:28:43 user cinder-volume[84696]: DEBUG cinder.utils [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_94610127-c8b8-4df4-9e39-a1f41b5a9b98/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:28:49 user cinder-volume[84696]: DEBUG cinder.utils [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.352(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:28:49 user cinder-volume[84696]: DEBUG cinder.utils [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:28:54 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=4,cinder:UPDATE=2 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:29:02 user cinder-volume[84696]: DEBUG cinder.utils [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 40.904(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:29:02 user cinder-volume[84696]: DEBUG cinder.utils [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_02aba338-fa31-4e46-8fcf-7f86191e44d1/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:29:02 user cinder-volume[84696]: DEBUG cinder.utils [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 40.914(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:29:02 user cinder-volume[84696]: DEBUG cinder.utils [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_94610127-c8b8-4df4-9e39-a1f41b5a9b98/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:29:05 user cinder-volume[84696]: DEBUG cinder.utils [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 139.936(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:29:05 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:29:05 user cinder-volume[84696]: INFO cinder.message.api [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Creating message record for request_id = req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 Apr 17 16:29:05 user cinder-volume[84696]: DEBUG cinder.utils [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 140.099(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:29:05 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_2e8580be-c7f2-4d3a-9007-d192a5b029b3/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:29:05 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (0c39da14-5af1-4383-938b-31d3f459502d) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 16:29:05 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (a26a2d6c-d88b-4ea5-abfa-8928c59bd5e6) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:29:05 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (a26a2d6c-d88b-4ea5-abfa-8928c59bd5e6) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:29:05 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Flow 'volume_create_manager' (c075b31d-6ee9-4251-a364-5c1f818b4541) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 16:29:05 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 16:29:05 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 16:29:05 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 16:29:05 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 16:29:05 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 16:29:05 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 16:29:05 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 16:29:05 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 16:29:05 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 16:29:05 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 16:29:05 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 16:29:05 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 16:29:05 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 16:29:05 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 16:29:05 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 16:29:05 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 16:29:05 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 16:29:05 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 16:29:05 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 16:29:05 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 16:29:05 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 16:29:05 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 16:29:05 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 16:29:05 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 16:29:05 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:29:05 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 16:29:05 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:29:05 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 16:29:05 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:29:05 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 16:29:05 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:29:05 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 16:29:05 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:29:05 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 16:29:05 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:29:05 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 16:29:05 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:29:05 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 16:29:05 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:29:05 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:29:05 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 16:29:05 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 16:29:05 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 16:29:05 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (927390f1-640a-47a6-8f48-6587ffb094a4) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 16:29:05 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 16:29:05 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:22:11Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesNegativeTest-Volume-375402817',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=2e8580be-c7f2-4d3a-9007-d192a5b029b3,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='935190a703294690b79c182d789f31e4',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:26:45Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:26:45Z,use_quota=True,user_id='9296aa0c28024371bac0c53928bff421',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 16:29:05 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:22:11Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesNegativeTest-Volume-375402817',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=2e8580be-c7f2-4d3a-9007-d192a5b029b3,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='935190a703294690b79c182d789f31e4',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:26:45Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:26:45Z,use_quota=True,user_id='9296aa0c28024371bac0c53928bff421',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(2e8580be-c7f2-4d3a-9007-d192a5b029b3),volume_id=2e8580be-c7f2-4d3a-9007-d192a5b029b3,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': '2e8580be-c7f2-4d3a-9007-d192a5b029b3', 'volume_name': 'volume-2e8580be-c7f2-4d3a-9007-d192a5b029b3', 'volume_size': 1}} Apr 17 16:29:05 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': '2e8580be-c7f2-4d3a-9007-d192a5b029b3', 'volume': {'id': '2e8580be-c7f2-4d3a-9007-d192a5b029b3', '_name_id': None, 'ec2_id': None, 'user_id': '9296aa0c28024371bac0c53928bff421', 'project_id': '935190a703294690b79c182d789f31e4', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-VolumesNegativeTest-Volume-375402817', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'admin_metadata': {}, 'glance_metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T16:22:11.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-2e8580be-c7f2-4d3a-9007-d192a5b029b3', 'name_id': '2e8580be-c7f2-4d3a-9007-d192a5b029b3', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumesNegativeTest-Volume-375402817', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '935190a703294690b79c182d789f31e4', 'qos_specs': None, 'reservations': ['8302a15a-cac9-4777-aeda-3cd0f18e59f0', '4f81147c-1b8e-4140-ae06-353eac50d673', '0d992e26-ac75-42fe-82b4-74e2842f6518', 'ce106309-de8e-43ad-89d6-aaedb5c859b5'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': '9296aa0c28024371bac0c53928bff421', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumesNegativeTest-Volume-375402817', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '935190a703294690b79c182d789f31e4', 'qos_specs': None, 'reservations': ['8302a15a-cac9-4777-aeda-3cd0f18e59f0', '4f81147c-1b8e-4140-ae06-353eac50d673', '0d992e26-ac75-42fe-82b4-74e2842f6518', 'ce106309-de8e-43ad-89d6-aaedb5c859b5'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': '9296aa0c28024371bac0c53928bff421', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': '9296aa0c28024371bac0c53928bff421', 'metadata': {}, 'qos_specs': None, 'retry': {'num_attempts': 3, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']}}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(2e8580be-c7f2-4d3a-9007-d192a5b029b3),volume_id=2e8580be-c7f2-4d3a-9007-d192a5b029b3,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:22:11Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesNegativeTest-Volume-375402817',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=2e8580be-c7f2-4d3a-9007-d192a5b029b3,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='935190a703294690b79c182d789f31e4',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:26:45Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:26:45Z,use_quota=True,user_id='9296aa0c28024371bac0c53928bff421',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 16:29:05 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:22:11Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesNegativeTest-Volume-375402817',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=2e8580be-c7f2-4d3a-9007-d192a5b029b3,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='935190a703294690b79c182d789f31e4',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:26:45Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:26:45Z,use_quota=True,user_id='9296aa0c28024371bac0c53928bff421',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:22:11Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesNegativeTest-Volume-375402817',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=2e8580be-c7f2-4d3a-9007-d192a5b029b3,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='935190a703294690b79c182d789f31e4',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:26:45Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:26:45Z,use_quota=True,user_id='9296aa0c28024371bac0c53928bff421',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 16:29:05 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 16:29:05 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 16:29:05 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:29:05 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 16:29:05 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:29:05 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 16:29:05 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:29:05 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 16:29:05 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:29:05 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 16:29:05 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:29:05 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 16:29:05 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:29:05 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 16:29:05 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:29:05 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 16:29:05 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:29:05 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:29:05 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 16:29:05 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 16:29:05 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (927390f1-640a-47a6-8f48-6587ffb094a4) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:29:05 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (927390f1-640a-47a6-8f48-6587ffb094a4) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:29:05 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (eea81c23-06ad-4cdf-a8d6-9fa401768843) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:29:05 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (eea81c23-06ad-4cdf-a8d6-9fa401768843) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:29:05 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (bc2fffc6-0dc8-4335-98cc-5934cc0c66fc) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:29:05 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (bc2fffc6-0dc8-4335-98cc-5934cc0c66fc) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:29:05 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (ab9fc0d2-624c-4bd2-a9aa-de04049a75b9) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:29:05 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Updating volume 2e8580be-c7f2-4d3a-9007-d192a5b029b3 with {'scheduled_at': datetime.datetime(2023, 4, 17, 16, 29, 5, 376154), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 16:29:05 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=7,cinder:UPDATE=2 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:29:05 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Volume 2e8580be-c7f2-4d3a-9007-d192a5b029b3: re-scheduling SchedulerAPI.create_volume attempt 3 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 16:29:05 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Volume 2e8580be-c7f2-4d3a-9007-d192a5b029b3: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 16:29:05 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] deleating volume v_2e8580be-c7f2-4d3a-9007-d192a5b029b3 {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 16:29:05 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] get snapshots for volume v_2e8580be-c7f2-4d3a-9007-d192a5b029b3 {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 16:29:05 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_2e8580be-c7f2-4d3a-9007-d192a5b029b3/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:29:05 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_2e8580be-c7f2-4d3a-9007-d192a5b029b3/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:29:08 user cinder-volume[84696]: DEBUG cinder.utils [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.943(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:29:08 user cinder-volume[84696]: DEBUG cinder.utils [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_2e8580be-c7f2-4d3a-9007-d192a5b029b3/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:29:08 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 140.091(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:29:08 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:29:08 user cinder-volume[84696]: ERROR oslo_service.periodic_task [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Error during VolumeManager.publish_service_capabilities: TypeError: 'NoneType' object is not subscriptable Apr 17 16:29:08 user cinder-volume[84696]: ERROR oslo_service.periodic_task Traceback (most recent call last): Apr 17 16:29:08 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/usr/local/lib/python3.10/dist-packages/oslo_service/periodic_task.py", line 216, in run_periodic_tasks Apr 17 16:29:08 user cinder-volume[84696]: ERROR oslo_service.periodic_task task(self, context) Apr 17 16:29:08 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/manager.py", line 2876, in publish_service_capabilities Apr 17 16:29:08 user cinder-volume[84696]: ERROR oslo_service.periodic_task self._report_driver_status(context) Apr 17 16:29:08 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/manager.py", line 2755, in _report_driver_status Apr 17 16:29:08 user cinder-volume[84696]: ERROR oslo_service.periodic_task volume_stats = get_stats() Apr 17 16:29:08 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/usr/local/lib/python3.10/dist-packages/oslo_utils/timeutils.py", line 319, in wrapper Apr 17 16:29:08 user cinder-volume[84696]: ERROR oslo_service.periodic_task result = func(*args, **kwargs) Apr 17 16:29:08 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/manager.py", line 2753, in get_stats Apr 17 16:29:08 user cinder-volume[84696]: ERROR oslo_service.periodic_task return self.driver.get_volume_stats(refresh=True) Apr 17 16:29:08 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/driver.py", line 666, in get_volume_stats Apr 17 16:29:08 user cinder-volume[84696]: ERROR oslo_service.periodic_task self._update_volume_stats() Apr 17 16:29:08 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 726, in _update_volume_stats Apr 17 16:29:08 user cinder-volume[84696]: ERROR oslo_service.periodic_task pool_stats = self.ra.get_pool_stats() Apr 17 16:29:08 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 897, in get_pool_stats Apr 17 16:29:08 user cinder-volume[84696]: ERROR oslo_service.periodic_task if not resp["error"] and resp["code"] == 200: Apr 17 16:29:08 user cinder-volume[84696]: ERROR oslo_service.periodic_task TypeError: 'NoneType' object is not subscriptable Apr 17 16:29:08 user cinder-volume[84696]: ERROR oslo_service.periodic_task Apr 17 16:29:08 user cinder-volume[84696]: WARNING oslo.service.loopingcall [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Function 'cinder.service.Service.periodic_tasks' run outlasted interval by 80.10 sec Apr 17 16:29:08 user cinder-volume[84696]: DEBUG oslo_service.periodic_task [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Running periodic task VolumeManager.publish_service_capabilities {{(pid=84696) run_periodic_tasks /usr/local/lib/python3.10/dist-packages/oslo_service/periodic_task.py:210}} Apr 17 16:29:08 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Updating volume stats {{(pid=84696) _update_volume_stats /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:724}} Apr 17 16:29:08 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Get pool Pool-0 fsprops {{(pid=84696) get_pool_stats /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:894}} Apr 17 16:29:08 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:29:08 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:29:11 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 3.063(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:29:11 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:29:11 user cinder-volume[84696]: DEBUG cinder.utils [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.013(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:29:11 user cinder-volume[84696]: DEBUG cinder.utils [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_2e8580be-c7f2-4d3a-9007-d192a5b029b3/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:29:14 user cinder-volume[84696]: DEBUG cinder.utils [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 136.898(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:29:14 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_53029faa-77b2-467d-8a64-eb5ef129e41c/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:29:14 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (d77b789b-f220-432a-8116-dd04f7251a2b) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 16:29:14 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (c1cc5b5b-0595-4919-a9be-081891bbc222) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:29:14 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (c1cc5b5b-0595-4919-a9be-081891bbc222) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:29:14 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Flow 'volume_create_manager' (8b4577f5-1014-4b6b-8e35-e6e3dea11c23) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 16:29:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 16:29:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 16:29:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 16:29:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 16:29:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 16:29:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 16:29:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 16:29:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 16:29:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 16:29:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 16:29:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 16:29:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 16:29:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 16:29:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 16:29:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 16:29:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 16:29:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 16:29:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 16:29:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 16:29:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 16:29:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 16:29:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 16:29:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 16:29:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 16:29:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:29:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 16:29:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:29:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 16:29:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:29:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 16:29:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:29:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 16:29:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:29:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 16:29:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:29:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 16:29:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:29:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 16:29:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:29:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:29:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 16:29:14 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 16:29:14 user cinder-volume[84696]: DEBUG cinder.utils [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 75.343(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:29:14 user cinder-volume[84696]: DEBUG cinder.utils [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:29:14 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.157(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:29:14 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:29:14 user cinder-volume[84696]: DEBUG cinder.utils [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 136.763(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:29:14 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:29:14 user cinder-volume[84696]: INFO cinder.message.api [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Creating message record for request_id = req-8d765f93-bce0-4815-ac07-22b8b3bd2556 Apr 17 16:29:14 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 16:29:14 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (17e59cd1-25dc-4cf1-a81d-545228670599) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 16:29:14 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 16:29:14 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:24:42Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesDeleteCascade-Volume-1549123617',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=53029faa-77b2-467d-8a64-eb5ef129e41c,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='abbfd52254f546b0bcfe289884d77f49',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:26:58Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:26:58Z,use_quota=True,user_id='6309fcbea41c471184457d276d32db73',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 16:29:14 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:24:42Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesDeleteCascade-Volume-1549123617',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=53029faa-77b2-467d-8a64-eb5ef129e41c,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='abbfd52254f546b0bcfe289884d77f49',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:26:58Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:26:58Z,use_quota=True,user_id='6309fcbea41c471184457d276d32db73',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(53029faa-77b2-467d-8a64-eb5ef129e41c),volume_id=53029faa-77b2-467d-8a64-eb5ef129e41c,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': '53029faa-77b2-467d-8a64-eb5ef129e41c', 'volume_name': 'volume-53029faa-77b2-467d-8a64-eb5ef129e41c', 'volume_size': 1}} Apr 17 16:29:14 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': '53029faa-77b2-467d-8a64-eb5ef129e41c', 'volume': {'id': '53029faa-77b2-467d-8a64-eb5ef129e41c', '_name_id': None, 'ec2_id': None, 'user_id': '6309fcbea41c471184457d276d32db73', 'project_id': 'abbfd52254f546b0bcfe289884d77f49', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-VolumesDeleteCascade-Volume-1549123617', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'admin_metadata': {}, 'glance_metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T16:24:42.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-53029faa-77b2-467d-8a64-eb5ef129e41c', 'name_id': '53029faa-77b2-467d-8a64-eb5ef129e41c', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumesDeleteCascade-Volume-1549123617', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': 'abbfd52254f546b0bcfe289884d77f49', 'qos_specs': None, 'reservations': ['15abf3ac-c420-4557-ab29-de463d495550', '4070e169-cef7-4670-80f8-a805062b3ed7', '598ac100-d775-4c42-82f1-6a2eb9c17061', 'c035551f-9406-4ac7-a6e8-e732d18bff88'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': '6309fcbea41c471184457d276d32db73', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumesDeleteCascade-Volume-1549123617', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': 'abbfd52254f546b0bcfe289884d77f49', 'qos_specs': None, 'reservations': ['15abf3ac-c420-4557-ab29-de463d495550', '4070e169-cef7-4670-80f8-a805062b3ed7', '598ac100-d775-4c42-82f1-6a2eb9c17061', 'c035551f-9406-4ac7-a6e8-e732d18bff88'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': '6309fcbea41c471184457d276d32db73', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': '6309fcbea41c471184457d276d32db73', 'metadata': {}, 'qos_specs': None, 'retry': {'num_attempts': 2, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']}}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(53029faa-77b2-467d-8a64-eb5ef129e41c),volume_id=53029faa-77b2-467d-8a64-eb5ef129e41c,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:24:42Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesDeleteCascade-Volume-1549123617',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=53029faa-77b2-467d-8a64-eb5ef129e41c,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='abbfd52254f546b0bcfe289884d77f49',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:26:58Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:26:58Z,use_quota=True,user_id='6309fcbea41c471184457d276d32db73',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 16:29:14 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:24:42Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesDeleteCascade-Volume-1549123617',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=53029faa-77b2-467d-8a64-eb5ef129e41c,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='abbfd52254f546b0bcfe289884d77f49',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:26:58Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:26:58Z,use_quota=True,user_id='6309fcbea41c471184457d276d32db73',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:24:42Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesDeleteCascade-Volume-1549123617',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=53029faa-77b2-467d-8a64-eb5ef129e41c,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='abbfd52254f546b0bcfe289884d77f49',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:26:58Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:26:58Z,use_quota=True,user_id='6309fcbea41c471184457d276d32db73',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 16:29:14 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 16:29:14 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 16:29:14 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:29:14 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 16:29:14 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:29:14 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 16:29:14 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:29:14 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 16:29:14 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:29:14 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 16:29:14 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:29:14 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 16:29:14 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:29:14 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 16:29:14 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:29:14 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 16:29:14 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:29:14 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:29:14 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 16:29:14 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 16:29:14 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (17e59cd1-25dc-4cf1-a81d-545228670599) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:29:14 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (17e59cd1-25dc-4cf1-a81d-545228670599) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:29:14 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (8235445a-ab69-4e76-b3ab-77ed3c2566ec) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:29:14 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (8235445a-ab69-4e76-b3ab-77ed3c2566ec) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:29:14 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (d9c097c3-7f72-4214-baa4-1a8f17ee01e1) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:29:14 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (d9c097c3-7f72-4214-baa4-1a8f17ee01e1) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:29:14 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (33495df2-1373-45cd-b3ea-3fcb107e4d47) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:29:14 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Updating volume 53029faa-77b2-467d-8a64-eb5ef129e41c with {'scheduled_at': datetime.datetime(2023, 4, 17, 16, 29, 14, 573683), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 16:29:14 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Volume 53029faa-77b2-467d-8a64-eb5ef129e41c: re-scheduling SchedulerAPI.create_volume attempt 2 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 16:29:14 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Volume 53029faa-77b2-467d-8a64-eb5ef129e41c: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 16:29:14 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] deleating volume v_53029faa-77b2-467d-8a64-eb5ef129e41c {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 16:29:14 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] get snapshots for volume v_53029faa-77b2-467d-8a64-eb5ef129e41c {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 16:29:14 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_53029faa-77b2-467d-8a64-eb5ef129e41c/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:29:14 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_53029faa-77b2-467d-8a64-eb5ef129e41c/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:29:14 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Volume reschedule parameters: True retry: {'num_attempts': 3, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 16:29:14 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Flow 'volume_create_manager' (7225a004-fd24-4f3f-9cc2-ab67de608fa3) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 16:29:14 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (dba2830d-e45d-44b7-94d6-02fe7969b66b) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:29:14 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (dba2830d-e45d-44b7-94d6-02fe7969b66b) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:24:42Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesDeleteCascade-Volume-1549123617',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=53029faa-77b2-467d-8a64-eb5ef129e41c,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='abbfd52254f546b0bcfe289884d77f49',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:29:15Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:29:15Z,use_quota=True,user_id='6309fcbea41c471184457d276d32db73',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:29:14 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (97deb593-97b0-4a98-b1d1-56dbf3eaf113) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:29:14 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (97deb593-97b0-4a98-b1d1-56dbf3eaf113) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:29:14 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (33fe0d1c-f2e3-4c6c-8f74-66cb120b2949) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:29:14 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (33fe0d1c-f2e3-4c6c-8f74-66cb120b2949) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': '53029faa-77b2-467d-8a64-eb5ef129e41c', 'volume_name': 'volume-53029faa-77b2-467d-8a64-eb5ef129e41c', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:29:14 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (50399269-425d-4934-9081-95e06fdd54f8) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:29:14 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (50399269-425d-4934-9081-95e06fdd54f8) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:29:14 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (c2eaf81a-f6a5-485f-a7f2-0955f699d5ac) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:29:14 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Volume 53029faa-77b2-467d-8a64-eb5ef129e41c: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-53029faa-77b2-467d-8a64-eb5ef129e41c', 'volume_size': 1} Apr 17 16:29:14 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] creating volume v_53029faa-77b2-467d-8a64-eb5ef129e41c. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 16:29:14 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] create volume {'name': 'v_53029faa-77b2-467d-8a64-eb5ef129e41c', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 16:29:14 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:29:14 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:29:17 user cinder-volume[84696]: DEBUG cinder.utils [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.800(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:29:17 user cinder-volume[84696]: DEBUG cinder.utils [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:29:17 user cinder-volume[84696]: DEBUG cinder.utils [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.967(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:29:17 user cinder-volume[84696]: DEBUG cinder.utils [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_53029faa-77b2-467d-8a64-eb5ef129e41c/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:29:17 user cinder-volume[84696]: DEBUG cinder.utils [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.160(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:29:17 user cinder-volume[84696]: DEBUG cinder.utils [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_2e8580be-c7f2-4d3a-9007-d192a5b029b3/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:29:17 user cinder-volume[84696]: DEBUG cinder.utils [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 136.752(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:29:17 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:29:17 user cinder-volume[84696]: INFO cinder.message.api [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Creating message record for request_id = req-61f60e22-802a-4068-b929-14f669a47260 Apr 17 16:29:17 user cinder-volume[84696]: DEBUG cinder.utils [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 74.413(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:29:17 user cinder-volume[84696]: DEBUG cinder.utils [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:29:17 user cinder-volume[84696]: DEBUG cinder.utils [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 136.937(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:29:17 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_35a67ca3-0f17-4a6c-a359-9173ce5f1aca/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:29:17 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (a86b9e02-957b-463f-8dc1-903e6e35d6d6) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 16:29:17 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (17a8ddec-f38d-4551-97a8-17d948ff30f2) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:29:17 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (17a8ddec-f38d-4551-97a8-17d948ff30f2) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:29:17 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Flow 'volume_create_manager' (08b4dd37-7b56-4cd7-9be7-a82dd3ecb693) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 16:29:17 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 16:29:17 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 16:29:17 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 16:29:17 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 16:29:17 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 16:29:17 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 16:29:17 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 16:29:17 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 16:29:17 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 16:29:17 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 16:29:17 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 16:29:17 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 16:29:17 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 16:29:17 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 16:29:17 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 16:29:17 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 16:29:17 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 16:29:17 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 16:29:17 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 16:29:17 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 16:29:17 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 16:29:17 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 16:29:17 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 16:29:17 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 16:29:17 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:29:17 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 16:29:17 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:29:17 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 16:29:17 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:29:17 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 16:29:17 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:29:17 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 16:29:17 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:29:17 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 16:29:17 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:29:17 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 16:29:17 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:29:17 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 16:29:17 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:29:17 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:29:17 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 16:29:17 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 16:29:17 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 16:29:17 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (d35eee88-b0be-4cea-b26c-e5539cb9bd12) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 16:29:17 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 16:29:17 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:24:45Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-SnapshotsActionsTest-Volume-1453311494',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=35a67ca3-0f17-4a6c-a359-9173ce5f1aca,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='5b416855df3047d79fd4d27b2b1695b8',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:27:01Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:27:01Z,use_quota=True,user_id='535feccd53b44a24a33905f2f2a4795d',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 16:29:17 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:24:45Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-SnapshotsActionsTest-Volume-1453311494',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=35a67ca3-0f17-4a6c-a359-9173ce5f1aca,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='5b416855df3047d79fd4d27b2b1695b8',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:27:01Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:27:01Z,use_quota=True,user_id='535feccd53b44a24a33905f2f2a4795d',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(35a67ca3-0f17-4a6c-a359-9173ce5f1aca),volume_id=35a67ca3-0f17-4a6c-a359-9173ce5f1aca,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': '35a67ca3-0f17-4a6c-a359-9173ce5f1aca', 'volume_name': 'volume-35a67ca3-0f17-4a6c-a359-9173ce5f1aca', 'volume_size': 1}} Apr 17 16:29:17 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': '35a67ca3-0f17-4a6c-a359-9173ce5f1aca', 'volume': {'id': '35a67ca3-0f17-4a6c-a359-9173ce5f1aca', '_name_id': None, 'ec2_id': None, 'user_id': '535feccd53b44a24a33905f2f2a4795d', 'project_id': '5b416855df3047d79fd4d27b2b1695b8', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-SnapshotsActionsTest-Volume-1453311494', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'admin_metadata': {}, 'glance_metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T16:24:45.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-35a67ca3-0f17-4a6c-a359-9173ce5f1aca', 'name_id': '35a67ca3-0f17-4a6c-a359-9173ce5f1aca', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-SnapshotsActionsTest-Volume-1453311494', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '5b416855df3047d79fd4d27b2b1695b8', 'qos_specs': None, 'reservations': ['7d4b6592-57f9-4b52-b860-f07e41add803', 'f96f8a6a-f71d-4298-a8e4-362b7bcfdfe8', '39019dff-14a7-4a6d-8e21-9297b7666adc', '3459ea23-ef9a-4aec-b56b-db2ebc6e2a90'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': '535feccd53b44a24a33905f2f2a4795d', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-SnapshotsActionsTest-Volume-1453311494', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '5b416855df3047d79fd4d27b2b1695b8', 'qos_specs': None, 'reservations': ['7d4b6592-57f9-4b52-b860-f07e41add803', 'f96f8a6a-f71d-4298-a8e4-362b7bcfdfe8', '39019dff-14a7-4a6d-8e21-9297b7666adc', '3459ea23-ef9a-4aec-b56b-db2ebc6e2a90'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': '535feccd53b44a24a33905f2f2a4795d', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': '535feccd53b44a24a33905f2f2a4795d', 'metadata': {}, 'qos_specs': None, 'retry': {'num_attempts': 2, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']}}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(35a67ca3-0f17-4a6c-a359-9173ce5f1aca),volume_id=35a67ca3-0f17-4a6c-a359-9173ce5f1aca,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:24:45Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-SnapshotsActionsTest-Volume-1453311494',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=35a67ca3-0f17-4a6c-a359-9173ce5f1aca,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='5b416855df3047d79fd4d27b2b1695b8',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:27:01Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:27:01Z,use_quota=True,user_id='535feccd53b44a24a33905f2f2a4795d',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 16:29:17 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:24:45Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-SnapshotsActionsTest-Volume-1453311494',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=35a67ca3-0f17-4a6c-a359-9173ce5f1aca,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='5b416855df3047d79fd4d27b2b1695b8',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:27:01Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:27:01Z,use_quota=True,user_id='535feccd53b44a24a33905f2f2a4795d',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:24:45Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-SnapshotsActionsTest-Volume-1453311494',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=35a67ca3-0f17-4a6c-a359-9173ce5f1aca,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='5b416855df3047d79fd4d27b2b1695b8',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:27:01Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:27:01Z,use_quota=True,user_id='535feccd53b44a24a33905f2f2a4795d',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 16:29:17 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 16:29:17 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 16:29:17 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:29:17 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 16:29:17 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:29:17 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 16:29:17 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:29:17 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 16:29:17 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:29:17 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 16:29:17 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:29:17 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 16:29:17 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:29:17 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 16:29:17 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:29:17 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 16:29:17 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:29:17 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:29:17 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 16:29:17 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 16:29:17 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (d35eee88-b0be-4cea-b26c-e5539cb9bd12) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:29:17 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (d35eee88-b0be-4cea-b26c-e5539cb9bd12) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:29:17 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (bebd6ca5-f6cf-413e-8bcd-fdbdd2a73ac0) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:29:17 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (bebd6ca5-f6cf-413e-8bcd-fdbdd2a73ac0) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:29:17 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (35a05545-70e2-44e8-b3d6-237b02ccb4da) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:29:17 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (35a05545-70e2-44e8-b3d6-237b02ccb4da) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:29:17 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (ace27a89-4176-490d-a4aa-fc8ee0980b04) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:29:17 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Updating volume 35a67ca3-0f17-4a6c-a359-9173ce5f1aca with {'scheduled_at': datetime.datetime(2023, 4, 17, 16, 29, 17, 644951), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 16:29:17 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Volume 35a67ca3-0f17-4a6c-a359-9173ce5f1aca: re-scheduling SchedulerAPI.create_volume attempt 2 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 16:29:17 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Volume 35a67ca3-0f17-4a6c-a359-9173ce5f1aca: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 16:29:17 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] deleating volume v_35a67ca3-0f17-4a6c-a359-9173ce5f1aca {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 16:29:17 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] get snapshots for volume v_35a67ca3-0f17-4a6c-a359-9173ce5f1aca {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 16:29:17 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_35a67ca3-0f17-4a6c-a359-9173ce5f1aca/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:29:17 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_35a67ca3-0f17-4a6c-a359-9173ce5f1aca/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:29:17 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Volume reschedule parameters: True retry: {'num_attempts': 3, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 16:29:17 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Flow 'volume_create_manager' (c0601d20-c7d6-4708-ac77-15a1cd3fc6e2) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 16:29:17 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (db751aca-e239-4852-9d10-12075ebac73b) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:29:17 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (db751aca-e239-4852-9d10-12075ebac73b) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:24:45Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-SnapshotsActionsTest-Volume-1453311494',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=35a67ca3-0f17-4a6c-a359-9173ce5f1aca,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='5b416855df3047d79fd4d27b2b1695b8',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:29:18Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:29:18Z,use_quota=True,user_id='535feccd53b44a24a33905f2f2a4795d',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:29:17 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (3ba1e240-9772-4d59-9646-a5e3f6b9937f) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:29:17 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (3ba1e240-9772-4d59-9646-a5e3f6b9937f) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:29:17 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (1d906177-e8c9-4ce4-bc90-fb1b8274afbc) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:29:17 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (1d906177-e8c9-4ce4-bc90-fb1b8274afbc) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': '35a67ca3-0f17-4a6c-a359-9173ce5f1aca', 'volume_name': 'volume-35a67ca3-0f17-4a6c-a359-9173ce5f1aca', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:29:17 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (c4e53235-e49d-496d-8b58-fff21be19391) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:29:17 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (c4e53235-e49d-496d-8b58-fff21be19391) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:29:17 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (a6908629-af82-4147-999b-21be37e1797c) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:29:17 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Volume 35a67ca3-0f17-4a6c-a359-9173ce5f1aca: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-35a67ca3-0f17-4a6c-a359-9173ce5f1aca', 'volume_size': 1} Apr 17 16:29:17 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] creating volume v_35a67ca3-0f17-4a6c-a359-9173ce5f1aca. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 16:29:17 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] create volume {'name': 'v_35a67ca3-0f17-4a6c-a359-9173ce5f1aca', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 16:29:17 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:29:17 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:29:20 user cinder-volume[84696]: DEBUG cinder.utils [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.966(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:29:20 user cinder-volume[84696]: DEBUG cinder.utils [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_35a67ca3-0f17-4a6c-a359-9173ce5f1aca/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:29:20 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.281(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:29:20 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:29:20 user cinder-volume[84696]: DEBUG cinder.utils [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.039(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:29:20 user cinder-volume[84696]: DEBUG cinder.utils [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_53029faa-77b2-467d-8a64-eb5ef129e41c/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:29:20 user cinder-volume[84696]: DEBUG cinder.utils [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.875(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:29:20 user cinder-volume[84696]: DEBUG cinder.utils [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:29:20 user cinder-volume[84696]: DEBUG cinder.utils [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.810(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:29:20 user cinder-volume[84696]: DEBUG cinder.utils [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:29:23 user cinder-volume[84696]: DEBUG cinder.utils [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.038(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:29:23 user cinder-volume[84696]: DEBUG cinder.utils [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_35a67ca3-0f17-4a6c-a359-9173ce5f1aca/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:29:23 user cinder-volume[84696]: DEBUG cinder.utils [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 73.722(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:29:23 user cinder-volume[84696]: DEBUG cinder.utils [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:29:23 user cinder-volume[84696]: DEBUG cinder.utils [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.880(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:29:23 user cinder-volume[84696]: DEBUG cinder.utils [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:29:24 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=17,cinder:INSERT=4,cinder:UPDATE=5 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:29:26 user cinder-volume[84696]: DEBUG cinder.utils [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.180(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:29:26 user cinder-volume[84696]: DEBUG cinder.utils [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_53029faa-77b2-467d-8a64-eb5ef129e41c/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:29:26 user cinder-volume[84696]: DEBUG cinder.utils [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.374(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:29:26 user cinder-volume[84696]: DEBUG cinder.utils [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_2e8580be-c7f2-4d3a-9007-d192a5b029b3/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:29:26 user cinder-volume[84696]: DEBUG cinder.utils [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.018(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:29:26 user cinder-volume[84696]: DEBUG cinder.utils [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:29:29 user cinder-volume[84696]: DEBUG cinder.utils [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.022(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:29:29 user cinder-volume[84696]: DEBUG cinder.utils [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:29:29 user cinder-volume[84696]: DEBUG cinder.utils [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.184(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:29:29 user cinder-volume[84696]: DEBUG cinder.utils [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_35a67ca3-0f17-4a6c-a359-9173ce5f1aca/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:29:29 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.499(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:29:29 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:29:36 user cinder-volume[84696]: DEBUG cinder.utils [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 74.856(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:29:36 user cinder-volume[84696]: DEBUG cinder.utils [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_02aba338-fa31-4e46-8fcf-7f86191e44d1/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:29:36 user cinder-volume[84696]: DEBUG cinder.utils [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.394(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:29:36 user cinder-volume[84696]: DEBUG cinder.utils [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:29:36 user cinder-volume[84696]: DEBUG cinder.utils [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.560(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:29:36 user cinder-volume[84696]: DEBUG cinder.utils [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_53029faa-77b2-467d-8a64-eb5ef129e41c/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:29:36 user cinder-volume[84696]: DEBUG cinder.utils [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 74.869(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:29:36 user cinder-volume[84696]: DEBUG cinder.utils [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_94610127-c8b8-4df4-9e39-a1f41b5a9b98/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:29:39 user cinder-volume[84696]: DEBUG cinder.utils [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.398(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:29:39 user cinder-volume[84696]: DEBUG cinder.utils [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:29:39 user cinder-volume[84696]: DEBUG cinder.utils [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.559(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:29:39 user cinder-volume[84696]: DEBUG cinder.utils [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_35a67ca3-0f17-4a6c-a359-9173ce5f1aca/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:29:44 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=2 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:29:44 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=14,cinder:UPDATE=6,cinder:INSERT=2 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:29:45 user cinder-volume[84696]: DEBUG cinder.utils [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.964(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:29:45 user cinder-volume[84696]: DEBUG cinder.utils [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_2e8580be-c7f2-4d3a-9007-d192a5b029b3/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:29:48 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 40.087(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:29:48 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:29:54 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=2,cinder:UPDATE=1 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:29:55 user cinder-volume[84696]: DEBUG cinder.utils [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 40.399(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:29:55 user cinder-volume[84696]: DEBUG cinder.utils [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:29:55 user cinder-volume[84696]: DEBUG cinder.utils [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 40.566(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:29:55 user cinder-volume[84696]: DEBUG cinder.utils [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_53029faa-77b2-467d-8a64-eb5ef129e41c/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:29:58 user cinder-volume[84696]: DEBUG cinder.utils [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 40.410(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:29:58 user cinder-volume[84696]: DEBUG cinder.utils [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:29:58 user cinder-volume[84696]: DEBUG cinder.utils [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 40.571(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:29:58 user cinder-volume[84696]: DEBUG cinder.utils [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_35a67ca3-0f17-4a6c-a359-9173ce5f1aca/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:30:14 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=2 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:30:18 user cinder-volume[84696]: DEBUG cinder.utils [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 72.860(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:30:18 user cinder-volume[84696]: DEBUG cinder.utils [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_2e8580be-c7f2-4d3a-9007-d192a5b029b3/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:30:21 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 73.243(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:30:21 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:30:21 user cinder-volume[84696]: DEBUG cinder.utils [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 138.442(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:30:21 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:30:21 user cinder-volume[84696]: INFO cinder.message.api [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Creating message record for request_id = req-34802d26-273b-4fa3-a0d4-af12da0d916e Apr 17 16:30:21 user cinder-volume[84696]: DEBUG cinder.utils [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 142.434(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:30:21 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:30:21 user cinder-volume[84696]: INFO cinder.message.api [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Creating message record for request_id = req-ccbb52f2-042b-41e0-9837-623e8a9a2738 Apr 17 16:30:21 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 16:30:21 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=7,cinder:UPDATE=2 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:30:21 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 16:30:21 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (570e6f99-fa26-4ab4-9479-4c30e864117b) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 16:30:21 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 16:30:21 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:28:03Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesSnapshotListTestJSON-Volume-552939668',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=2159f1f1-2564-4d84-8110-c1f7d843edb1,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='c4b1a8a0074e4798bd385ee96e8ac8b7',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:28:03Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:28:03Z,use_quota=True,user_id='476228814d42467c880dce2b1c5c5113',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 16:30:21 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:28:03Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesSnapshotListTestJSON-Volume-552939668',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=2159f1f1-2564-4d84-8110-c1f7d843edb1,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='c4b1a8a0074e4798bd385ee96e8ac8b7',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:28:03Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:28:03Z,use_quota=True,user_id='476228814d42467c880dce2b1c5c5113',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(2159f1f1-2564-4d84-8110-c1f7d843edb1),volume_id=2159f1f1-2564-4d84-8110-c1f7d843edb1,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': '2159f1f1-2564-4d84-8110-c1f7d843edb1', 'volume_name': 'volume-2159f1f1-2564-4d84-8110-c1f7d843edb1', 'volume_size': 1}} Apr 17 16:30:21 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'retry': {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']}, 'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': '2159f1f1-2564-4d84-8110-c1f7d843edb1', 'volume': {'id': '2159f1f1-2564-4d84-8110-c1f7d843edb1', '_name_id': None, 'ec2_id': None, 'user_id': '476228814d42467c880dce2b1c5c5113', 'project_id': 'c4b1a8a0074e4798bd385ee96e8ac8b7', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-VolumesSnapshotListTestJSON-Volume-552939668', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T16:28:02.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-2159f1f1-2564-4d84-8110-c1f7d843edb1', 'name_id': '2159f1f1-2564-4d84-8110-c1f7d843edb1', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumesSnapshotListTestJSON-Volume-552939668', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': 'c4b1a8a0074e4798bd385ee96e8ac8b7', 'qos_specs': None, 'reservations': ['0c108957-403f-4dc6-adfa-0a0de976576c', 'e72136a8-7407-4f95-a72e-b484cca53b29', 'cc5aa602-fa98-4b80-9896-3829bb01e999', 'ec5862b0-0012-4eba-b940-a35906fa2d44'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': '476228814d42467c880dce2b1c5c5113', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumesSnapshotListTestJSON-Volume-552939668', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': 'c4b1a8a0074e4798bd385ee96e8ac8b7', 'qos_specs': None, 'reservations': ['0c108957-403f-4dc6-adfa-0a0de976576c', 'e72136a8-7407-4f95-a72e-b484cca53b29', 'cc5aa602-fa98-4b80-9896-3829bb01e999', 'ec5862b0-0012-4eba-b940-a35906fa2d44'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': '476228814d42467c880dce2b1c5c5113', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': '476228814d42467c880dce2b1c5c5113', 'metadata': {}, 'qos_specs': None}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(2159f1f1-2564-4d84-8110-c1f7d843edb1),volume_id=2159f1f1-2564-4d84-8110-c1f7d843edb1,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:28:03Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesSnapshotListTestJSON-Volume-552939668',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=2159f1f1-2564-4d84-8110-c1f7d843edb1,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='c4b1a8a0074e4798bd385ee96e8ac8b7',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:28:03Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:28:03Z,use_quota=True,user_id='476228814d42467c880dce2b1c5c5113',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 16:30:21 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:28:03Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesSnapshotListTestJSON-Volume-552939668',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=2159f1f1-2564-4d84-8110-c1f7d843edb1,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='c4b1a8a0074e4798bd385ee96e8ac8b7',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:28:03Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:28:03Z,use_quota=True,user_id='476228814d42467c880dce2b1c5c5113',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:28:03Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesSnapshotListTestJSON-Volume-552939668',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=2159f1f1-2564-4d84-8110-c1f7d843edb1,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='c4b1a8a0074e4798bd385ee96e8ac8b7',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:28:03Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:28:03Z,use_quota=True,user_id='476228814d42467c880dce2b1c5c5113',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 16:30:21 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 16:30:21 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 16:30:21 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:30:21 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 16:30:21 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:30:21 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 16:30:21 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:30:21 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 16:30:21 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:30:21 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 16:30:21 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:30:21 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 16:30:21 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:30:21 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 16:30:21 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:30:21 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 16:30:21 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:30:21 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:30:21 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 16:30:21 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 16:30:21 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (570e6f99-fa26-4ab4-9479-4c30e864117b) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:30:21 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (570e6f99-fa26-4ab4-9479-4c30e864117b) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:30:21 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (2c0ee00b-1090-4004-8623-c56a01c14570) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:30:21 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (2c0ee00b-1090-4004-8623-c56a01c14570) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:30:21 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (09582591-6c95-4ae1-a204-94f972fa7c5a) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:30:21 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (09582591-6c95-4ae1-a204-94f972fa7c5a) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:30:21 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (95c2cf7a-1593-47ae-a6ac-d14d8e803005) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:30:21 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Updating volume 2159f1f1-2564-4d84-8110-c1f7d843edb1 with {'scheduled_at': datetime.datetime(2023, 4, 17, 16, 30, 21, 682607), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 16:30:21 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (98e22037-912c-49d2-98ed-1476c4c337fc) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 16:30:21 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 16:30:21 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:27:59Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesDeleteCascade-Volume-2055115456',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=8b7f1136-3d20-4cf8-a521-6db1d3a0c8a8,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='abbfd52254f546b0bcfe289884d77f49',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:27:59Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:27:59Z,use_quota=True,user_id='6309fcbea41c471184457d276d32db73',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 16:30:21 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:27:59Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesDeleteCascade-Volume-2055115456',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=8b7f1136-3d20-4cf8-a521-6db1d3a0c8a8,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='abbfd52254f546b0bcfe289884d77f49',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:27:59Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:27:59Z,use_quota=True,user_id='6309fcbea41c471184457d276d32db73',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(8b7f1136-3d20-4cf8-a521-6db1d3a0c8a8),volume_id=8b7f1136-3d20-4cf8-a521-6db1d3a0c8a8,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': '8b7f1136-3d20-4cf8-a521-6db1d3a0c8a8', 'volume_name': 'volume-8b7f1136-3d20-4cf8-a521-6db1d3a0c8a8', 'volume_size': 1}} Apr 17 16:30:21 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'retry': {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']}, 'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': '8b7f1136-3d20-4cf8-a521-6db1d3a0c8a8', 'volume': {'id': '8b7f1136-3d20-4cf8-a521-6db1d3a0c8a8', '_name_id': None, 'ec2_id': None, 'user_id': '6309fcbea41c471184457d276d32db73', 'project_id': 'abbfd52254f546b0bcfe289884d77f49', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-VolumesDeleteCascade-Volume-2055115456', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T16:27:58.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-8b7f1136-3d20-4cf8-a521-6db1d3a0c8a8', 'name_id': '8b7f1136-3d20-4cf8-a521-6db1d3a0c8a8', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumesDeleteCascade-Volume-2055115456', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': 'abbfd52254f546b0bcfe289884d77f49', 'qos_specs': None, 'reservations': ['374a7e05-986c-4abf-a2c9-4761a21cff44', 'b5097575-0972-436f-8ef4-38f9b292ab7b', '56c0c623-13ab-497a-bc50-d552acc88144', 'c6eb5369-b08c-4737-8356-45cc9d705e20'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': '6309fcbea41c471184457d276d32db73', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumesDeleteCascade-Volume-2055115456', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': 'abbfd52254f546b0bcfe289884d77f49', 'qos_specs': None, 'reservations': ['374a7e05-986c-4abf-a2c9-4761a21cff44', 'b5097575-0972-436f-8ef4-38f9b292ab7b', '56c0c623-13ab-497a-bc50-d552acc88144', 'c6eb5369-b08c-4737-8356-45cc9d705e20'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': '6309fcbea41c471184457d276d32db73', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': '6309fcbea41c471184457d276d32db73', 'metadata': {}, 'qos_specs': None}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(8b7f1136-3d20-4cf8-a521-6db1d3a0c8a8),volume_id=8b7f1136-3d20-4cf8-a521-6db1d3a0c8a8,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:27:59Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesDeleteCascade-Volume-2055115456',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=8b7f1136-3d20-4cf8-a521-6db1d3a0c8a8,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='abbfd52254f546b0bcfe289884d77f49',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:27:59Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:27:59Z,use_quota=True,user_id='6309fcbea41c471184457d276d32db73',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 16:30:21 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:27:59Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesDeleteCascade-Volume-2055115456',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=8b7f1136-3d20-4cf8-a521-6db1d3a0c8a8,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='abbfd52254f546b0bcfe289884d77f49',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:27:59Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:27:59Z,use_quota=True,user_id='6309fcbea41c471184457d276d32db73',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:27:59Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesDeleteCascade-Volume-2055115456',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=8b7f1136-3d20-4cf8-a521-6db1d3a0c8a8,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='abbfd52254f546b0bcfe289884d77f49',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:27:59Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:27:59Z,use_quota=True,user_id='6309fcbea41c471184457d276d32db73',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 16:30:21 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 16:30:21 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 16:30:21 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:30:21 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 16:30:21 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:30:21 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 16:30:21 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:30:21 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 16:30:21 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:30:21 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 16:30:21 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:30:21 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 16:30:21 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:30:21 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 16:30:21 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:30:21 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 16:30:21 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:30:21 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:30:21 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 16:30:21 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 16:30:21 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (98e22037-912c-49d2-98ed-1476c4c337fc) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:30:21 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (98e22037-912c-49d2-98ed-1476c4c337fc) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:30:21 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (a688c952-58b5-4e72-b557-e3f6aa59fc9f) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:30:21 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (a688c952-58b5-4e72-b557-e3f6aa59fc9f) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:30:21 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (968f20d9-3011-4ae0-8aeb-b95644443b93) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:30:21 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (968f20d9-3011-4ae0-8aeb-b95644443b93) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:30:21 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (1e87cdd6-a8de-472d-9153-6a29d1e99877) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:30:21 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Updating volume 8b7f1136-3d20-4cf8-a521-6db1d3a0c8a8 with {'scheduled_at': datetime.datetime(2023, 4, 17, 16, 30, 21, 746112), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 16:30:21 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Volume 8b7f1136-3d20-4cf8-a521-6db1d3a0c8a8: re-scheduling SchedulerAPI.create_volume attempt 1 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 16:30:21 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Volume 2159f1f1-2564-4d84-8110-c1f7d843edb1: re-scheduling SchedulerAPI.create_volume attempt 1 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 16:30:21 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Volume 8b7f1136-3d20-4cf8-a521-6db1d3a0c8a8: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 16:30:21 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] deleating volume v_8b7f1136-3d20-4cf8-a521-6db1d3a0c8a8 {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 16:30:21 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] get snapshots for volume v_8b7f1136-3d20-4cf8-a521-6db1d3a0c8a8 {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 16:30:21 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_8b7f1136-3d20-4cf8-a521-6db1d3a0c8a8/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:30:21 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_8b7f1136-3d20-4cf8-a521-6db1d3a0c8a8/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:30:21 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Volume 2159f1f1-2564-4d84-8110-c1f7d843edb1: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 16:30:21 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] deleating volume v_2159f1f1-2564-4d84-8110-c1f7d843edb1 {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 16:30:21 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] get snapshots for volume v_2159f1f1-2564-4d84-8110-c1f7d843edb1 {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 16:30:21 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_2159f1f1-2564-4d84-8110-c1f7d843edb1/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:30:21 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_2159f1f1-2564-4d84-8110-c1f7d843edb1/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:30:21 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Volume reschedule parameters: True retry: {'num_attempts': 2, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 16:30:21 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Flow 'volume_create_manager' (29dad5d0-e988-49b6-af92-fc21216548da) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 16:30:21 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (ee09d421-0a04-428d-b136-daa84c67a8f4) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:30:21 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Volume reschedule parameters: True retry: {'num_attempts': 2, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 16:30:21 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Flow 'volume_create_manager' (6530cd85-6c71-42b4-86dd-de43ea2dfb9d) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 16:30:21 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (3d429a40-9c64-404f-86b3-68a0a8d4e377) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:30:22 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (ee09d421-0a04-428d-b136-daa84c67a8f4) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:27:59Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesDeleteCascade-Volume-2055115456',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=8b7f1136-3d20-4cf8-a521-6db1d3a0c8a8,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='abbfd52254f546b0bcfe289884d77f49',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:30:22Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:30:22Z,use_quota=True,user_id='6309fcbea41c471184457d276d32db73',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:30:22 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (ea985edf-0b1d-4ffc-ade1-837a9b4be8b3) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:30:22 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (ea985edf-0b1d-4ffc-ade1-837a9b4be8b3) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:30:22 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (0ab75f76-a544-46e8-a21a-9cdba7271dc7) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:30:22 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (0ab75f76-a544-46e8-a21a-9cdba7271dc7) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': '8b7f1136-3d20-4cf8-a521-6db1d3a0c8a8', 'volume_name': 'volume-8b7f1136-3d20-4cf8-a521-6db1d3a0c8a8', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:30:22 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (5d1ae647-1830-451d-b121-24ea22118891) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:30:22 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (5d1ae647-1830-451d-b121-24ea22118891) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:30:22 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (d2da08f2-a5d7-48f5-a044-18660301597a) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:30:22 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Volume 8b7f1136-3d20-4cf8-a521-6db1d3a0c8a8: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-8b7f1136-3d20-4cf8-a521-6db1d3a0c8a8', 'volume_size': 1} Apr 17 16:30:22 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] creating volume v_8b7f1136-3d20-4cf8-a521-6db1d3a0c8a8. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 16:30:22 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] create volume {'name': 'v_8b7f1136-3d20-4cf8-a521-6db1d3a0c8a8', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 16:30:22 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:30:22 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:30:22 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (3d429a40-9c64-404f-86b3-68a0a8d4e377) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:28:03Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesSnapshotListTestJSON-Volume-552939668',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=2159f1f1-2564-4d84-8110-c1f7d843edb1,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='c4b1a8a0074e4798bd385ee96e8ac8b7',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:30:22Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:30:22Z,use_quota=True,user_id='476228814d42467c880dce2b1c5c5113',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:30:22 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (cdbb57d0-c62f-4c33-aeb2-e64d3e362f8b) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:30:22 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (cdbb57d0-c62f-4c33-aeb2-e64d3e362f8b) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:30:22 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (3f4f1285-5b3c-438b-a500-33e76120465f) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:30:22 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (3f4f1285-5b3c-438b-a500-33e76120465f) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': '2159f1f1-2564-4d84-8110-c1f7d843edb1', 'volume_name': 'volume-2159f1f1-2564-4d84-8110-c1f7d843edb1', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:30:22 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (d23e7a46-645a-4d37-84fd-fd7a03a3c54d) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:30:22 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (d23e7a46-645a-4d37-84fd-fd7a03a3c54d) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:30:22 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (902eba17-4574-415c-ae23-cbb7f3931065) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:30:22 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Volume 2159f1f1-2564-4d84-8110-c1f7d843edb1: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-2159f1f1-2564-4d84-8110-c1f7d843edb1', 'volume_size': 1} Apr 17 16:30:22 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] creating volume v_2159f1f1-2564-4d84-8110-c1f7d843edb1. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 16:30:22 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] create volume {'name': 'v_2159f1f1-2564-4d84-8110-c1f7d843edb1', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 16:30:22 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:30:22 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:30:24 user cinder-volume[84696]: DEBUG cinder.utils [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.865(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:30:24 user cinder-volume[84696]: DEBUG cinder.utils [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_2159f1f1-2564-4d84-8110-c1f7d843edb1/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:30:24 user cinder-volume[84696]: DEBUG cinder.utils [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.630(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:30:24 user cinder-volume[84696]: DEBUG cinder.utils [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:30:24 user cinder-volume[84696]: DEBUG cinder.utils [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.881(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:30:24 user cinder-volume[84696]: DEBUG cinder.utils [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_8b7f1136-3d20-4cf8-a521-6db1d3a0c8a8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:30:24 user cinder-volume[84696]: DEBUG cinder.utils [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.594(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:30:24 user cinder-volume[84696]: DEBUG cinder.utils [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:30:27 user cinder-volume[84696]: DEBUG cinder.utils [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.699(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:30:27 user cinder-volume[84696]: DEBUG cinder.utils [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:30:27 user cinder-volume[84696]: DEBUG cinder.utils [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.938(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:30:27 user cinder-volume[84696]: DEBUG cinder.utils [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_2159f1f1-2564-4d84-8110-c1f7d843edb1/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:30:27 user cinder-volume[84696]: DEBUG cinder.utils [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 73.143(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:30:27 user cinder-volume[84696]: DEBUG cinder.utils [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_53029faa-77b2-467d-8a64-eb5ef129e41c/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:30:27 user cinder-volume[84696]: DEBUG cinder.utils [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.663(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:30:27 user cinder-volume[84696]: DEBUG cinder.utils [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:30:27 user cinder-volume[84696]: DEBUG cinder.utils [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 72.979(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:30:27 user cinder-volume[84696]: DEBUG cinder.utils [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:30:27 user cinder-volume[84696]: DEBUG cinder.utils [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 137.757(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:30:27 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:30:27 user cinder-volume[84696]: INFO cinder.message.api [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Creating message record for request_id = req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d Apr 17 16:30:27 user cinder-volume[84696]: DEBUG cinder.utils [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.958(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:30:27 user cinder-volume[84696]: DEBUG cinder.utils [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_8b7f1136-3d20-4cf8-a521-6db1d3a0c8a8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:30:27 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 16:30:27 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (731f3428-59f2-4f05-be5a-40a8f39522a5) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 16:30:27 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 16:30:27 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:28:10Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumeQuotasAdminTestJSON-Volume-332841798',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=f6c0e341-6f00-4e32-bd07-31f954c1ad8a,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='6e4c2ec9177349a08ce53e07b24ad79c',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:28:10Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:28:10Z,use_quota=True,user_id='ab5017da30ac4d47ba665ff135deb309',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 16:30:27 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:28:10Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumeQuotasAdminTestJSON-Volume-332841798',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=f6c0e341-6f00-4e32-bd07-31f954c1ad8a,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='6e4c2ec9177349a08ce53e07b24ad79c',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:28:10Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:28:10Z,use_quota=True,user_id='ab5017da30ac4d47ba665ff135deb309',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(f6c0e341-6f00-4e32-bd07-31f954c1ad8a),volume_id=f6c0e341-6f00-4e32-bd07-31f954c1ad8a,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': 'f6c0e341-6f00-4e32-bd07-31f954c1ad8a', 'volume_name': 'volume-f6c0e341-6f00-4e32-bd07-31f954c1ad8a', 'volume_size': 1}} Apr 17 16:30:27 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'retry': {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']}, 'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': 'f6c0e341-6f00-4e32-bd07-31f954c1ad8a', 'volume': {'id': 'f6c0e341-6f00-4e32-bd07-31f954c1ad8a', '_name_id': None, 'ec2_id': None, 'user_id': 'ab5017da30ac4d47ba665ff135deb309', 'project_id': '6e4c2ec9177349a08ce53e07b24ad79c', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-VolumeQuotasAdminTestJSON-Volume-332841798', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T16:28:09.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-f6c0e341-6f00-4e32-bd07-31f954c1ad8a', 'name_id': 'f6c0e341-6f00-4e32-bd07-31f954c1ad8a', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumeQuotasAdminTestJSON-Volume-332841798', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '6e4c2ec9177349a08ce53e07b24ad79c', 'qos_specs': None, 'reservations': ['f21078bb-20eb-4029-93a1-5b4cb253f25d', 'bbd80d77-344a-4970-b1bb-2892c08c1c72', 'a38f5177-62fc-4eb7-b072-f15a54689ed0', 'b5e5e1fe-ca2e-48fa-b5e1-b6f6d98fe349'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'ab5017da30ac4d47ba665ff135deb309', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumeQuotasAdminTestJSON-Volume-332841798', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '6e4c2ec9177349a08ce53e07b24ad79c', 'qos_specs': None, 'reservations': ['f21078bb-20eb-4029-93a1-5b4cb253f25d', 'bbd80d77-344a-4970-b1bb-2892c08c1c72', 'a38f5177-62fc-4eb7-b072-f15a54689ed0', 'b5e5e1fe-ca2e-48fa-b5e1-b6f6d98fe349'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'ab5017da30ac4d47ba665ff135deb309', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': 'ab5017da30ac4d47ba665ff135deb309', 'metadata': {}, 'qos_specs': None}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(f6c0e341-6f00-4e32-bd07-31f954c1ad8a),volume_id=f6c0e341-6f00-4e32-bd07-31f954c1ad8a,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:28:10Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumeQuotasAdminTestJSON-Volume-332841798',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=f6c0e341-6f00-4e32-bd07-31f954c1ad8a,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='6e4c2ec9177349a08ce53e07b24ad79c',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:28:10Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:28:10Z,use_quota=True,user_id='ab5017da30ac4d47ba665ff135deb309',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 16:30:27 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:28:10Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumeQuotasAdminTestJSON-Volume-332841798',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=f6c0e341-6f00-4e32-bd07-31f954c1ad8a,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='6e4c2ec9177349a08ce53e07b24ad79c',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:28:10Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:28:10Z,use_quota=True,user_id='ab5017da30ac4d47ba665ff135deb309',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:28:10Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumeQuotasAdminTestJSON-Volume-332841798',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=f6c0e341-6f00-4e32-bd07-31f954c1ad8a,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='6e4c2ec9177349a08ce53e07b24ad79c',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:28:10Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:28:10Z,use_quota=True,user_id='ab5017da30ac4d47ba665ff135deb309',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 16:30:27 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 16:30:27 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 16:30:27 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:30:27 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 16:30:27 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:30:27 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 16:30:27 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:30:27 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 16:30:27 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:30:27 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 16:30:27 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:30:27 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 16:30:27 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:30:27 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 16:30:27 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:30:27 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 16:30:27 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:30:27 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:30:27 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 16:30:27 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 16:30:27 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (731f3428-59f2-4f05-be5a-40a8f39522a5) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:30:27 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (731f3428-59f2-4f05-be5a-40a8f39522a5) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:30:27 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (e84dbb19-68b3-46b9-a71c-47dfc92a01e4) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:30:27 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (e84dbb19-68b3-46b9-a71c-47dfc92a01e4) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:30:27 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (64f343b0-d0c4-416d-ba0c-c437b0cfcc5d) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:30:27 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (64f343b0-d0c4-416d-ba0c-c437b0cfcc5d) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:30:27 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (17697f70-2399-4d33-91c6-50d7e87f28f6) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:30:27 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Updating volume f6c0e341-6f00-4e32-bd07-31f954c1ad8a with {'scheduled_at': datetime.datetime(2023, 4, 17, 16, 30, 27, 810955), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 16:30:27 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Volume f6c0e341-6f00-4e32-bd07-31f954c1ad8a: re-scheduling SchedulerAPI.create_volume attempt 1 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 16:30:27 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Volume f6c0e341-6f00-4e32-bd07-31f954c1ad8a: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 16:30:27 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] deleating volume v_f6c0e341-6f00-4e32-bd07-31f954c1ad8a {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 16:30:27 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] get snapshots for volume v_f6c0e341-6f00-4e32-bd07-31f954c1ad8a {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 16:30:27 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_f6c0e341-6f00-4e32-bd07-31f954c1ad8a/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:30:27 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_f6c0e341-6f00-4e32-bd07-31f954c1ad8a/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:30:27 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Volume reschedule parameters: True retry: {'num_attempts': 2, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 16:30:27 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Flow 'volume_create_manager' (a8598034-edf8-495f-88e2-8b4c9e8e4fcf) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 16:30:27 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (ef826f8e-a283-47ce-92c4-d3525c4caa4a) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:30:27 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (ef826f8e-a283-47ce-92c4-d3525c4caa4a) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:28:10Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumeQuotasAdminTestJSON-Volume-332841798',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=f6c0e341-6f00-4e32-bd07-31f954c1ad8a,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='6e4c2ec9177349a08ce53e07b24ad79c',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:30:28Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:30:28Z,use_quota=True,user_id='ab5017da30ac4d47ba665ff135deb309',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:30:27 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (c5686eb2-064c-40f9-8c44-e9ea3b76a86b) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:30:27 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (c5686eb2-064c-40f9-8c44-e9ea3b76a86b) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:30:27 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (f390019f-5182-40de-a488-958e73806d06) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:30:27 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (f390019f-5182-40de-a488-958e73806d06) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': 'f6c0e341-6f00-4e32-bd07-31f954c1ad8a', 'volume_name': 'volume-f6c0e341-6f00-4e32-bd07-31f954c1ad8a', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:30:28 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (e4d0c2e2-7545-40cb-96b1-5a69c20aef10) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:30:28 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (e4d0c2e2-7545-40cb-96b1-5a69c20aef10) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:30:28 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (c9402295-b2c7-45b7-a764-5f181be36aae) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:30:28 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Volume f6c0e341-6f00-4e32-bd07-31f954c1ad8a: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-f6c0e341-6f00-4e32-bd07-31f954c1ad8a', 'volume_size': 1} Apr 17 16:30:28 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] creating volume v_f6c0e341-6f00-4e32-bd07-31f954c1ad8a. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 16:30:28 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] create volume {'name': 'v_f6c0e341-6f00-4e32-bd07-31f954c1ad8a', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 16:30:28 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:30:28 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:30:30 user cinder-volume[84696]: DEBUG cinder.utils [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 72.982(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:30:30 user cinder-volume[84696]: DEBUG cinder.utils [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:30:30 user cinder-volume[84696]: DEBUG cinder.utils [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.978(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:30:30 user cinder-volume[84696]: DEBUG cinder.utils [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_f6c0e341-6f00-4e32-bd07-31f954c1ad8a/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:30:30 user cinder-volume[84696]: DEBUG cinder.utils [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.805(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:30:30 user cinder-volume[84696]: DEBUG cinder.utils [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:30:30 user cinder-volume[84696]: DEBUG cinder.utils [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 73.148(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:30:30 user cinder-volume[84696]: DEBUG cinder.utils [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_35a67ca3-0f17-4a6c-a359-9173ce5f1aca/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:30:33 user cinder-volume[84696]: DEBUG cinder.utils [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.092(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:30:33 user cinder-volume[84696]: DEBUG cinder.utils [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_8b7f1136-3d20-4cf8-a521-6db1d3a0c8a8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:30:33 user cinder-volume[84696]: DEBUG cinder.utils [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 11.805(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:30:33 user cinder-volume[84696]: DEBUG cinder.utils [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:30:33 user cinder-volume[84696]: DEBUG cinder.utils [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.083(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:30:33 user cinder-volume[84696]: DEBUG cinder.utils [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_2159f1f1-2564-4d84-8110-c1f7d843edb1/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:30:33 user cinder-volume[84696]: DEBUG cinder.utils [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 11.846(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:30:33 user cinder-volume[84696]: DEBUG cinder.utils [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:30:33 user cinder-volume[84696]: DEBUG cinder.utils [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.052(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:30:33 user cinder-volume[84696]: DEBUG cinder.utils [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_f6c0e341-6f00-4e32-bd07-31f954c1ad8a/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:30:33 user cinder-volume[84696]: DEBUG cinder.utils [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.877(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:30:33 user cinder-volume[84696]: DEBUG cinder.utils [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:30:40 user cinder-volume[84696]: DEBUG cinder.utils [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.352(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:30:40 user cinder-volume[84696]: DEBUG cinder.utils [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_f6c0e341-6f00-4e32-bd07-31f954c1ad8a/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:30:40 user cinder-volume[84696]: DEBUG cinder.utils [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.179(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:30:40 user cinder-volume[84696]: DEBUG cinder.utils [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:30:40 user cinder-volume[84696]: DEBUG cinder.utils [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 138.900(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:30:40 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_94610127-c8b8-4df4-9e39-a1f41b5a9b98/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:30:40 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (66979d03-bf50-4ec4-a32e-e4b96a752e54) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 16:30:40 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (f8f3a404-fc9a-498c-afaf-d9ab99c14c6c) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:30:40 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (f8f3a404-fc9a-498c-afaf-d9ab99c14c6c) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:30:40 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Flow 'volume_create_manager' (eee06777-deee-475c-a4c0-0678fb763ca9) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 16:30:40 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-d3245848-3d3a-4130-8413-3f811df2f47d tempest-AbsoluteLimitsTests-986281014 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 16:30:40 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 16:30:40 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 16:30:40 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 16:30:40 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 16:30:40 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 16:30:40 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 16:30:40 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 16:30:40 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 16:30:40 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 16:30:40 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 16:30:40 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 16:30:40 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 16:30:40 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 16:30:40 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 16:30:40 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 16:30:40 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 16:30:40 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 16:30:40 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 16:30:40 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 16:30:40 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 16:30:40 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 16:30:40 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 16:30:40 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 16:30:40 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:30:40 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 16:30:40 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:30:40 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 16:30:40 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:30:40 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 16:30:40 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:30:40 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 16:30:40 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:30:40 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 16:30:40 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:30:40 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 16:30:40 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:30:40 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 16:30:40 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:30:40 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:30:40 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 16:30:40 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 16:30:40 user cinder-volume[84696]: DEBUG cinder.utils [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 138.912(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:30:40 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_02aba338-fa31-4e46-8fcf-7f86191e44d1/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:30:40 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (64080c9c-bb96-4290-97cf-4bc092461a3e) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 16:30:40 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (cac58a2b-56c0-4720-bcba-d46d30f9eb16) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:30:40 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (cac58a2b-56c0-4720-bcba-d46d30f9eb16) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:30:40 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Flow 'volume_create_manager' (ae635d45-5956-4347-8615-a522c49f73fd) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 16:30:40 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-476320d2-f85f-4dd0-bac3-38a67c9e0e47 tempest-GroupsTest-789764105 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 16:30:40 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 16:30:40 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 16:30:40 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 16:30:40 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 16:30:40 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 16:30:40 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 16:30:40 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 16:30:40 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 16:30:40 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 16:30:40 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 16:30:40 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 16:30:40 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 16:30:40 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 16:30:40 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 16:30:40 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 16:30:40 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 16:30:40 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 16:30:40 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 16:30:40 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 16:30:40 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 16:30:40 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 16:30:40 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 16:30:40 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 16:30:40 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:30:40 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 16:30:40 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:30:40 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 16:30:40 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:30:40 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 16:30:40 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:30:40 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 16:30:40 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:30:40 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 16:30:40 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:30:40 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 16:30:40 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:30:40 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 16:30:40 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:30:40 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:30:40 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 16:30:40 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 16:30:43 user cinder-volume[84696]: DEBUG cinder.utils [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.179(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:30:43 user cinder-volume[84696]: DEBUG cinder.utils [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:30:43 user cinder-volume[84696]: DEBUG cinder.utils [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.458(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:30:43 user cinder-volume[84696]: DEBUG cinder.utils [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_2159f1f1-2564-4d84-8110-c1f7d843edb1/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:30:43 user cinder-volume[84696]: DEBUG cinder.utils [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.471(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:30:43 user cinder-volume[84696]: DEBUG cinder.utils [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_8b7f1136-3d20-4cf8-a521-6db1d3a0c8a8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:30:43 user cinder-volume[84696]: DEBUG cinder.utils [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.223(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:30:43 user cinder-volume[84696]: DEBUG cinder.utils [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:30:44 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=17,cinder:UPDATE=4,cinder:INSERT=3 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:30:49 user cinder-volume[84696]: DEBUG cinder.utils [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.568(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:30:49 user cinder-volume[84696]: DEBUG cinder.utils [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_f6c0e341-6f00-4e32-bd07-31f954c1ad8a/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:30:49 user cinder-volume[84696]: DEBUG cinder.utils [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.394(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:30:49 user cinder-volume[84696]: DEBUG cinder.utils [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:30:54 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=20,cinder:INSERT=3,cinder:UPDATE=9 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:31:02 user cinder-volume[84696]: DEBUG cinder.utils [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 40.219(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:31:02 user cinder-volume[84696]: DEBUG cinder.utils [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:31:02 user cinder-volume[84696]: DEBUG cinder.utils [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 40.500(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:31:02 user cinder-volume[84696]: DEBUG cinder.utils [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_2159f1f1-2564-4d84-8110-c1f7d843edb1/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:31:02 user cinder-volume[84696]: DEBUG cinder.utils [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 40.512(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:31:02 user cinder-volume[84696]: DEBUG cinder.utils [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_8b7f1136-3d20-4cf8-a521-6db1d3a0c8a8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:31:02 user cinder-volume[84696]: DEBUG cinder.utils [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 40.264(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:31:02 user cinder-volume[84696]: DEBUG cinder.utils [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:31:08 user cinder-volume[84696]: DEBUG cinder.utils [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 40.608(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:31:08 user cinder-volume[84696]: DEBUG cinder.utils [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_f6c0e341-6f00-4e32-bd07-31f954c1ad8a/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:31:08 user cinder-volume[84696]: DEBUG cinder.utils [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 40.433(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:31:08 user cinder-volume[84696]: DEBUG cinder.utils [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:31:14 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=1 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:31:17 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=7,cinder:UPDATE=3 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:31:17 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Volume reschedule parameters: True retry: {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 16:31:17 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Flow 'volume_create_manager' (9f70d528-7342-4973-8e87-91c98296c96e) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 16:31:17 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (86c21eaf-9bad-46cc-b34e-900c1110fc18) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:31:17 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (86c21eaf-9bad-46cc-b34e-900c1110fc18) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:31:18Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-CreateVolumesFromSnapshotTest-Volume-2022997390',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=bf85ce23-1db3-4881-bd7a-b20b520718b8,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='93bbb103c036439da2861c7be15c3e7b',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:31:18Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:31:18Z,use_quota=True,user_id='ae7cdf32438c44b6b3f428e9e0142af1',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:31:17 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (e593ecf2-32cd-45c5-818c-0bdc7f96fd98) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:31:17 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (e593ecf2-32cd-45c5-818c-0bdc7f96fd98) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:31:17 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (e37a7628-6d0e-41e9-a099-bdb921c727f7) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:31:17 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (e37a7628-6d0e-41e9-a099-bdb921c727f7) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': 'bf85ce23-1db3-4881-bd7a-b20b520718b8', 'volume_name': 'volume-bf85ce23-1db3-4881-bd7a-b20b520718b8', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:31:17 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (5057cb85-82cb-42d5-a41c-97e2d9560856) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:31:17 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (5057cb85-82cb-42d5-a41c-97e2d9560856) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:31:17 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (b7c86137-f116-4878-a7bd-6a6acf4ae801) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:31:17 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Volume bf85ce23-1db3-4881-bd7a-b20b520718b8: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-bf85ce23-1db3-4881-bd7a-b20b520718b8', 'volume_size': 1} Apr 17 16:31:17 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] creating volume v_bf85ce23-1db3-4881-bd7a-b20b520718b8. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 16:31:17 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] create volume {'name': 'v_bf85ce23-1db3-4881-bd7a-b20b520718b8', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 16:31:17 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:31:17 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:31:18 user cinder-volume[84696]: DEBUG cinder.utils [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 0.487(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:31:18 user cinder-volume[84696]: DEBUG cinder.utils [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:31:21 user cinder-volume[84696]: DEBUG cinder.utils [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 3.563(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:31:21 user cinder-volume[84696]: DEBUG cinder.utils [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:31:21 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Volume reschedule parameters: True retry: {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 16:31:21 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Flow 'volume_create_manager' (ce92c14e-8b17-4992-905a-0fe1f43b987c) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 16:31:21 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (3702dcb6-db1c-4ced-a2fa-3f0bb0d53193) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:31:21 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (3702dcb6-db1c-4ced-a2fa-3f0bb0d53193) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:31:21Z,deleted=False,deleted_at=None,display_description=None,display_name='CinderUnicodeTest塵㼗‽',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=7384f51d-c3a2-4eea-a968-ec043946989f,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='9ccd4e98adcb4fb18f651ead3736305a',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:31:22Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:31:22Z,use_quota=True,user_id='d3c58fb72fb443e19315cc15a76767fe',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:31:21 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (6d2f64ff-fe0b-4f90-b3bd-5481f65a4834) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:31:21 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (6d2f64ff-fe0b-4f90-b3bd-5481f65a4834) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:31:21 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (1a3641ba-e8e5-41de-82c5-13ae38a18b5c) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:31:21 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (1a3641ba-e8e5-41de-82c5-13ae38a18b5c) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': '7384f51d-c3a2-4eea-a968-ec043946989f', 'volume_name': 'volume-7384f51d-c3a2-4eea-a968-ec043946989f', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:31:21 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (36d794c2-c2f3-4d60-83e4-7e5b252762bf) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:31:21 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (36d794c2-c2f3-4d60-83e4-7e5b252762bf) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:31:21 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (35759e7e-48a4-4d93-902f-a7a53b329f1c) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:31:21 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Volume 7384f51d-c3a2-4eea-a968-ec043946989f: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-7384f51d-c3a2-4eea-a968-ec043946989f', 'volume_size': 1} Apr 17 16:31:21 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] creating volume v_7384f51d-c3a2-4eea-a968-ec043946989f. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 16:31:21 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] create volume {'name': 'v_7384f51d-c3a2-4eea-a968-ec043946989f', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 16:31:21 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:31:21 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:31:24 user cinder-volume[84696]: DEBUG cinder.utils [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.717(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:31:24 user cinder-volume[84696]: DEBUG cinder.utils [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:31:24 user cinder-volume[84696]: DEBUG cinder.utils [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 138.974(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:31:24 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_2e8580be-c7f2-4d3a-9007-d192a5b029b3/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:31:24 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (ab9fc0d2-624c-4bd2-a9aa-de04049a75b9) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 16:31:24 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (f2c6c15e-04e0-4e2c-876e-9a3ee61201b2) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:31:24 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (f2c6c15e-04e0-4e2c-876e-9a3ee61201b2) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:31:24 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Flow 'volume_create_manager' (b5c76848-fd44-4b33-80b0-8ddd572ad41b) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 16:31:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-f30cda1b-0a2e-45d5-b75e-6fcb7871e267 tempest-VolumesNegativeTest-1370732729 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 16:31:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 16:31:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 16:31:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 16:31:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 16:31:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 16:31:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 16:31:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 16:31:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 16:31:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 16:31:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 16:31:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 16:31:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 16:31:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 16:31:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 16:31:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 16:31:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 16:31:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 16:31:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 16:31:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 16:31:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 16:31:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 16:31:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 16:31:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 16:31:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:31:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 16:31:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:31:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 16:31:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:31:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 16:31:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:31:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 16:31:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:31:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 16:31:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:31:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 16:31:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:31:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 16:31:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:31:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:31:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 16:31:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 16:31:26 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Volume reschedule parameters: True retry: {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 16:31:26 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Flow 'volume_create_manager' (c616e438-ecd0-42ad-8cd3-155c73f9aa09) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 16:31:26 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (b34b2727-7fbf-4685-8368-a6ec3373855b) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:31:26 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (b34b2727-7fbf-4685-8368-a6ec3373855b) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:31:26Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumeQuotasAdminTestJSON-Volume-2080868017',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=480a94a0-d1c9-46c0-bf9f-c42f907b4b82,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='6e4c2ec9177349a08ce53e07b24ad79c',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:31:26Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:31:26Z,use_quota=True,user_id='ab5017da30ac4d47ba665ff135deb309',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:31:26 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (8e29d783-ca28-40a8-aa00-4942cc3231e3) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:31:26 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (8e29d783-ca28-40a8-aa00-4942cc3231e3) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:31:26 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (7fbe4791-10fa-4579-af06-c077988388ea) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:31:26 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (7fbe4791-10fa-4579-af06-c077988388ea) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': '480a94a0-d1c9-46c0-bf9f-c42f907b4b82', 'volume_name': 'volume-480a94a0-d1c9-46c0-bf9f-c42f907b4b82', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:31:26 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (2a754555-aa79-41c5-8b60-17f320357232) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:31:26 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (2a754555-aa79-41c5-8b60-17f320357232) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:31:26 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (1b692ae0-b6bd-4f39-b4c3-14746eef8df0) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:31:26 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Volume 480a94a0-d1c9-46c0-bf9f-c42f907b4b82: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-480a94a0-d1c9-46c0-bf9f-c42f907b4b82', 'volume_size': 1} Apr 17 16:31:26 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] creating volume v_480a94a0-d1c9-46c0-bf9f-c42f907b4b82. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 16:31:26 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] create volume {'name': 'v_480a94a0-d1c9-46c0-bf9f-c42f907b4b82', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 16:31:26 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:31:26 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:31:27 user cinder-volume[84696]: DEBUG cinder.utils [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 1.311(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:31:27 user cinder-volume[84696]: DEBUG cinder.utils [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:31:27 user cinder-volume[84696]: DEBUG cinder.utils [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 9.705(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:31:27 user cinder-volume[84696]: DEBUG cinder.utils [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:31:27 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 139.098(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:31:27 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:31:27 user cinder-volume[84696]: ERROR oslo_service.periodic_task [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Error during VolumeManager.publish_service_capabilities: TypeError: 'NoneType' object is not subscriptable Apr 17 16:31:27 user cinder-volume[84696]: ERROR oslo_service.periodic_task Traceback (most recent call last): Apr 17 16:31:27 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/usr/local/lib/python3.10/dist-packages/oslo_service/periodic_task.py", line 216, in run_periodic_tasks Apr 17 16:31:27 user cinder-volume[84696]: ERROR oslo_service.periodic_task task(self, context) Apr 17 16:31:27 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/manager.py", line 2876, in publish_service_capabilities Apr 17 16:31:27 user cinder-volume[84696]: ERROR oslo_service.periodic_task self._report_driver_status(context) Apr 17 16:31:27 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/manager.py", line 2755, in _report_driver_status Apr 17 16:31:27 user cinder-volume[84696]: ERROR oslo_service.periodic_task volume_stats = get_stats() Apr 17 16:31:27 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/usr/local/lib/python3.10/dist-packages/oslo_utils/timeutils.py", line 319, in wrapper Apr 17 16:31:27 user cinder-volume[84696]: ERROR oslo_service.periodic_task result = func(*args, **kwargs) Apr 17 16:31:27 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/manager.py", line 2753, in get_stats Apr 17 16:31:27 user cinder-volume[84696]: ERROR oslo_service.periodic_task return self.driver.get_volume_stats(refresh=True) Apr 17 16:31:27 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/driver.py", line 666, in get_volume_stats Apr 17 16:31:27 user cinder-volume[84696]: ERROR oslo_service.periodic_task self._update_volume_stats() Apr 17 16:31:27 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 726, in _update_volume_stats Apr 17 16:31:27 user cinder-volume[84696]: ERROR oslo_service.periodic_task pool_stats = self.ra.get_pool_stats() Apr 17 16:31:27 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 897, in get_pool_stats Apr 17 16:31:27 user cinder-volume[84696]: ERROR oslo_service.periodic_task if not resp["error"] and resp["code"] == 200: Apr 17 16:31:27 user cinder-volume[84696]: ERROR oslo_service.periodic_task TypeError: 'NoneType' object is not subscriptable Apr 17 16:31:27 user cinder-volume[84696]: ERROR oslo_service.periodic_task Apr 17 16:31:27 user cinder-volume[84696]: DEBUG cinder.utils [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.794(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:31:27 user cinder-volume[84696]: DEBUG cinder.utils [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:31:27 user cinder-volume[84696]: WARNING oslo.service.loopingcall [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Function 'cinder.service.Service.periodic_tasks' run outlasted interval by 79.10 sec Apr 17 16:31:27 user cinder-volume[84696]: DEBUG oslo_service.periodic_task [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Running periodic task VolumeManager.publish_service_capabilities {{(pid=84696) run_periodic_tasks /usr/local/lib/python3.10/dist-packages/oslo_service/periodic_task.py:210}} Apr 17 16:31:27 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Updating volume stats {{(pid=84696) _update_volume_stats /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:724}} Apr 17 16:31:27 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Get pool Pool-0 fsprops {{(pid=84696) get_pool_stats /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:894}} Apr 17 16:31:27 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:31:27 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:31:30 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 3.065(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:31:30 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:31:30 user cinder-volume[84696]: DEBUG cinder.utils [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 4.386(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:31:30 user cinder-volume[84696]: DEBUG cinder.utils [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:31:33 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.136(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:31:33 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:31:33 user cinder-volume[84696]: DEBUG cinder.utils [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 11.935(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:31:33 user cinder-volume[84696]: DEBUG cinder.utils [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:31:33 user cinder-volume[84696]: DEBUG cinder.utils [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 138.833(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:31:33 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:31:33 user cinder-volume[84696]: INFO cinder.message.api [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Creating message record for request_id = req-8d765f93-bce0-4815-ac07-22b8b3bd2556 Apr 17 16:31:33 user cinder-volume[84696]: DEBUG cinder.utils [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 139.006(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:31:33 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_53029faa-77b2-467d-8a64-eb5ef129e41c/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:31:33 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (33495df2-1373-45cd-b3ea-3fcb107e4d47) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 16:31:33 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (2a596980-4c2a-4f58-80d8-0259ceffa0aa) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:31:33 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (2a596980-4c2a-4f58-80d8-0259ceffa0aa) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:31:33 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Flow 'volume_create_manager' (4e5cd157-86df-4a33-91e8-ec5f2f8fba8c) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 16:31:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 16:31:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 16:31:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 16:31:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 16:31:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 16:31:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 16:31:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 16:31:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 16:31:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 16:31:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 16:31:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 16:31:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 16:31:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 16:31:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 16:31:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 16:31:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 16:31:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 16:31:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 16:31:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 16:31:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 16:31:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 16:31:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 16:31:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 16:31:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 16:31:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:31:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 16:31:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:31:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 16:31:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:31:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 16:31:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:31:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 16:31:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:31:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 16:31:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:31:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 16:31:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:31:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 16:31:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:31:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:31:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 16:31:33 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 16:31:33 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 16:31:33 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (c2eaf81a-f6a5-485f-a7f2-0955f699d5ac) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 16:31:33 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 16:31:33 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:24:42Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesDeleteCascade-Volume-1549123617',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=53029faa-77b2-467d-8a64-eb5ef129e41c,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='abbfd52254f546b0bcfe289884d77f49',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:29:15Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:29:15Z,use_quota=True,user_id='6309fcbea41c471184457d276d32db73',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 16:31:33 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:24:42Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesDeleteCascade-Volume-1549123617',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=53029faa-77b2-467d-8a64-eb5ef129e41c,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='abbfd52254f546b0bcfe289884d77f49',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:29:15Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:29:15Z,use_quota=True,user_id='6309fcbea41c471184457d276d32db73',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(53029faa-77b2-467d-8a64-eb5ef129e41c),volume_id=53029faa-77b2-467d-8a64-eb5ef129e41c,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': '53029faa-77b2-467d-8a64-eb5ef129e41c', 'volume_name': 'volume-53029faa-77b2-467d-8a64-eb5ef129e41c', 'volume_size': 1}} Apr 17 16:31:33 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': '53029faa-77b2-467d-8a64-eb5ef129e41c', 'volume': {'id': '53029faa-77b2-467d-8a64-eb5ef129e41c', '_name_id': None, 'ec2_id': None, 'user_id': '6309fcbea41c471184457d276d32db73', 'project_id': 'abbfd52254f546b0bcfe289884d77f49', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-VolumesDeleteCascade-Volume-1549123617', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'admin_metadata': {}, 'glance_metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T16:24:42.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-53029faa-77b2-467d-8a64-eb5ef129e41c', 'name_id': '53029faa-77b2-467d-8a64-eb5ef129e41c', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumesDeleteCascade-Volume-1549123617', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': 'abbfd52254f546b0bcfe289884d77f49', 'qos_specs': None, 'reservations': ['15abf3ac-c420-4557-ab29-de463d495550', '4070e169-cef7-4670-80f8-a805062b3ed7', '598ac100-d775-4c42-82f1-6a2eb9c17061', 'c035551f-9406-4ac7-a6e8-e732d18bff88'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': '6309fcbea41c471184457d276d32db73', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumesDeleteCascade-Volume-1549123617', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': 'abbfd52254f546b0bcfe289884d77f49', 'qos_specs': None, 'reservations': ['15abf3ac-c420-4557-ab29-de463d495550', '4070e169-cef7-4670-80f8-a805062b3ed7', '598ac100-d775-4c42-82f1-6a2eb9c17061', 'c035551f-9406-4ac7-a6e8-e732d18bff88'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': '6309fcbea41c471184457d276d32db73', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': '6309fcbea41c471184457d276d32db73', 'metadata': {}, 'qos_specs': None, 'retry': {'num_attempts': 3, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']}}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(53029faa-77b2-467d-8a64-eb5ef129e41c),volume_id=53029faa-77b2-467d-8a64-eb5ef129e41c,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:24:42Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesDeleteCascade-Volume-1549123617',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=53029faa-77b2-467d-8a64-eb5ef129e41c,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='abbfd52254f546b0bcfe289884d77f49',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:29:15Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:29:15Z,use_quota=True,user_id='6309fcbea41c471184457d276d32db73',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 16:31:33 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:24:42Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesDeleteCascade-Volume-1549123617',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=53029faa-77b2-467d-8a64-eb5ef129e41c,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='abbfd52254f546b0bcfe289884d77f49',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:29:15Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:29:15Z,use_quota=True,user_id='6309fcbea41c471184457d276d32db73',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:24:42Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesDeleteCascade-Volume-1549123617',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=53029faa-77b2-467d-8a64-eb5ef129e41c,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='abbfd52254f546b0bcfe289884d77f49',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:29:15Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:29:15Z,use_quota=True,user_id='6309fcbea41c471184457d276d32db73',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 16:31:33 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 16:31:33 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 16:31:33 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:31:33 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 16:31:33 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:31:33 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 16:31:33 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:31:33 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 16:31:33 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:31:33 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 16:31:33 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:31:33 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 16:31:33 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:31:33 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 16:31:33 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:31:33 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 16:31:33 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:31:33 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:31:33 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 16:31:33 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 16:31:33 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (c2eaf81a-f6a5-485f-a7f2-0955f699d5ac) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:31:33 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (c2eaf81a-f6a5-485f-a7f2-0955f699d5ac) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:31:33 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (50399269-425d-4934-9081-95e06fdd54f8) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:31:33 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (50399269-425d-4934-9081-95e06fdd54f8) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:31:33 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (33fe0d1c-f2e3-4c6c-8f74-66cb120b2949) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:31:33 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (33fe0d1c-f2e3-4c6c-8f74-66cb120b2949) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:31:33 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (97deb593-97b0-4a98-b1d1-56dbf3eaf113) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:31:33 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Updating volume 53029faa-77b2-467d-8a64-eb5ef129e41c with {'scheduled_at': datetime.datetime(2023, 4, 17, 16, 31, 33, 680231), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 16:31:33 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Volume 53029faa-77b2-467d-8a64-eb5ef129e41c: re-scheduling SchedulerAPI.create_volume attempt 3 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 16:31:33 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Volume 53029faa-77b2-467d-8a64-eb5ef129e41c: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 16:31:33 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] deleating volume v_53029faa-77b2-467d-8a64-eb5ef129e41c {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 16:31:33 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] get snapshots for volume v_53029faa-77b2-467d-8a64-eb5ef129e41c {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 16:31:33 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_53029faa-77b2-467d-8a64-eb5ef129e41c/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:31:33 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_53029faa-77b2-467d-8a64-eb5ef129e41c/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:31:36 user cinder-volume[84696]: DEBUG cinder.utils [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 74.588(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:31:36 user cinder-volume[84696]: DEBUG cinder.utils [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:31:36 user cinder-volume[84696]: DEBUG cinder.utils [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.962(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:31:36 user cinder-volume[84696]: DEBUG cinder.utils [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_53029faa-77b2-467d-8a64-eb5ef129e41c/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:31:36 user cinder-volume[84696]: DEBUG cinder.utils [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 139.003(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:31:36 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_35a67ca3-0f17-4a6c-a359-9173ce5f1aca/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:31:36 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (ace27a89-4176-490d-a4aa-fc8ee0980b04) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 16:31:36 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (fa74d58f-3eab-46df-a605-41884a549be8) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:31:36 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (fa74d58f-3eab-46df-a605-41884a549be8) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:31:36 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Flow 'volume_create_manager' (2d7a2364-63fa-4118-bdb1-5126c469be94) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 16:31:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 16:31:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 16:31:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 16:31:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 16:31:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 16:31:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 16:31:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 16:31:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 16:31:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 16:31:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 16:31:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 16:31:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 16:31:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 16:31:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 16:31:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 16:31:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 16:31:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 16:31:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 16:31:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 16:31:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 16:31:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 16:31:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 16:31:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 16:31:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 16:31:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:31:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 16:31:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:31:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 16:31:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:31:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 16:31:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:31:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 16:31:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:31:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 16:31:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:31:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 16:31:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:31:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 16:31:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:31:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:31:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 16:31:36 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 16:31:36 user cinder-volume[84696]: DEBUG cinder.utils [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 74.652(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:31:36 user cinder-volume[84696]: DEBUG cinder.utils [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:31:36 user cinder-volume[84696]: DEBUG cinder.utils [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 18.945(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:31:36 user cinder-volume[84696]: DEBUG cinder.utils [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:31:36 user cinder-volume[84696]: DEBUG cinder.utils [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 138.864(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:31:36 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:31:36 user cinder-volume[84696]: INFO cinder.message.api [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Creating message record for request_id = req-61f60e22-802a-4068-b929-14f669a47260 Apr 17 16:31:36 user cinder-volume[84696]: DEBUG cinder.utils [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 74.906(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:31:36 user cinder-volume[84696]: DEBUG cinder.utils [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_8b7f1136-3d20-4cf8-a521-6db1d3a0c8a8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:31:36 user cinder-volume[84696]: DEBUG cinder.utils [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 10.557(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:31:36 user cinder-volume[84696]: DEBUG cinder.utils [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:31:36 user cinder-volume[84696]: DEBUG cinder.utils [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 74.896(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:31:36 user cinder-volume[84696]: DEBUG cinder.utils [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_2159f1f1-2564-4d84-8110-c1f7d843edb1/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:31:36 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 16:31:36 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (a6908629-af82-4147-999b-21be37e1797c) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 16:31:36 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 16:31:36 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:24:45Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-SnapshotsActionsTest-Volume-1453311494',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=35a67ca3-0f17-4a6c-a359-9173ce5f1aca,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='5b416855df3047d79fd4d27b2b1695b8',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:29:18Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:29:18Z,use_quota=True,user_id='535feccd53b44a24a33905f2f2a4795d',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 16:31:36 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:24:45Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-SnapshotsActionsTest-Volume-1453311494',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=35a67ca3-0f17-4a6c-a359-9173ce5f1aca,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='5b416855df3047d79fd4d27b2b1695b8',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:29:18Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:29:18Z,use_quota=True,user_id='535feccd53b44a24a33905f2f2a4795d',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(35a67ca3-0f17-4a6c-a359-9173ce5f1aca),volume_id=35a67ca3-0f17-4a6c-a359-9173ce5f1aca,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': '35a67ca3-0f17-4a6c-a359-9173ce5f1aca', 'volume_name': 'volume-35a67ca3-0f17-4a6c-a359-9173ce5f1aca', 'volume_size': 1}} Apr 17 16:31:36 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': '35a67ca3-0f17-4a6c-a359-9173ce5f1aca', 'volume': {'id': '35a67ca3-0f17-4a6c-a359-9173ce5f1aca', '_name_id': None, 'ec2_id': None, 'user_id': '535feccd53b44a24a33905f2f2a4795d', 'project_id': '5b416855df3047d79fd4d27b2b1695b8', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-SnapshotsActionsTest-Volume-1453311494', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'admin_metadata': {}, 'glance_metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T16:24:45.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-35a67ca3-0f17-4a6c-a359-9173ce5f1aca', 'name_id': '35a67ca3-0f17-4a6c-a359-9173ce5f1aca', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-SnapshotsActionsTest-Volume-1453311494', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '5b416855df3047d79fd4d27b2b1695b8', 'qos_specs': None, 'reservations': ['7d4b6592-57f9-4b52-b860-f07e41add803', 'f96f8a6a-f71d-4298-a8e4-362b7bcfdfe8', '39019dff-14a7-4a6d-8e21-9297b7666adc', '3459ea23-ef9a-4aec-b56b-db2ebc6e2a90'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': '535feccd53b44a24a33905f2f2a4795d', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-SnapshotsActionsTest-Volume-1453311494', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '5b416855df3047d79fd4d27b2b1695b8', 'qos_specs': None, 'reservations': ['7d4b6592-57f9-4b52-b860-f07e41add803', 'f96f8a6a-f71d-4298-a8e4-362b7bcfdfe8', '39019dff-14a7-4a6d-8e21-9297b7666adc', '3459ea23-ef9a-4aec-b56b-db2ebc6e2a90'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': '535feccd53b44a24a33905f2f2a4795d', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': '535feccd53b44a24a33905f2f2a4795d', 'metadata': {}, 'qos_specs': None, 'retry': {'num_attempts': 3, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']}}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(35a67ca3-0f17-4a6c-a359-9173ce5f1aca),volume_id=35a67ca3-0f17-4a6c-a359-9173ce5f1aca,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:24:45Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-SnapshotsActionsTest-Volume-1453311494',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=35a67ca3-0f17-4a6c-a359-9173ce5f1aca,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='5b416855df3047d79fd4d27b2b1695b8',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:29:18Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:29:18Z,use_quota=True,user_id='535feccd53b44a24a33905f2f2a4795d',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 16:31:36 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:24:45Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-SnapshotsActionsTest-Volume-1453311494',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=35a67ca3-0f17-4a6c-a359-9173ce5f1aca,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='5b416855df3047d79fd4d27b2b1695b8',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:29:18Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:29:18Z,use_quota=True,user_id='535feccd53b44a24a33905f2f2a4795d',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:24:45Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-SnapshotsActionsTest-Volume-1453311494',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=35a67ca3-0f17-4a6c-a359-9173ce5f1aca,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='5b416855df3047d79fd4d27b2b1695b8',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:29:18Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:29:18Z,use_quota=True,user_id='535feccd53b44a24a33905f2f2a4795d',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 16:31:36 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 16:31:36 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 16:31:36 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:31:36 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 16:31:36 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:31:36 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 16:31:36 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:31:36 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 16:31:36 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:31:36 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 16:31:36 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:31:36 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 16:31:36 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:31:36 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 16:31:36 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:31:36 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 16:31:36 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:31:36 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:31:36 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 16:31:36 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 16:31:36 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (a6908629-af82-4147-999b-21be37e1797c) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:31:36 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (a6908629-af82-4147-999b-21be37e1797c) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:31:36 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (c4e53235-e49d-496d-8b58-fff21be19391) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:31:36 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (c4e53235-e49d-496d-8b58-fff21be19391) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:31:36 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (1d906177-e8c9-4ce4-bc90-fb1b8274afbc) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:31:36 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (1d906177-e8c9-4ce4-bc90-fb1b8274afbc) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:31:36 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (3ba1e240-9772-4d59-9646-a5e3f6b9937f) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:31:36 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Updating volume 35a67ca3-0f17-4a6c-a359-9173ce5f1aca with {'scheduled_at': datetime.datetime(2023, 4, 17, 16, 31, 36, 753837), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 16:31:36 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Volume 35a67ca3-0f17-4a6c-a359-9173ce5f1aca: re-scheduling SchedulerAPI.create_volume attempt 3 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 16:31:36 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Volume 35a67ca3-0f17-4a6c-a359-9173ce5f1aca: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 16:31:36 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] deleating volume v_35a67ca3-0f17-4a6c-a359-9173ce5f1aca {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 16:31:36 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] get snapshots for volume v_35a67ca3-0f17-4a6c-a359-9173ce5f1aca {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 16:31:36 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_35a67ca3-0f17-4a6c-a359-9173ce5f1aca/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:31:36 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_35a67ca3-0f17-4a6c-a359-9173ce5f1aca/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:31:39 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.280(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:31:39 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:31:39 user cinder-volume[84696]: DEBUG cinder.utils [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.962(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:31:39 user cinder-volume[84696]: DEBUG cinder.utils [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_35a67ca3-0f17-4a6c-a359-9173ce5f1aca/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:31:39 user cinder-volume[84696]: DEBUG cinder.utils [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.035(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:31:39 user cinder-volume[84696]: DEBUG cinder.utils [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_53029faa-77b2-467d-8a64-eb5ef129e41c/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:31:42 user cinder-volume[84696]: DEBUG cinder.utils [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 74.801(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:31:42 user cinder-volume[84696]: DEBUG cinder.utils [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:31:42 user cinder-volume[84696]: DEBUG cinder.utils [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 74.979(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:31:42 user cinder-volume[84696]: DEBUG cinder.utils [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_f6c0e341-6f00-4e32-bd07-31f954c1ad8a/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:31:42 user cinder-volume[84696]: DEBUG cinder.utils [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.154(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:31:42 user cinder-volume[84696]: DEBUG cinder.utils [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:31:42 user cinder-volume[84696]: DEBUG cinder.utils [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.039(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:31:42 user cinder-volume[84696]: DEBUG cinder.utils [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_35a67ca3-0f17-4a6c-a359-9173ce5f1aca/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:31:44 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=20,cinder:UPDATE=7,cinder:INSERT=1 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:31:45 user cinder-volume[84696]: DEBUG cinder.utils [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 19.743(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:31:45 user cinder-volume[84696]: DEBUG cinder.utils [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:31:45 user cinder-volume[84696]: DEBUG cinder.utils [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.178(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:31:45 user cinder-volume[84696]: DEBUG cinder.utils [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_53029faa-77b2-467d-8a64-eb5ef129e41c/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:31:48 user cinder-volume[84696]: DEBUG cinder.utils [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.177(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:31:48 user cinder-volume[84696]: DEBUG cinder.utils [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_35a67ca3-0f17-4a6c-a359-9173ce5f1aca/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:31:48 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.497(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:31:48 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:31:54 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=13,cinder:UPDATE=3,cinder:INSERT=3 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:31:55 user cinder-volume[84696]: DEBUG cinder.utils [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.552(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:31:55 user cinder-volume[84696]: DEBUG cinder.utils [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_53029faa-77b2-467d-8a64-eb5ef129e41c/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:31:55 user cinder-volume[84696]: DEBUG cinder.utils [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 37.512(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:31:55 user cinder-volume[84696]: DEBUG cinder.utils [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:31:58 user cinder-volume[84696]: DEBUG cinder.utils [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.553(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:31:58 user cinder-volume[84696]: DEBUG cinder.utils [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_35a67ca3-0f17-4a6c-a359-9173ce5f1aca/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:32:01 user cinder-volume[84696]: DEBUG cinder.utils [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.745(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:32:01 user cinder-volume[84696]: DEBUG cinder.utils [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:32:04 user cinder-volume[84696]: DEBUG cinder.utils [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 38.335(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:32:04 user cinder-volume[84696]: DEBUG cinder.utils [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:32:07 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 40.089(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:32:07 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:32:14 user cinder-volume[84696]: DEBUG cinder.utils [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 40.560(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:32:14 user cinder-volume[84696]: DEBUG cinder.utils [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_53029faa-77b2-467d-8a64-eb5ef129e41c/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:32:14 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=1,cinder:UPDATE=1 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:32:17 user cinder-volume[84696]: DEBUG cinder.utils [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 40.561(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:32:17 user cinder-volume[84696]: DEBUG cinder.utils [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_35a67ca3-0f17-4a6c-a359-9173ce5f1aca/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:32:24 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=8,cinder:UPDATE=2 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:32:27 user cinder-volume[84696]: DEBUG cinder.utils [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 69.543(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:32:27 user cinder-volume[84696]: DEBUG cinder.utils [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:32:34 user cinder-volume[84696]: DEBUG cinder.utils [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 72.642(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:32:34 user cinder-volume[84696]: DEBUG cinder.utils [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:32:34 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=1,cinder:UPDATE=1 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:32:37 user cinder-volume[84696]: DEBUG cinder.utils [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 71.231(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:32:37 user cinder-volume[84696]: DEBUG cinder.utils [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:32:41 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 73.816(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:32:41 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:32:41 user cinder-volume[84696]: DEBUG cinder.utils [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 139.485(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:32:41 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_8b7f1136-3d20-4cf8-a521-6db1d3a0c8a8/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:32:41 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (1e87cdd6-a8de-472d-9153-6a29d1e99877) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 16:32:41 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (e8eb09f7-e8c6-41cc-ad12-49dea48c296e) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:32:41 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (e8eb09f7-e8c6-41cc-ad12-49dea48c296e) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:32:41 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Flow 'volume_create_manager' (02c1cb2a-e9f0-4d27-b9f3-3c239ebbae7f) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 16:32:41 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 16:32:41 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 16:32:41 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 16:32:41 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 16:32:41 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 16:32:41 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 16:32:41 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 16:32:41 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 16:32:41 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 16:32:41 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 16:32:41 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 16:32:41 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 16:32:41 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 16:32:41 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 16:32:41 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 16:32:41 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 16:32:41 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 16:32:41 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 16:32:41 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 16:32:41 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 16:32:41 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 16:32:41 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 16:32:41 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 16:32:41 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 16:32:41 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:32:41 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 16:32:41 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:32:41 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 16:32:41 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:32:41 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 16:32:41 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:32:41 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 16:32:41 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:32:41 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 16:32:41 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:32:41 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 16:32:41 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:32:41 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 16:32:41 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:32:41 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:32:41 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 16:32:41 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 16:32:41 user cinder-volume[84696]: DEBUG cinder.utils [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 139.216(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:32:41 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:32:41 user cinder-volume[84696]: INFO cinder.message.api [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Creating message record for request_id = req-34802d26-273b-4fa3-a0d4-af12da0d916e Apr 17 16:32:41 user cinder-volume[84696]: DEBUG cinder.utils [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 139.259(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:32:41 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:32:41 user cinder-volume[84696]: INFO cinder.message.api [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Creating message record for request_id = req-ccbb52f2-042b-41e0-9837-623e8a9a2738 Apr 17 16:32:41 user cinder-volume[84696]: DEBUG cinder.utils [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 139.499(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:32:41 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_2159f1f1-2564-4d84-8110-c1f7d843edb1/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:32:41 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (95c2cf7a-1593-47ae-a6ac-d14d8e803005) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 16:32:41 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (7e67e899-dc4f-4a7f-8e99-b3ef0885e2df) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:32:41 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (7e67e899-dc4f-4a7f-8e99-b3ef0885e2df) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:32:41 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Flow 'volume_create_manager' (bf763189-a29f-46d2-8264-0dc7ac20dcae) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 16:32:41 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 16:32:41 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 16:32:41 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 16:32:41 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 16:32:41 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 16:32:41 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 16:32:41 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 16:32:41 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 16:32:41 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 16:32:41 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 16:32:41 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 16:32:41 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 16:32:41 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 16:32:41 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 16:32:41 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 16:32:41 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 16:32:41 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 16:32:41 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 16:32:41 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 16:32:41 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 16:32:41 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 16:32:41 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 16:32:41 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 16:32:41 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 16:32:41 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:32:41 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 16:32:41 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:32:41 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 16:32:41 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:32:41 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 16:32:41 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:32:41 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 16:32:41 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:32:41 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 16:32:41 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:32:41 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 16:32:41 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:32:41 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 16:32:41 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:32:41 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:32:41 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 16:32:41 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 16:32:41 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 16:32:41 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 16:32:41 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (d2da08f2-a5d7-48f5-a044-18660301597a) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 16:32:41 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 16:32:41 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:27:59Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesDeleteCascade-Volume-2055115456',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=8b7f1136-3d20-4cf8-a521-6db1d3a0c8a8,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='abbfd52254f546b0bcfe289884d77f49',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:30:22Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:30:22Z,use_quota=True,user_id='6309fcbea41c471184457d276d32db73',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 16:32:41 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:27:59Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesDeleteCascade-Volume-2055115456',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=8b7f1136-3d20-4cf8-a521-6db1d3a0c8a8,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='abbfd52254f546b0bcfe289884d77f49',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:30:22Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:30:22Z,use_quota=True,user_id='6309fcbea41c471184457d276d32db73',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(8b7f1136-3d20-4cf8-a521-6db1d3a0c8a8),volume_id=8b7f1136-3d20-4cf8-a521-6db1d3a0c8a8,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': '8b7f1136-3d20-4cf8-a521-6db1d3a0c8a8', 'volume_name': 'volume-8b7f1136-3d20-4cf8-a521-6db1d3a0c8a8', 'volume_size': 1}} Apr 17 16:32:41 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': '8b7f1136-3d20-4cf8-a521-6db1d3a0c8a8', 'volume': {'id': '8b7f1136-3d20-4cf8-a521-6db1d3a0c8a8', '_name_id': None, 'ec2_id': None, 'user_id': '6309fcbea41c471184457d276d32db73', 'project_id': 'abbfd52254f546b0bcfe289884d77f49', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-VolumesDeleteCascade-Volume-2055115456', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'admin_metadata': {}, 'glance_metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T16:27:58.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-8b7f1136-3d20-4cf8-a521-6db1d3a0c8a8', 'name_id': '8b7f1136-3d20-4cf8-a521-6db1d3a0c8a8', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumesDeleteCascade-Volume-2055115456', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': 'abbfd52254f546b0bcfe289884d77f49', 'qos_specs': None, 'reservations': ['374a7e05-986c-4abf-a2c9-4761a21cff44', 'b5097575-0972-436f-8ef4-38f9b292ab7b', '56c0c623-13ab-497a-bc50-d552acc88144', 'c6eb5369-b08c-4737-8356-45cc9d705e20'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': '6309fcbea41c471184457d276d32db73', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumesDeleteCascade-Volume-2055115456', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': 'abbfd52254f546b0bcfe289884d77f49', 'qos_specs': None, 'reservations': ['374a7e05-986c-4abf-a2c9-4761a21cff44', 'b5097575-0972-436f-8ef4-38f9b292ab7b', '56c0c623-13ab-497a-bc50-d552acc88144', 'c6eb5369-b08c-4737-8356-45cc9d705e20'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': '6309fcbea41c471184457d276d32db73', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': '6309fcbea41c471184457d276d32db73', 'metadata': {}, 'qos_specs': None, 'retry': {'num_attempts': 2, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']}}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(8b7f1136-3d20-4cf8-a521-6db1d3a0c8a8),volume_id=8b7f1136-3d20-4cf8-a521-6db1d3a0c8a8,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:27:59Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesDeleteCascade-Volume-2055115456',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=8b7f1136-3d20-4cf8-a521-6db1d3a0c8a8,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='abbfd52254f546b0bcfe289884d77f49',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:30:22Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:30:22Z,use_quota=True,user_id='6309fcbea41c471184457d276d32db73',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 16:32:41 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:27:59Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesDeleteCascade-Volume-2055115456',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=8b7f1136-3d20-4cf8-a521-6db1d3a0c8a8,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='abbfd52254f546b0bcfe289884d77f49',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:30:22Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:30:22Z,use_quota=True,user_id='6309fcbea41c471184457d276d32db73',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:27:59Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesDeleteCascade-Volume-2055115456',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=8b7f1136-3d20-4cf8-a521-6db1d3a0c8a8,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='abbfd52254f546b0bcfe289884d77f49',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:30:22Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:30:22Z,use_quota=True,user_id='6309fcbea41c471184457d276d32db73',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 16:32:41 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 16:32:41 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 16:32:41 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:32:41 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 16:32:41 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:32:41 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 16:32:41 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:32:41 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 16:32:41 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:32:41 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 16:32:41 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:32:41 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 16:32:41 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:32:41 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 16:32:41 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:32:41 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 16:32:41 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:32:41 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:32:41 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 16:32:41 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 16:32:41 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (d2da08f2-a5d7-48f5-a044-18660301597a) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:32:41 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (d2da08f2-a5d7-48f5-a044-18660301597a) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:32:41 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (5d1ae647-1830-451d-b121-24ea22118891) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:32:41 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (5d1ae647-1830-451d-b121-24ea22118891) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:32:41 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (0ab75f76-a544-46e8-a21a-9cdba7271dc7) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:32:41 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (0ab75f76-a544-46e8-a21a-9cdba7271dc7) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:32:41 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (ea985edf-0b1d-4ffc-ade1-837a9b4be8b3) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:32:41 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Updating volume 8b7f1136-3d20-4cf8-a521-6db1d3a0c8a8 with {'scheduled_at': datetime.datetime(2023, 4, 17, 16, 32, 41, 375564), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 16:32:41 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (902eba17-4574-415c-ae23-cbb7f3931065) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 16:32:41 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 16:32:41 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:28:03Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesSnapshotListTestJSON-Volume-552939668',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=2159f1f1-2564-4d84-8110-c1f7d843edb1,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='c4b1a8a0074e4798bd385ee96e8ac8b7',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:30:22Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:30:22Z,use_quota=True,user_id='476228814d42467c880dce2b1c5c5113',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 16:32:41 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:28:03Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesSnapshotListTestJSON-Volume-552939668',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=2159f1f1-2564-4d84-8110-c1f7d843edb1,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='c4b1a8a0074e4798bd385ee96e8ac8b7',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:30:22Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:30:22Z,use_quota=True,user_id='476228814d42467c880dce2b1c5c5113',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(2159f1f1-2564-4d84-8110-c1f7d843edb1),volume_id=2159f1f1-2564-4d84-8110-c1f7d843edb1,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': '2159f1f1-2564-4d84-8110-c1f7d843edb1', 'volume_name': 'volume-2159f1f1-2564-4d84-8110-c1f7d843edb1', 'volume_size': 1}} Apr 17 16:32:41 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': '2159f1f1-2564-4d84-8110-c1f7d843edb1', 'volume': {'id': '2159f1f1-2564-4d84-8110-c1f7d843edb1', '_name_id': None, 'ec2_id': None, 'user_id': '476228814d42467c880dce2b1c5c5113', 'project_id': 'c4b1a8a0074e4798bd385ee96e8ac8b7', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-VolumesSnapshotListTestJSON-Volume-552939668', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'admin_metadata': {}, 'glance_metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T16:28:02.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-2159f1f1-2564-4d84-8110-c1f7d843edb1', 'name_id': '2159f1f1-2564-4d84-8110-c1f7d843edb1', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumesSnapshotListTestJSON-Volume-552939668', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': 'c4b1a8a0074e4798bd385ee96e8ac8b7', 'qos_specs': None, 'reservations': ['0c108957-403f-4dc6-adfa-0a0de976576c', 'e72136a8-7407-4f95-a72e-b484cca53b29', 'cc5aa602-fa98-4b80-9896-3829bb01e999', 'ec5862b0-0012-4eba-b940-a35906fa2d44'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': '476228814d42467c880dce2b1c5c5113', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumesSnapshotListTestJSON-Volume-552939668', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': 'c4b1a8a0074e4798bd385ee96e8ac8b7', 'qos_specs': None, 'reservations': ['0c108957-403f-4dc6-adfa-0a0de976576c', 'e72136a8-7407-4f95-a72e-b484cca53b29', 'cc5aa602-fa98-4b80-9896-3829bb01e999', 'ec5862b0-0012-4eba-b940-a35906fa2d44'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': '476228814d42467c880dce2b1c5c5113', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': '476228814d42467c880dce2b1c5c5113', 'metadata': {}, 'qos_specs': None, 'retry': {'num_attempts': 2, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']}}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(2159f1f1-2564-4d84-8110-c1f7d843edb1),volume_id=2159f1f1-2564-4d84-8110-c1f7d843edb1,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:28:03Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesSnapshotListTestJSON-Volume-552939668',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=2159f1f1-2564-4d84-8110-c1f7d843edb1,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='c4b1a8a0074e4798bd385ee96e8ac8b7',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:30:22Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:30:22Z,use_quota=True,user_id='476228814d42467c880dce2b1c5c5113',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 16:32:41 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:28:03Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesSnapshotListTestJSON-Volume-552939668',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=2159f1f1-2564-4d84-8110-c1f7d843edb1,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='c4b1a8a0074e4798bd385ee96e8ac8b7',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:30:22Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:30:22Z,use_quota=True,user_id='476228814d42467c880dce2b1c5c5113',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:28:03Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesSnapshotListTestJSON-Volume-552939668',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=2159f1f1-2564-4d84-8110-c1f7d843edb1,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='c4b1a8a0074e4798bd385ee96e8ac8b7',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:30:22Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:30:22Z,use_quota=True,user_id='476228814d42467c880dce2b1c5c5113',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 16:32:41 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 16:32:41 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 16:32:41 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:32:41 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 16:32:41 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:32:41 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 16:32:41 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:32:41 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 16:32:41 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:32:41 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 16:32:41 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:32:41 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 16:32:41 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:32:41 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 16:32:41 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:32:41 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 16:32:41 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:32:41 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:32:41 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 16:32:41 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 16:32:41 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (902eba17-4574-415c-ae23-cbb7f3931065) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:32:41 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (902eba17-4574-415c-ae23-cbb7f3931065) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:32:41 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (d23e7a46-645a-4d37-84fd-fd7a03a3c54d) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:32:41 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (d23e7a46-645a-4d37-84fd-fd7a03a3c54d) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:32:41 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (3f4f1285-5b3c-438b-a500-33e76120465f) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:32:41 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (3f4f1285-5b3c-438b-a500-33e76120465f) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:32:41 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (cdbb57d0-c62f-4c33-aeb2-e64d3e362f8b) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:32:41 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Updating volume 2159f1f1-2564-4d84-8110-c1f7d843edb1 with {'scheduled_at': datetime.datetime(2023, 4, 17, 16, 32, 41, 423222), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 16:32:41 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Volume 8b7f1136-3d20-4cf8-a521-6db1d3a0c8a8: re-scheduling SchedulerAPI.create_volume attempt 2 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 16:32:41 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Volume 2159f1f1-2564-4d84-8110-c1f7d843edb1: re-scheduling SchedulerAPI.create_volume attempt 2 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 16:32:41 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Volume 8b7f1136-3d20-4cf8-a521-6db1d3a0c8a8: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 16:32:41 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] deleating volume v_8b7f1136-3d20-4cf8-a521-6db1d3a0c8a8 {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 16:32:41 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] get snapshots for volume v_8b7f1136-3d20-4cf8-a521-6db1d3a0c8a8 {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 16:32:41 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_8b7f1136-3d20-4cf8-a521-6db1d3a0c8a8/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:32:41 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_8b7f1136-3d20-4cf8-a521-6db1d3a0c8a8/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:32:41 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Volume 2159f1f1-2564-4d84-8110-c1f7d843edb1: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 16:32:41 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] deleating volume v_2159f1f1-2564-4d84-8110-c1f7d843edb1 {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 16:32:41 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] get snapshots for volume v_2159f1f1-2564-4d84-8110-c1f7d843edb1 {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 16:32:41 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_2159f1f1-2564-4d84-8110-c1f7d843edb1/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:32:41 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_2159f1f1-2564-4d84-8110-c1f7d843edb1/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:32:41 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Volume reschedule parameters: True retry: {'num_attempts': 3, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 16:32:41 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Flow 'volume_create_manager' (66ae15f6-be27-41c1-a5ca-8479c8adc816) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 16:32:41 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (fdd97590-2600-48c4-8049-02c324ef348b) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:32:41 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (fdd97590-2600-48c4-8049-02c324ef348b) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:27:59Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesDeleteCascade-Volume-2055115456',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=8b7f1136-3d20-4cf8-a521-6db1d3a0c8a8,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='abbfd52254f546b0bcfe289884d77f49',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:32:42Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:32:42Z,use_quota=True,user_id='6309fcbea41c471184457d276d32db73',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:32:41 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (39ce3927-4fa9-4393-b9eb-4475175819de) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:32:41 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (39ce3927-4fa9-4393-b9eb-4475175819de) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:32:41 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (31b2a566-87bb-48fe-965d-d7584cdfa324) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:32:41 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (31b2a566-87bb-48fe-965d-d7584cdfa324) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': '8b7f1136-3d20-4cf8-a521-6db1d3a0c8a8', 'volume_name': 'volume-8b7f1136-3d20-4cf8-a521-6db1d3a0c8a8', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:32:41 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (0c4a9744-69dd-471e-9e26-85aafa0d3deb) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:32:41 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (0c4a9744-69dd-471e-9e26-85aafa0d3deb) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:32:41 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (5e72c775-bd35-4c0a-8bb6-234f65b4515a) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:32:41 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Volume 8b7f1136-3d20-4cf8-a521-6db1d3a0c8a8: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-8b7f1136-3d20-4cf8-a521-6db1d3a0c8a8', 'volume_size': 1} Apr 17 16:32:41 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] creating volume v_8b7f1136-3d20-4cf8-a521-6db1d3a0c8a8. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 16:32:41 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] create volume {'name': 'v_8b7f1136-3d20-4cf8-a521-6db1d3a0c8a8', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 16:32:41 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:32:41 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:32:41 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Volume reschedule parameters: True retry: {'num_attempts': 3, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 16:32:41 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Flow 'volume_create_manager' (7111dffa-ede2-4b36-8afd-1d12c61e9309) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 16:32:41 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (433a3676-9d92-4297-a633-daed55cb3767) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:32:41 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (433a3676-9d92-4297-a633-daed55cb3767) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:28:03Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesSnapshotListTestJSON-Volume-552939668',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=2159f1f1-2564-4d84-8110-c1f7d843edb1,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='c4b1a8a0074e4798bd385ee96e8ac8b7',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:32:42Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:32:42Z,use_quota=True,user_id='476228814d42467c880dce2b1c5c5113',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:32:41 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (66f629a8-6266-48fd-8ac5-26a9a6bec901) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:32:41 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (66f629a8-6266-48fd-8ac5-26a9a6bec901) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:32:41 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (4140f006-344c-4130-a620-de9a99f38596) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:32:41 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (4140f006-344c-4130-a620-de9a99f38596) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': '2159f1f1-2564-4d84-8110-c1f7d843edb1', 'volume_name': 'volume-2159f1f1-2564-4d84-8110-c1f7d843edb1', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:32:41 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (a114f1d5-edd3-4a2c-acac-5cd6c8fc610a) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:32:41 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (a114f1d5-edd3-4a2c-acac-5cd6c8fc610a) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:32:41 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (721e1133-e9db-4d81-8eb1-755d8155c29f) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:32:41 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Volume 2159f1f1-2564-4d84-8110-c1f7d843edb1: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-2159f1f1-2564-4d84-8110-c1f7d843edb1', 'volume_size': 1} Apr 17 16:32:41 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] creating volume v_2159f1f1-2564-4d84-8110-c1f7d843edb1. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 16:32:41 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] create volume {'name': 'v_2159f1f1-2564-4d84-8110-c1f7d843edb1', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 16:32:41 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:32:41 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:32:44 user cinder-volume[84696]: DEBUG cinder.utils [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.683(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:32:44 user cinder-volume[84696]: DEBUG cinder.utils [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:32:44 user cinder-volume[84696]: DEBUG cinder.utils [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.896(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:32:44 user cinder-volume[84696]: DEBUG cinder.utils [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_8b7f1136-3d20-4cf8-a521-6db1d3a0c8a8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:32:44 user cinder-volume[84696]: DEBUG cinder.utils [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.575(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:32:44 user cinder-volume[84696]: DEBUG cinder.utils [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:32:44 user cinder-volume[84696]: DEBUG cinder.utils [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.891(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:32:44 user cinder-volume[84696]: DEBUG cinder.utils [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_2159f1f1-2564-4d84-8110-c1f7d843edb1/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:32:47 user cinder-volume[84696]: DEBUG cinder.utils [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 139.408(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:32:47 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:32:47 user cinder-volume[84696]: INFO cinder.message.api [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Creating message record for request_id = req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d Apr 17 16:32:47 user cinder-volume[84696]: DEBUG cinder.utils [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 139.590(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:32:47 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_f6c0e341-6f00-4e32-bd07-31f954c1ad8a/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:32:47 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (17697f70-2399-4d33-91c6-50d7e87f28f6) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 16:32:47 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (70340d43-3477-43ac-b1f9-b7fbc29f47b0) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:32:47 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (70340d43-3477-43ac-b1f9-b7fbc29f47b0) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:32:47 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Flow 'volume_create_manager' (4bec709d-dabc-402a-b320-5c8918e025d7) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 16:32:47 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 16:32:47 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 16:32:47 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 16:32:47 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 16:32:47 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 16:32:47 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 16:32:47 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 16:32:47 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 16:32:47 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 16:32:47 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 16:32:47 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 16:32:47 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 16:32:47 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 16:32:47 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 16:32:47 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 16:32:47 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 16:32:47 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 16:32:47 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 16:32:47 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 16:32:47 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 16:32:47 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 16:32:47 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 16:32:47 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 16:32:47 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 16:32:47 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:32:47 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 16:32:47 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:32:47 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 16:32:47 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:32:47 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 16:32:47 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:32:47 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 16:32:47 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:32:47 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 16:32:47 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:32:47 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 16:32:47 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:32:47 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 16:32:47 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:32:47 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:32:47 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 16:32:47 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 16:32:47 user cinder-volume[84696]: DEBUG cinder.utils [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.668(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:32:47 user cinder-volume[84696]: DEBUG cinder.utils [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:32:47 user cinder-volume[84696]: DEBUG cinder.utils [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.992(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:32:47 user cinder-volume[84696]: DEBUG cinder.utils [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_8b7f1136-3d20-4cf8-a521-6db1d3a0c8a8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:32:47 user cinder-volume[84696]: DEBUG cinder.utils [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.781(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:32:47 user cinder-volume[84696]: DEBUG cinder.utils [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:32:47 user cinder-volume[84696]: DEBUG cinder.utils [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 73.739(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:32:47 user cinder-volume[84696]: DEBUG cinder.utils [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_53029faa-77b2-467d-8a64-eb5ef129e41c/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:32:47 user cinder-volume[84696]: DEBUG cinder.utils [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.986(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:32:47 user cinder-volume[84696]: DEBUG cinder.utils [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_2159f1f1-2564-4d84-8110-c1f7d843edb1/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:32:47 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 16:32:47 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (c9402295-b2c7-45b7-a764-5f181be36aae) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 16:32:47 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 16:32:47 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:28:10Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumeQuotasAdminTestJSON-Volume-332841798',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=f6c0e341-6f00-4e32-bd07-31f954c1ad8a,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='6e4c2ec9177349a08ce53e07b24ad79c',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:30:28Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:30:28Z,use_quota=True,user_id='ab5017da30ac4d47ba665ff135deb309',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 16:32:47 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:28:10Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumeQuotasAdminTestJSON-Volume-332841798',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=f6c0e341-6f00-4e32-bd07-31f954c1ad8a,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='6e4c2ec9177349a08ce53e07b24ad79c',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:30:28Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:30:28Z,use_quota=True,user_id='ab5017da30ac4d47ba665ff135deb309',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(f6c0e341-6f00-4e32-bd07-31f954c1ad8a),volume_id=f6c0e341-6f00-4e32-bd07-31f954c1ad8a,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': 'f6c0e341-6f00-4e32-bd07-31f954c1ad8a', 'volume_name': 'volume-f6c0e341-6f00-4e32-bd07-31f954c1ad8a', 'volume_size': 1}} Apr 17 16:32:47 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': 'f6c0e341-6f00-4e32-bd07-31f954c1ad8a', 'volume': {'id': 'f6c0e341-6f00-4e32-bd07-31f954c1ad8a', '_name_id': None, 'ec2_id': None, 'user_id': 'ab5017da30ac4d47ba665ff135deb309', 'project_id': '6e4c2ec9177349a08ce53e07b24ad79c', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-VolumeQuotasAdminTestJSON-Volume-332841798', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'admin_metadata': {}, 'glance_metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T16:28:09.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-f6c0e341-6f00-4e32-bd07-31f954c1ad8a', 'name_id': 'f6c0e341-6f00-4e32-bd07-31f954c1ad8a', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumeQuotasAdminTestJSON-Volume-332841798', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '6e4c2ec9177349a08ce53e07b24ad79c', 'qos_specs': None, 'reservations': ['f21078bb-20eb-4029-93a1-5b4cb253f25d', 'bbd80d77-344a-4970-b1bb-2892c08c1c72', 'a38f5177-62fc-4eb7-b072-f15a54689ed0', 'b5e5e1fe-ca2e-48fa-b5e1-b6f6d98fe349'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'ab5017da30ac4d47ba665ff135deb309', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumeQuotasAdminTestJSON-Volume-332841798', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '6e4c2ec9177349a08ce53e07b24ad79c', 'qos_specs': None, 'reservations': ['f21078bb-20eb-4029-93a1-5b4cb253f25d', 'bbd80d77-344a-4970-b1bb-2892c08c1c72', 'a38f5177-62fc-4eb7-b072-f15a54689ed0', 'b5e5e1fe-ca2e-48fa-b5e1-b6f6d98fe349'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'ab5017da30ac4d47ba665ff135deb309', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': 'ab5017da30ac4d47ba665ff135deb309', 'metadata': {}, 'qos_specs': None, 'retry': {'num_attempts': 2, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']}}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(f6c0e341-6f00-4e32-bd07-31f954c1ad8a),volume_id=f6c0e341-6f00-4e32-bd07-31f954c1ad8a,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:28:10Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumeQuotasAdminTestJSON-Volume-332841798',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=f6c0e341-6f00-4e32-bd07-31f954c1ad8a,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='6e4c2ec9177349a08ce53e07b24ad79c',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:30:28Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:30:28Z,use_quota=True,user_id='ab5017da30ac4d47ba665ff135deb309',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 16:32:47 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:28:10Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumeQuotasAdminTestJSON-Volume-332841798',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=f6c0e341-6f00-4e32-bd07-31f954c1ad8a,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='6e4c2ec9177349a08ce53e07b24ad79c',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:30:28Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:30:28Z,use_quota=True,user_id='ab5017da30ac4d47ba665ff135deb309',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:28:10Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumeQuotasAdminTestJSON-Volume-332841798',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=f6c0e341-6f00-4e32-bd07-31f954c1ad8a,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='6e4c2ec9177349a08ce53e07b24ad79c',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:30:28Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:30:28Z,use_quota=True,user_id='ab5017da30ac4d47ba665ff135deb309',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 16:32:47 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 16:32:47 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 16:32:47 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:32:47 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 16:32:47 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:32:47 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 16:32:47 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:32:47 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 16:32:47 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:32:47 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 16:32:47 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:32:47 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 16:32:47 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:32:47 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 16:32:47 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:32:47 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 16:32:47 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:32:47 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:32:47 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 16:32:47 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 16:32:47 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (c9402295-b2c7-45b7-a764-5f181be36aae) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:32:47 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (c9402295-b2c7-45b7-a764-5f181be36aae) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:32:47 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (e4d0c2e2-7545-40cb-96b1-5a69c20aef10) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:32:47 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (e4d0c2e2-7545-40cb-96b1-5a69c20aef10) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:32:47 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (f390019f-5182-40de-a488-958e73806d06) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:32:47 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (f390019f-5182-40de-a488-958e73806d06) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:32:47 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (c5686eb2-064c-40f9-8c44-e9ea3b76a86b) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:32:47 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Updating volume f6c0e341-6f00-4e32-bd07-31f954c1ad8a with {'scheduled_at': datetime.datetime(2023, 4, 17, 16, 32, 47, 502671), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 16:32:47 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Volume f6c0e341-6f00-4e32-bd07-31f954c1ad8a: re-scheduling SchedulerAPI.create_volume attempt 2 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 16:32:47 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Volume f6c0e341-6f00-4e32-bd07-31f954c1ad8a: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 16:32:47 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] deleating volume v_f6c0e341-6f00-4e32-bd07-31f954c1ad8a {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 16:32:47 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] get snapshots for volume v_f6c0e341-6f00-4e32-bd07-31f954c1ad8a {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 16:32:47 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_f6c0e341-6f00-4e32-bd07-31f954c1ad8a/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:32:47 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_f6c0e341-6f00-4e32-bd07-31f954c1ad8a/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:32:47 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Volume reschedule parameters: True retry: {'num_attempts': 3, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 16:32:47 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Flow 'volume_create_manager' (d2d134d5-e4bd-498d-aff4-08ba0e172157) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 16:32:47 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (5b747472-600c-466e-9baf-9cf308b7082c) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:32:47 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (5b747472-600c-466e-9baf-9cf308b7082c) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:28:10Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumeQuotasAdminTestJSON-Volume-332841798',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=f6c0e341-6f00-4e32-bd07-31f954c1ad8a,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='6e4c2ec9177349a08ce53e07b24ad79c',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:32:48Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:32:48Z,use_quota=True,user_id='ab5017da30ac4d47ba665ff135deb309',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:32:47 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (854e74aa-081b-4a34-9787-2c0edccf9773) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:32:47 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (854e74aa-081b-4a34-9787-2c0edccf9773) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:32:47 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (f2fb9d7b-2629-40e2-863b-82c9f30bb379) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:32:47 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (f2fb9d7b-2629-40e2-863b-82c9f30bb379) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': 'f6c0e341-6f00-4e32-bd07-31f954c1ad8a', 'volume_name': 'volume-f6c0e341-6f00-4e32-bd07-31f954c1ad8a', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:32:47 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (228d2dc0-bc93-44c5-9838-57fce29d2f99) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:32:47 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (228d2dc0-bc93-44c5-9838-57fce29d2f99) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:32:47 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (12b6b127-c2dd-448d-9d71-e4010e676eb2) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:32:47 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Volume f6c0e341-6f00-4e32-bd07-31f954c1ad8a: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-f6c0e341-6f00-4e32-bd07-31f954c1ad8a', 'volume_size': 1} Apr 17 16:32:47 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] creating volume v_f6c0e341-6f00-4e32-bd07-31f954c1ad8a. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 16:32:47 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] create volume {'name': 'v_f6c0e341-6f00-4e32-bd07-31f954c1ad8a', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 16:32:47 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:32:47 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:32:50 user cinder-volume[84696]: DEBUG cinder.utils [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.961(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:32:50 user cinder-volume[84696]: DEBUG cinder.utils [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_f6c0e341-6f00-4e32-bd07-31f954c1ad8a/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:32:50 user cinder-volume[84696]: DEBUG cinder.utils [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.737(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:32:50 user cinder-volume[84696]: DEBUG cinder.utils [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:32:50 user cinder-volume[84696]: DEBUG cinder.utils [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 73.716(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:32:50 user cinder-volume[84696]: DEBUG cinder.utils [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_35a67ca3-0f17-4a6c-a359-9173ce5f1aca/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:32:53 user cinder-volume[84696]: DEBUG cinder.utils [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 11.788(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:32:53 user cinder-volume[84696]: DEBUG cinder.utils [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:32:53 user cinder-volume[84696]: DEBUG cinder.utils [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.103(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:32:53 user cinder-volume[84696]: DEBUG cinder.utils [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_2159f1f1-2564-4d84-8110-c1f7d843edb1/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:32:53 user cinder-volume[84696]: DEBUG cinder.utils [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 11.901(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:32:53 user cinder-volume[84696]: DEBUG cinder.utils [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:32:53 user cinder-volume[84696]: DEBUG cinder.utils [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.113(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:32:53 user cinder-volume[84696]: DEBUG cinder.utils [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_8b7f1136-3d20-4cf8-a521-6db1d3a0c8a8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:32:53 user cinder-volume[84696]: DEBUG cinder.utils [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.037(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:32:53 user cinder-volume[84696]: DEBUG cinder.utils [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_f6c0e341-6f00-4e32-bd07-31f954c1ad8a/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:32:53 user cinder-volume[84696]: DEBUG cinder.utils [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.813(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:32:53 user cinder-volume[84696]: DEBUG cinder.utils [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:33:00 user cinder-volume[84696]: DEBUG cinder.utils [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.785(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:33:00 user cinder-volume[84696]: DEBUG cinder.utils [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_f6c0e341-6f00-4e32-bd07-31f954c1ad8a/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:33:00 user cinder-volume[84696]: DEBUG cinder.utils [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.562(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:33:00 user cinder-volume[84696]: DEBUG cinder.utils [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:33:03 user cinder-volume[84696]: DEBUG cinder.utils [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.939(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:33:03 user cinder-volume[84696]: DEBUG cinder.utils [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_8b7f1136-3d20-4cf8-a521-6db1d3a0c8a8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:33:03 user cinder-volume[84696]: DEBUG cinder.utils [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.932(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:33:03 user cinder-volume[84696]: DEBUG cinder.utils [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_2159f1f1-2564-4d84-8110-c1f7d843edb1/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:33:03 user cinder-volume[84696]: DEBUG cinder.utils [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.620(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:33:03 user cinder-volume[84696]: DEBUG cinder.utils [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:33:03 user cinder-volume[84696]: DEBUG cinder.utils [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.732(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:33:03 user cinder-volume[84696]: DEBUG cinder.utils [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:33:04 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:INSERT=3,cinder:SELECT=20,cinder:UPDATE=3 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:33:04 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=18,cinder:UPDATE=9,cinder:INSERT=3 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:33:09 user cinder-volume[84696]: DEBUG cinder.utils [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.776(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:33:09 user cinder-volume[84696]: DEBUG cinder.utils [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:33:09 user cinder-volume[84696]: DEBUG cinder.utils [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 22.002(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:33:09 user cinder-volume[84696]: DEBUG cinder.utils [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_f6c0e341-6f00-4e32-bd07-31f954c1ad8a/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:33:22 user cinder-volume[84696]: DEBUG cinder.utils [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 40.942(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:33:22 user cinder-volume[84696]: DEBUG cinder.utils [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_8b7f1136-3d20-4cf8-a521-6db1d3a0c8a8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:33:22 user cinder-volume[84696]: DEBUG cinder.utils [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 40.935(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:33:22 user cinder-volume[84696]: DEBUG cinder.utils [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_2159f1f1-2564-4d84-8110-c1f7d843edb1/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:33:22 user cinder-volume[84696]: DEBUG cinder.utils [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 40.733(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:33:22 user cinder-volume[84696]: DEBUG cinder.utils [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:33:22 user cinder-volume[84696]: DEBUG cinder.utils [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 40.623(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:33:22 user cinder-volume[84696]: DEBUG cinder.utils [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:33:28 user cinder-volume[84696]: DEBUG cinder.utils [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 41.009(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:33:28 user cinder-volume[84696]: DEBUG cinder.utils [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_f6c0e341-6f00-4e32-bd07-31f954c1ad8a/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:33:28 user cinder-volume[84696]: DEBUG cinder.utils [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 40.785(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:33:28 user cinder-volume[84696]: DEBUG cinder.utils [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:33:31 user cinder-volume[84696]: DEBUG cinder.utils [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 133.863(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:33:31 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:33:31 user cinder-volume[84696]: INFO cinder.message.api [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Creating message record for request_id = req-208fb94c-86af-4459-a279-1c8be9781cb3 Apr 17 16:33:31 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 16:33:31 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (b7c86137-f116-4878-a7bd-6a6acf4ae801) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 16:33:31 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 16:33:31 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:31:18Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-CreateVolumesFromSnapshotTest-Volume-2022997390',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=bf85ce23-1db3-4881-bd7a-b20b520718b8,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='93bbb103c036439da2861c7be15c3e7b',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:31:18Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:31:18Z,use_quota=True,user_id='ae7cdf32438c44b6b3f428e9e0142af1',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 16:33:31 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:31:18Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-CreateVolumesFromSnapshotTest-Volume-2022997390',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=bf85ce23-1db3-4881-bd7a-b20b520718b8,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='93bbb103c036439da2861c7be15c3e7b',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:31:18Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:31:18Z,use_quota=True,user_id='ae7cdf32438c44b6b3f428e9e0142af1',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(bf85ce23-1db3-4881-bd7a-b20b520718b8),volume_id=bf85ce23-1db3-4881-bd7a-b20b520718b8,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': 'bf85ce23-1db3-4881-bd7a-b20b520718b8', 'volume_name': 'volume-bf85ce23-1db3-4881-bd7a-b20b520718b8', 'volume_size': 1}} Apr 17 16:33:31 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'retry': {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']}, 'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': 'bf85ce23-1db3-4881-bd7a-b20b520718b8', 'volume': {'id': 'bf85ce23-1db3-4881-bd7a-b20b520718b8', '_name_id': None, 'ec2_id': None, 'user_id': 'ae7cdf32438c44b6b3f428e9e0142af1', 'project_id': '93bbb103c036439da2861c7be15c3e7b', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-CreateVolumesFromSnapshotTest-Volume-2022997390', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T16:31:17.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-bf85ce23-1db3-4881-bd7a-b20b520718b8', 'name_id': 'bf85ce23-1db3-4881-bd7a-b20b520718b8', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-CreateVolumesFromSnapshotTest-Volume-2022997390', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '93bbb103c036439da2861c7be15c3e7b', 'qos_specs': None, 'reservations': ['b69f8abb-8ed7-4741-8211-ad267bc6119b', '37f3f14f-b707-4dd5-868b-e177f7131d19', 'c87fa212-ff4d-42ec-8c41-d50482477074', '51763846-c4e0-4572-a1af-a15208766488'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'ae7cdf32438c44b6b3f428e9e0142af1', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-CreateVolumesFromSnapshotTest-Volume-2022997390', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '93bbb103c036439da2861c7be15c3e7b', 'qos_specs': None, 'reservations': ['b69f8abb-8ed7-4741-8211-ad267bc6119b', '37f3f14f-b707-4dd5-868b-e177f7131d19', 'c87fa212-ff4d-42ec-8c41-d50482477074', '51763846-c4e0-4572-a1af-a15208766488'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'ae7cdf32438c44b6b3f428e9e0142af1', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': 'ae7cdf32438c44b6b3f428e9e0142af1', 'metadata': {}, 'qos_specs': None}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(bf85ce23-1db3-4881-bd7a-b20b520718b8),volume_id=bf85ce23-1db3-4881-bd7a-b20b520718b8,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:31:18Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-CreateVolumesFromSnapshotTest-Volume-2022997390',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=bf85ce23-1db3-4881-bd7a-b20b520718b8,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='93bbb103c036439da2861c7be15c3e7b',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:31:18Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:31:18Z,use_quota=True,user_id='ae7cdf32438c44b6b3f428e9e0142af1',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 16:33:31 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:31:18Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-CreateVolumesFromSnapshotTest-Volume-2022997390',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=bf85ce23-1db3-4881-bd7a-b20b520718b8,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='93bbb103c036439da2861c7be15c3e7b',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:31:18Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:31:18Z,use_quota=True,user_id='ae7cdf32438c44b6b3f428e9e0142af1',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:31:18Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-CreateVolumesFromSnapshotTest-Volume-2022997390',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=bf85ce23-1db3-4881-bd7a-b20b520718b8,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='93bbb103c036439da2861c7be15c3e7b',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:31:18Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:31:18Z,use_quota=True,user_id='ae7cdf32438c44b6b3f428e9e0142af1',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 16:33:31 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 16:33:31 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 16:33:31 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:33:31 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 16:33:31 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:33:31 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 16:33:31 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:33:31 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 16:33:31 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:33:31 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 16:33:31 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:33:31 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 16:33:31 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:33:31 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 16:33:31 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:33:31 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 16:33:31 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:33:31 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:33:31 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 16:33:31 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 16:33:31 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (b7c86137-f116-4878-a7bd-6a6acf4ae801) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:33:31 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (b7c86137-f116-4878-a7bd-6a6acf4ae801) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:33:31 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (5057cb85-82cb-42d5-a41c-97e2d9560856) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:33:31 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (5057cb85-82cb-42d5-a41c-97e2d9560856) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:33:31 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (e37a7628-6d0e-41e9-a099-bdb921c727f7) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:33:31 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (e37a7628-6d0e-41e9-a099-bdb921c727f7) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:33:31 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (e593ecf2-32cd-45c5-818c-0bdc7f96fd98) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:33:31 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Updating volume bf85ce23-1db3-4881-bd7a-b20b520718b8 with {'scheduled_at': datetime.datetime(2023, 4, 17, 16, 33, 31, 676790), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 16:33:31 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Volume bf85ce23-1db3-4881-bd7a-b20b520718b8: re-scheduling SchedulerAPI.create_volume attempt 1 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 16:33:31 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Volume bf85ce23-1db3-4881-bd7a-b20b520718b8: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 16:33:31 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] deleating volume v_bf85ce23-1db3-4881-bd7a-b20b520718b8 {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 16:33:31 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] get snapshots for volume v_bf85ce23-1db3-4881-bd7a-b20b520718b8 {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 16:33:31 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_bf85ce23-1db3-4881-bd7a-b20b520718b8/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:33:31 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_bf85ce23-1db3-4881-bd7a-b20b520718b8/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:33:31 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Volume reschedule parameters: True retry: {'num_attempts': 2, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 16:33:31 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Flow 'volume_create_manager' (338fa437-d5f6-4108-b918-c18a712406d7) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 16:33:31 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (b5d69e36-7b38-44e2-bd71-3bbb2648de38) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:33:31 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (b5d69e36-7b38-44e2-bd71-3bbb2648de38) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:31:18Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-CreateVolumesFromSnapshotTest-Volume-2022997390',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=bf85ce23-1db3-4881-bd7a-b20b520718b8,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='93bbb103c036439da2861c7be15c3e7b',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:33:32Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:33:32Z,use_quota=True,user_id='ae7cdf32438c44b6b3f428e9e0142af1',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:33:31 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (d75b44e1-b13c-467d-a901-1faee8a72f76) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:33:31 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (d75b44e1-b13c-467d-a901-1faee8a72f76) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:33:31 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (c4e52081-a823-46bf-83f9-4909fef6584e) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:33:31 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (c4e52081-a823-46bf-83f9-4909fef6584e) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': 'bf85ce23-1db3-4881-bd7a-b20b520718b8', 'volume_name': 'volume-bf85ce23-1db3-4881-bd7a-b20b520718b8', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:33:31 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (f33a6fe8-09c9-413d-a0eb-39b9da1708c9) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:33:31 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (f33a6fe8-09c9-413d-a0eb-39b9da1708c9) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:33:31 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (0fa03cdc-ba54-4ba5-a200-38d38cef5097) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:33:31 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Volume bf85ce23-1db3-4881-bd7a-b20b520718b8: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-bf85ce23-1db3-4881-bd7a-b20b520718b8', 'volume_size': 1} Apr 17 16:33:31 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] creating volume v_bf85ce23-1db3-4881-bd7a-b20b520718b8. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 16:33:31 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] create volume {'name': 'v_bf85ce23-1db3-4881-bd7a-b20b520718b8', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 16:33:31 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:33:31 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:33:34 user cinder-volume[84696]: DEBUG cinder.utils [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.804(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:33:34 user cinder-volume[84696]: DEBUG cinder.utils [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:33:34 user cinder-volume[84696]: DEBUG cinder.utils [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.986(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:33:34 user cinder-volume[84696]: DEBUG cinder.utils [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_bf85ce23-1db3-4881-bd7a-b20b520718b8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:33:37 user cinder-volume[84696]: DEBUG cinder.utils [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.876(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:33:37 user cinder-volume[84696]: DEBUG cinder.utils [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:33:37 user cinder-volume[84696]: DEBUG cinder.utils [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.057(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:33:37 user cinder-volume[84696]: DEBUG cinder.utils [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_bf85ce23-1db3-4881-bd7a-b20b520718b8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:33:40 user cinder-volume[84696]: DEBUG cinder.utils [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 139.166(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:33:40 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:33:40 user cinder-volume[84696]: INFO cinder.message.api [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Creating message record for request_id = req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 Apr 17 16:33:40 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 16:33:40 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=9,cinder:INSERT=1,cinder:UPDATE=3 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:33:40 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=12,cinder:UPDATE=4,cinder:INSERT=2 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:33:40 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (35759e7e-48a4-4d93-902f-a7a53b329f1c) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 16:33:40 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 16:33:40 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:31:21Z,deleted=False,deleted_at=None,display_description=None,display_name='CinderUnicodeTest塵㼗‽',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=7384f51d-c3a2-4eea-a968-ec043946989f,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='9ccd4e98adcb4fb18f651ead3736305a',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:31:22Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:31:22Z,use_quota=True,user_id='d3c58fb72fb443e19315cc15a76767fe',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 16:33:40 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:31:21Z,deleted=False,deleted_at=None,display_description=None,display_name='CinderUnicodeTest塵㼗‽',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=7384f51d-c3a2-4eea-a968-ec043946989f,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='9ccd4e98adcb4fb18f651ead3736305a',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:31:22Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:31:22Z,use_quota=True,user_id='d3c58fb72fb443e19315cc15a76767fe',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(7384f51d-c3a2-4eea-a968-ec043946989f),volume_id=7384f51d-c3a2-4eea-a968-ec043946989f,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': '7384f51d-c3a2-4eea-a968-ec043946989f', 'volume_name': 'volume-7384f51d-c3a2-4eea-a968-ec043946989f', 'volume_size': 1}} Apr 17 16:33:40 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'retry': {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']}, 'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': '7384f51d-c3a2-4eea-a968-ec043946989f', 'volume': {'id': '7384f51d-c3a2-4eea-a968-ec043946989f', '_name_id': None, 'ec2_id': None, 'user_id': 'd3c58fb72fb443e19315cc15a76767fe', 'project_id': '9ccd4e98adcb4fb18f651ead3736305a', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'CinderUnicodeTest塵㼗‽', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T16:31:21.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-7384f51d-c3a2-4eea-a968-ec043946989f', 'name_id': '7384f51d-c3a2-4eea-a968-ec043946989f', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'CinderUnicodeTest塵㼗‽', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '9ccd4e98adcb4fb18f651ead3736305a', 'qos_specs': None, 'reservations': ['e840cac3-58bf-4622-96ed-8d715415019e', '88e95250-2b05-45ff-b59d-697235ec7436', '29eb19dc-9f45-4e12-a7b9-01875910c4ae', 'be37e533-c8a6-464f-942b-ce65599da74b'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'd3c58fb72fb443e19315cc15a76767fe', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'CinderUnicodeTest塵㼗‽', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '9ccd4e98adcb4fb18f651ead3736305a', 'qos_specs': None, 'reservations': ['e840cac3-58bf-4622-96ed-8d715415019e', '88e95250-2b05-45ff-b59d-697235ec7436', '29eb19dc-9f45-4e12-a7b9-01875910c4ae', 'be37e533-c8a6-464f-942b-ce65599da74b'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'd3c58fb72fb443e19315cc15a76767fe', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': 'd3c58fb72fb443e19315cc15a76767fe', 'metadata': {}, 'qos_specs': None}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(7384f51d-c3a2-4eea-a968-ec043946989f),volume_id=7384f51d-c3a2-4eea-a968-ec043946989f,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:31:21Z,deleted=False,deleted_at=None,display_description=None,display_name='CinderUnicodeTest塵㼗‽',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=7384f51d-c3a2-4eea-a968-ec043946989f,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='9ccd4e98adcb4fb18f651ead3736305a',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:31:22Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:31:22Z,use_quota=True,user_id='d3c58fb72fb443e19315cc15a76767fe',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 16:33:40 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:31:21Z,deleted=False,deleted_at=None,display_description=None,display_name='CinderUnicodeTest塵㼗‽',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=7384f51d-c3a2-4eea-a968-ec043946989f,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='9ccd4e98adcb4fb18f651ead3736305a',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:31:22Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:31:22Z,use_quota=True,user_id='d3c58fb72fb443e19315cc15a76767fe',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:31:21Z,deleted=False,deleted_at=None,display_description=None,display_name='CinderUnicodeTest塵㼗‽',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=7384f51d-c3a2-4eea-a968-ec043946989f,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='9ccd4e98adcb4fb18f651ead3736305a',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:31:22Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:31:22Z,use_quota=True,user_id='d3c58fb72fb443e19315cc15a76767fe',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 16:33:40 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 16:33:40 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 16:33:40 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:33:40 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 16:33:40 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:33:40 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 16:33:40 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:33:40 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 16:33:40 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:33:40 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 16:33:40 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:33:40 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 16:33:40 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:33:40 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 16:33:40 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:33:40 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 16:33:40 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:33:40 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:33:40 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 16:33:40 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 16:33:40 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (35759e7e-48a4-4d93-902f-a7a53b329f1c) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:33:40 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (35759e7e-48a4-4d93-902f-a7a53b329f1c) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:33:40 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (36d794c2-c2f3-4d60-83e4-7e5b252762bf) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:33:40 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (36d794c2-c2f3-4d60-83e4-7e5b252762bf) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:33:40 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (1a3641ba-e8e5-41de-82c5-13ae38a18b5c) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:33:40 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (1a3641ba-e8e5-41de-82c5-13ae38a18b5c) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:33:40 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (6d2f64ff-fe0b-4f90-b3bd-5481f65a4834) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:33:40 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Updating volume 7384f51d-c3a2-4eea-a968-ec043946989f with {'scheduled_at': datetime.datetime(2023, 4, 17, 16, 33, 40, 893779), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 16:33:40 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Volume 7384f51d-c3a2-4eea-a968-ec043946989f: re-scheduling SchedulerAPI.create_volume attempt 1 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 16:33:40 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Volume 7384f51d-c3a2-4eea-a968-ec043946989f: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 16:33:40 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] deleating volume v_7384f51d-c3a2-4eea-a968-ec043946989f {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 16:33:40 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] get snapshots for volume v_7384f51d-c3a2-4eea-a968-ec043946989f {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 16:33:40 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_7384f51d-c3a2-4eea-a968-ec043946989f/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:33:40 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_7384f51d-c3a2-4eea-a968-ec043946989f/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:33:41 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Volume reschedule parameters: True retry: {'num_attempts': 2, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 16:33:41 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Flow 'volume_create_manager' (101546a3-efad-4df1-87b5-2dbcadd98f8e) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 16:33:41 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (80489580-749f-442e-9351-662f3d5b9906) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:33:41 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (80489580-749f-442e-9351-662f3d5b9906) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:31:21Z,deleted=False,deleted_at=None,display_description=None,display_name='CinderUnicodeTest塵㼗‽',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=7384f51d-c3a2-4eea-a968-ec043946989f,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='9ccd4e98adcb4fb18f651ead3736305a',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:33:41Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:33:41Z,use_quota=True,user_id='d3c58fb72fb443e19315cc15a76767fe',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:33:41 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (9b5a87a7-4b89-42c4-8980-3b6a59e23bc8) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:33:41 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (9b5a87a7-4b89-42c4-8980-3b6a59e23bc8) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:33:41 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (5e52c3ca-bbe7-4158-8f40-a24d08d03c17) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:33:41 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (5e52c3ca-bbe7-4158-8f40-a24d08d03c17) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': '7384f51d-c3a2-4eea-a968-ec043946989f', 'volume_name': 'volume-7384f51d-c3a2-4eea-a968-ec043946989f', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:33:41 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (1b532189-727b-4106-840c-e32a634bef49) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:33:41 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (1b532189-727b-4106-840c-e32a634bef49) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:33:41 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (381d9c70-eeb0-489c-a13e-dc36e3289945) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:33:41 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Volume 7384f51d-c3a2-4eea-a968-ec043946989f: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-7384f51d-c3a2-4eea-a968-ec043946989f', 'volume_size': 1} Apr 17 16:33:41 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] creating volume v_7384f51d-c3a2-4eea-a968-ec043946989f. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 16:33:41 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] create volume {'name': 'v_7384f51d-c3a2-4eea-a968-ec043946989f', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 16:33:41 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:33:41 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:33:43 user cinder-volume[84696]: DEBUG cinder.utils [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.979(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:33:43 user cinder-volume[84696]: DEBUG cinder.utils [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_7384f51d-c3a2-4eea-a968-ec043946989f/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:33:43 user cinder-volume[84696]: DEBUG cinder.utils [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.754(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:33:43 user cinder-volume[84696]: DEBUG cinder.utils [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:33:43 user cinder-volume[84696]: DEBUG cinder.utils [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.203(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:33:43 user cinder-volume[84696]: DEBUG cinder.utils [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_bf85ce23-1db3-4881-bd7a-b20b520718b8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:33:43 user cinder-volume[84696]: DEBUG cinder.utils [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 137.762(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:33:43 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:33:43 user cinder-volume[84696]: INFO cinder.message.api [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Creating message record for request_id = req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b Apr 17 16:33:43 user cinder-volume[84696]: DEBUG cinder.utils [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.026(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:33:43 user cinder-volume[84696]: DEBUG cinder.utils [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:33:43 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 16:33:43 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (1b692ae0-b6bd-4f39-b4c3-14746eef8df0) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 16:33:43 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 16:33:43 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:31:26Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumeQuotasAdminTestJSON-Volume-2080868017',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=480a94a0-d1c9-46c0-bf9f-c42f907b4b82,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='6e4c2ec9177349a08ce53e07b24ad79c',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:31:26Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:31:26Z,use_quota=True,user_id='ab5017da30ac4d47ba665ff135deb309',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 16:33:43 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:31:26Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumeQuotasAdminTestJSON-Volume-2080868017',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=480a94a0-d1c9-46c0-bf9f-c42f907b4b82,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='6e4c2ec9177349a08ce53e07b24ad79c',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:31:26Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:31:26Z,use_quota=True,user_id='ab5017da30ac4d47ba665ff135deb309',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(480a94a0-d1c9-46c0-bf9f-c42f907b4b82),volume_id=480a94a0-d1c9-46c0-bf9f-c42f907b4b82,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': '480a94a0-d1c9-46c0-bf9f-c42f907b4b82', 'volume_name': 'volume-480a94a0-d1c9-46c0-bf9f-c42f907b4b82', 'volume_size': 1}} Apr 17 16:33:43 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'retry': {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']}, 'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': '480a94a0-d1c9-46c0-bf9f-c42f907b4b82', 'volume': {'id': '480a94a0-d1c9-46c0-bf9f-c42f907b4b82', '_name_id': None, 'ec2_id': None, 'user_id': 'ab5017da30ac4d47ba665ff135deb309', 'project_id': '6e4c2ec9177349a08ce53e07b24ad79c', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-VolumeQuotasAdminTestJSON-Volume-2080868017', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T16:31:25.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-480a94a0-d1c9-46c0-bf9f-c42f907b4b82', 'name_id': '480a94a0-d1c9-46c0-bf9f-c42f907b4b82', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumeQuotasAdminTestJSON-Volume-2080868017', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '6e4c2ec9177349a08ce53e07b24ad79c', 'qos_specs': None, 'reservations': ['00e83981-2b5c-4129-be4e-d251babb9099', '4e65b249-f315-4ce2-9106-e50bca02bf14', '166cbb51-86ba-4d39-9450-64b18ab85006', '47198bf3-67c0-44fb-9cca-8e33c9ae31c6'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'ab5017da30ac4d47ba665ff135deb309', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumeQuotasAdminTestJSON-Volume-2080868017', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '6e4c2ec9177349a08ce53e07b24ad79c', 'qos_specs': None, 'reservations': ['00e83981-2b5c-4129-be4e-d251babb9099', '4e65b249-f315-4ce2-9106-e50bca02bf14', '166cbb51-86ba-4d39-9450-64b18ab85006', '47198bf3-67c0-44fb-9cca-8e33c9ae31c6'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'ab5017da30ac4d47ba665ff135deb309', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': 'ab5017da30ac4d47ba665ff135deb309', 'metadata': {}, 'qos_specs': None}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(480a94a0-d1c9-46c0-bf9f-c42f907b4b82),volume_id=480a94a0-d1c9-46c0-bf9f-c42f907b4b82,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:31:26Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumeQuotasAdminTestJSON-Volume-2080868017',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=480a94a0-d1c9-46c0-bf9f-c42f907b4b82,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='6e4c2ec9177349a08ce53e07b24ad79c',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:31:26Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:31:26Z,use_quota=True,user_id='ab5017da30ac4d47ba665ff135deb309',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 16:33:43 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:31:26Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumeQuotasAdminTestJSON-Volume-2080868017',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=480a94a0-d1c9-46c0-bf9f-c42f907b4b82,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='6e4c2ec9177349a08ce53e07b24ad79c',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:31:26Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:31:26Z,use_quota=True,user_id='ab5017da30ac4d47ba665ff135deb309',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:31:26Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumeQuotasAdminTestJSON-Volume-2080868017',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=480a94a0-d1c9-46c0-bf9f-c42f907b4b82,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='6e4c2ec9177349a08ce53e07b24ad79c',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:31:26Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:31:26Z,use_quota=True,user_id='ab5017da30ac4d47ba665ff135deb309',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 16:33:43 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 16:33:43 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 16:33:43 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:33:43 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 16:33:43 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:33:43 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 16:33:43 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:33:43 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 16:33:43 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:33:43 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 16:33:43 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:33:43 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 16:33:43 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:33:43 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 16:33:43 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:33:43 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 16:33:43 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:33:43 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:33:43 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 16:33:43 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 16:33:43 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (1b692ae0-b6bd-4f39-b4c3-14746eef8df0) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:33:43 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (1b692ae0-b6bd-4f39-b4c3-14746eef8df0) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:33:43 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (2a754555-aa79-41c5-8b60-17f320357232) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:33:43 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (2a754555-aa79-41c5-8b60-17f320357232) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:33:43 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (7fbe4791-10fa-4579-af06-c077988388ea) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:33:43 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (7fbe4791-10fa-4579-af06-c077988388ea) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:33:43 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (8e29d783-ca28-40a8-aa00-4942cc3231e3) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:33:43 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Updating volume 480a94a0-d1c9-46c0-bf9f-c42f907b4b82 with {'scheduled_at': datetime.datetime(2023, 4, 17, 16, 33, 43, 973362), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 16:33:43 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Volume 480a94a0-d1c9-46c0-bf9f-c42f907b4b82: re-scheduling SchedulerAPI.create_volume attempt 1 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 16:33:44 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Volume 480a94a0-d1c9-46c0-bf9f-c42f907b4b82: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 16:33:44 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] deleating volume v_480a94a0-d1c9-46c0-bf9f-c42f907b4b82 {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 16:33:44 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] get snapshots for volume v_480a94a0-d1c9-46c0-bf9f-c42f907b4b82 {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 16:33:44 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_480a94a0-d1c9-46c0-bf9f-c42f907b4b82/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:33:44 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_480a94a0-d1c9-46c0-bf9f-c42f907b4b82/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:33:44 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Volume reschedule parameters: True retry: {'num_attempts': 2, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 16:33:44 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Flow 'volume_create_manager' (7daf9069-8e3f-4c38-8e02-de0fa1668442) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 16:33:44 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (fb48d423-dc2a-4726-8949-0eb0bb594051) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:33:44 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (fb48d423-dc2a-4726-8949-0eb0bb594051) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:31:26Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumeQuotasAdminTestJSON-Volume-2080868017',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=480a94a0-d1c9-46c0-bf9f-c42f907b4b82,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='6e4c2ec9177349a08ce53e07b24ad79c',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:33:44Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:33:44Z,use_quota=True,user_id='ab5017da30ac4d47ba665ff135deb309',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:33:44 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (f7be13b7-4f59-4082-8999-dd5515bc049f) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:33:44 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (f7be13b7-4f59-4082-8999-dd5515bc049f) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:33:44 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (16ae2e5e-0eab-4600-81d8-ae7d86c5d940) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:33:44 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (16ae2e5e-0eab-4600-81d8-ae7d86c5d940) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': '480a94a0-d1c9-46c0-bf9f-c42f907b4b82', 'volume_name': 'volume-480a94a0-d1c9-46c0-bf9f-c42f907b4b82', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:33:44 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (7b27e4ea-25e7-48a3-8aeb-408c54c046ed) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:33:44 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (7b27e4ea-25e7-48a3-8aeb-408c54c046ed) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:33:44 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (367ce42b-db7b-40b8-9c29-a43986a55b2d) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:33:44 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Volume 480a94a0-d1c9-46c0-bf9f-c42f907b4b82: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-480a94a0-d1c9-46c0-bf9f-c42f907b4b82', 'volume_size': 1} Apr 17 16:33:44 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] creating volume v_480a94a0-d1c9-46c0-bf9f-c42f907b4b82. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 16:33:44 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] create volume {'name': 'v_480a94a0-d1c9-46c0-bf9f-c42f907b4b82', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 16:33:44 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:33:44 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:33:46 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 139.512(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:33:46 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:33:46 user cinder-volume[84696]: ERROR oslo_service.periodic_task [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Error during VolumeManager.publish_service_capabilities: TypeError: 'NoneType' object is not subscriptable Apr 17 16:33:46 user cinder-volume[84696]: ERROR oslo_service.periodic_task Traceback (most recent call last): Apr 17 16:33:46 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/usr/local/lib/python3.10/dist-packages/oslo_service/periodic_task.py", line 216, in run_periodic_tasks Apr 17 16:33:46 user cinder-volume[84696]: ERROR oslo_service.periodic_task task(self, context) Apr 17 16:33:46 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/manager.py", line 2876, in publish_service_capabilities Apr 17 16:33:46 user cinder-volume[84696]: ERROR oslo_service.periodic_task self._report_driver_status(context) Apr 17 16:33:46 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/manager.py", line 2755, in _report_driver_status Apr 17 16:33:46 user cinder-volume[84696]: ERROR oslo_service.periodic_task volume_stats = get_stats() Apr 17 16:33:46 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/usr/local/lib/python3.10/dist-packages/oslo_utils/timeutils.py", line 319, in wrapper Apr 17 16:33:46 user cinder-volume[84696]: ERROR oslo_service.periodic_task result = func(*args, **kwargs) Apr 17 16:33:46 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/manager.py", line 2753, in get_stats Apr 17 16:33:46 user cinder-volume[84696]: ERROR oslo_service.periodic_task return self.driver.get_volume_stats(refresh=True) Apr 17 16:33:46 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/driver.py", line 666, in get_volume_stats Apr 17 16:33:46 user cinder-volume[84696]: ERROR oslo_service.periodic_task self._update_volume_stats() Apr 17 16:33:46 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 726, in _update_volume_stats Apr 17 16:33:46 user cinder-volume[84696]: ERROR oslo_service.periodic_task pool_stats = self.ra.get_pool_stats() Apr 17 16:33:46 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 897, in get_pool_stats Apr 17 16:33:46 user cinder-volume[84696]: ERROR oslo_service.periodic_task if not resp["error"] and resp["code"] == 200: Apr 17 16:33:46 user cinder-volume[84696]: ERROR oslo_service.periodic_task TypeError: 'NoneType' object is not subscriptable Apr 17 16:33:46 user cinder-volume[84696]: ERROR oslo_service.periodic_task Apr 17 16:33:46 user cinder-volume[84696]: DEBUG cinder.utils [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.950(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:33:46 user cinder-volume[84696]: DEBUG cinder.utils [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_480a94a0-d1c9-46c0-bf9f-c42f907b4b82/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:33:46 user cinder-volume[84696]: DEBUG cinder.utils [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.783(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:33:46 user cinder-volume[84696]: DEBUG cinder.utils [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:33:46 user cinder-volume[84696]: DEBUG cinder.utils [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.056(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:33:46 user cinder-volume[84696]: DEBUG cinder.utils [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_7384f51d-c3a2-4eea-a968-ec043946989f/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:33:46 user cinder-volume[84696]: WARNING oslo.service.loopingcall [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Function 'cinder.service.Service.periodic_tasks' run outlasted interval by 79.52 sec Apr 17 16:33:46 user cinder-volume[84696]: DEBUG cinder.utils [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.831(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:33:46 user cinder-volume[84696]: DEBUG cinder.utils [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:33:46 user cinder-volume[84696]: DEBUG oslo_service.periodic_task [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Running periodic task VolumeManager.publish_service_capabilities {{(pid=84696) run_periodic_tasks /usr/local/lib/python3.10/dist-packages/oslo_service/periodic_task.py:210}} Apr 17 16:33:46 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Updating volume stats {{(pid=84696) _update_volume_stats /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:724}} Apr 17 16:33:46 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Get pool Pool-0 fsprops {{(pid=84696) get_pool_stats /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:894}} Apr 17 16:33:46 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:33:46 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:33:50 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 3.067(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:33:50 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:33:50 user cinder-volume[84696]: DEBUG cinder.utils [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.025(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:33:50 user cinder-volume[84696]: DEBUG cinder.utils [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_480a94a0-d1c9-46c0-bf9f-c42f907b4b82/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:33:50 user cinder-volume[84696]: DEBUG cinder.utils [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.858(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:33:50 user cinder-volume[84696]: DEBUG cinder.utils [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:33:53 user cinder-volume[84696]: DEBUG cinder.utils [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 139.408(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:33:53 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_53029faa-77b2-467d-8a64-eb5ef129e41c/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:33:53 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (97deb593-97b0-4a98-b1d1-56dbf3eaf113) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 16:33:53 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (dba2830d-e45d-44b7-94d6-02fe7969b66b) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:33:53 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (dba2830d-e45d-44b7-94d6-02fe7969b66b) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:33:53 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Flow 'volume_create_manager' (7225a004-fd24-4f3f-9cc2-ab67de608fa3) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 16:33:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-8d765f93-bce0-4815-ac07-22b8b3bd2556 tempest-VolumesDeleteCascade-645320261 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 16:33:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 16:33:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 16:33:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 16:33:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 16:33:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 16:33:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 16:33:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 16:33:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 16:33:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 16:33:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 16:33:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 16:33:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 16:33:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 16:33:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 16:33:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 16:33:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 16:33:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 16:33:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 16:33:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 16:33:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 16:33:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 16:33:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 16:33:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 16:33:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:33:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 16:33:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:33:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 16:33:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:33:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 16:33:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:33:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 16:33:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:33:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 16:33:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:33:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 16:33:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:33:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 16:33:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:33:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:33:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 16:33:53 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 16:33:53 user cinder-volume[84696]: DEBUG cinder.utils [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 11.987(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:33:53 user cinder-volume[84696]: DEBUG cinder.utils [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:33:53 user cinder-volume[84696]: DEBUG cinder.utils [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.214(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:33:53 user cinder-volume[84696]: DEBUG cinder.utils [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_7384f51d-c3a2-4eea-a968-ec043946989f/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:33:53 user cinder-volume[84696]: DEBUG cinder.utils [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.436(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:33:53 user cinder-volume[84696]: DEBUG cinder.utils [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_bf85ce23-1db3-4881-bd7a-b20b520718b8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:33:53 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.157(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:33:53 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:33:53 user cinder-volume[84696]: DEBUG cinder.utils [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.258(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:33:53 user cinder-volume[84696]: DEBUG cinder.utils [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:33:54 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=9,cinder:INSERT=3,cinder:UPDATE=1 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:33:56 user cinder-volume[84696]: DEBUG cinder.utils [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 11.996(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:33:56 user cinder-volume[84696]: DEBUG cinder.utils [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:33:56 user cinder-volume[84696]: DEBUG cinder.utils [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.166(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:33:56 user cinder-volume[84696]: DEBUG cinder.utils [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_480a94a0-d1c9-46c0-bf9f-c42f907b4b82/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:33:56 user cinder-volume[84696]: DEBUG cinder.utils [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 74.527(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:33:56 user cinder-volume[84696]: DEBUG cinder.utils [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:33:56 user cinder-volume[84696]: DEBUG cinder.utils [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 74.740(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:33:56 user cinder-volume[84696]: DEBUG cinder.utils [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_8b7f1136-3d20-4cf8-a521-6db1d3a0c8a8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:33:56 user cinder-volume[84696]: DEBUG cinder.utils [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 74.733(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:33:56 user cinder-volume[84696]: DEBUG cinder.utils [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_2159f1f1-2564-4d84-8110-c1f7d843edb1/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:33:56 user cinder-volume[84696]: DEBUG cinder.utils [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 74.420(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:33:56 user cinder-volume[84696]: DEBUG cinder.utils [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:33:56 user cinder-volume[84696]: DEBUG cinder.utils [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 139.419(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:33:56 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_35a67ca3-0f17-4a6c-a359-9173ce5f1aca/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:33:56 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (3ba1e240-9772-4d59-9646-a5e3f6b9937f) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 16:33:56 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (db751aca-e239-4852-9d10-12075ebac73b) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:33:56 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (db751aca-e239-4852-9d10-12075ebac73b) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:33:56 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Flow 'volume_create_manager' (c0601d20-c7d6-4708-ac77-15a1cd3fc6e2) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 16:33:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-61f60e22-802a-4068-b929-14f669a47260 tempest-SnapshotsActionsTest-1211835385 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 16:33:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 16:33:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 16:33:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 16:33:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 16:33:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 16:33:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 16:33:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 16:33:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 16:33:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 16:33:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 16:33:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 16:33:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 16:33:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 16:33:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 16:33:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 16:33:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 16:33:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 16:33:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 16:33:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 16:33:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 16:33:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 16:33:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 16:33:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 16:33:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:33:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 16:33:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:33:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 16:33:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:33:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 16:33:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:33:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 16:33:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:33:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 16:33:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:33:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 16:33:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:33:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 16:33:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:33:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:33:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 16:33:56 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 16:33:59 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.279(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:33:59 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:34:02 user cinder-volume[84696]: DEBUG cinder.utils [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 74.576(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:34:02 user cinder-volume[84696]: DEBUG cinder.utils [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:34:02 user cinder-volume[84696]: DEBUG cinder.utils [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 74.802(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:34:02 user cinder-volume[84696]: DEBUG cinder.utils [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_f6c0e341-6f00-4e32-bd07-31f954c1ad8a/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:34:02 user cinder-volume[84696]: DEBUG cinder.utils [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.187(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:34:02 user cinder-volume[84696]: DEBUG cinder.utils [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:34:02 user cinder-volume[84696]: DEBUG cinder.utils [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.414(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:34:02 user cinder-volume[84696]: DEBUG cinder.utils [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_7384f51d-c3a2-4eea-a968-ec043946989f/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:34:05 user cinder-volume[84696]: DEBUG cinder.utils [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.215(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:34:05 user cinder-volume[84696]: DEBUG cinder.utils [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:34:05 user cinder-volume[84696]: DEBUG cinder.utils [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.385(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:34:05 user cinder-volume[84696]: DEBUG cinder.utils [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_480a94a0-d1c9-46c0-bf9f-c42f907b4b82/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:34:08 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.495(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:34:08 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:34:11 user cinder-volume[84696]: DEBUG cinder.utils [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.848(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:34:11 user cinder-volume[84696]: DEBUG cinder.utils [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_bf85ce23-1db3-4881-bd7a-b20b520718b8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:34:11 user cinder-volume[84696]: DEBUG cinder.utils [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.670(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:34:11 user cinder-volume[84696]: DEBUG cinder.utils [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:34:14 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=3 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:34:14 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:UPDATE=8,cinder:SELECT=15 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:34:21 user cinder-volume[84696]: DEBUG cinder.utils [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 40.451(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:34:21 user cinder-volume[84696]: DEBUG cinder.utils [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_7384f51d-c3a2-4eea-a968-ec043946989f/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:34:21 user cinder-volume[84696]: DEBUG cinder.utils [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 40.227(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:34:21 user cinder-volume[84696]: DEBUG cinder.utils [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:34:24 user cinder-volume[84696]: DEBUG cinder.utils [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 40.251(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:34:24 user cinder-volume[84696]: DEBUG cinder.utils [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:34:24 user cinder-volume[84696]: DEBUG cinder.utils [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 40.422(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:34:24 user cinder-volume[84696]: DEBUG cinder.utils [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_480a94a0-d1c9-46c0-bf9f-c42f907b4b82/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:34:27 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 40.535(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:34:27 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:34:44 user cinder-volume[84696]: DEBUG cinder.utils [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 72.564(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:34:44 user cinder-volume[84696]: DEBUG cinder.utils [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:34:44 user cinder-volume[84696]: DEBUG cinder.utils [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 72.745(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:34:44 user cinder-volume[84696]: DEBUG cinder.utils [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_bf85ce23-1db3-4881-bd7a-b20b520718b8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:34:44 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=6,cinder:UPDATE=2 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:34:44 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=5,cinder:UPDATE=1 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:34:47 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Volume reschedule parameters: True retry: {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 16:34:47 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Flow 'volume_create_manager' (f7e5f1e4-b757-4576-8d8b-71f8f7d2a5fb) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 16:34:47 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (a65b7553-8ac5-4db0-ae38-47b3a3f6d21d) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:34:47 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (a65b7553-8ac5-4db0-ae38-47b3a3f6d21d) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:34:47Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesMetadataTest-Volume-1367745540',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=e1bb78ac-29e3-46df-9bbe-3b152e39b69a,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='7f9b364e902a445082273ba68da887b4',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:34:47Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:34:47Z,use_quota=True,user_id='ad9fef0ffef84b1783a6075d92bb27c1',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:34:47 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (375784c9-e555-42dd-8541-5e77aed44c93) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:34:47 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (375784c9-e555-42dd-8541-5e77aed44c93) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:34:47 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (c6fb897e-a233-4286-b4e4-260cb051ef6b) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:34:47 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (c6fb897e-a233-4286-b4e4-260cb051ef6b) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': 'e1bb78ac-29e3-46df-9bbe-3b152e39b69a', 'volume_name': 'volume-e1bb78ac-29e3-46df-9bbe-3b152e39b69a', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:34:47 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (9fc9f51e-0719-4af4-9c62-1e1b064ac7ee) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:34:47 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (9fc9f51e-0719-4af4-9c62-1e1b064ac7ee) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:34:47 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (f5ba8a12-e4f8-4a35-a487-af09bc1fce02) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:34:47 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Volume e1bb78ac-29e3-46df-9bbe-3b152e39b69a: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-e1bb78ac-29e3-46df-9bbe-3b152e39b69a', 'volume_size': 1} Apr 17 16:34:47 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] creating volume v_e1bb78ac-29e3-46df-9bbe-3b152e39b69a. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 16:34:47 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] create volume {'name': 'v_e1bb78ac-29e3-46df-9bbe-3b152e39b69a', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 16:34:47 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:34:47 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:34:48 user cinder-volume[84696]: DEBUG cinder.utils [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 0.950(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:34:48 user cinder-volume[84696]: DEBUG cinder.utils [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:34:51 user cinder-volume[84696]: DEBUG cinder.utils [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 4.018(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:34:51 user cinder-volume[84696]: DEBUG cinder.utils [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:34:54 user cinder-volume[84696]: DEBUG cinder.utils [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 73.377(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:34:54 user cinder-volume[84696]: DEBUG cinder.utils [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:34:54 user cinder-volume[84696]: DEBUG cinder.utils [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 73.605(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:34:54 user cinder-volume[84696]: DEBUG cinder.utils [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_7384f51d-c3a2-4eea-a968-ec043946989f/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:34:57 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:UPDATE=2,cinder:SELECT=4 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:34:57 user cinder-volume[84696]: DEBUG cinder.utils [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 73.576(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:34:57 user cinder-volume[84696]: DEBUG cinder.utils [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_480a94a0-d1c9-46c0-bf9f-c42f907b4b82/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:34:57 user cinder-volume[84696]: DEBUG cinder.utils [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 10.167(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:34:57 user cinder-volume[84696]: DEBUG cinder.utils [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:34:57 user cinder-volume[84696]: DEBUG cinder.utils [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 73.410(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:34:57 user cinder-volume[84696]: DEBUG cinder.utils [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:35:00 user cinder-volume[84696]: DEBUG cinder.utils [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 139.007(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:35:00 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:35:00 user cinder-volume[84696]: INFO cinder.message.api [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Creating message record for request_id = req-ccbb52f2-042b-41e0-9837-623e8a9a2738 Apr 17 16:35:00 user cinder-volume[84696]: DEBUG cinder.utils [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 139.214(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:35:00 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_2159f1f1-2564-4d84-8110-c1f7d843edb1/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:35:00 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (cdbb57d0-c62f-4c33-aeb2-e64d3e362f8b) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 16:35:00 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (3d429a40-9c64-404f-86b3-68a0a8d4e377) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:35:00 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (3d429a40-9c64-404f-86b3-68a0a8d4e377) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:35:00 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Flow 'volume_create_manager' (6530cd85-6c71-42b4-86dd-de43ea2dfb9d) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 16:35:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 16:35:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 16:35:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 16:35:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 16:35:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 16:35:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 16:35:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 16:35:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 16:35:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 16:35:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 16:35:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 16:35:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 16:35:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 16:35:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 16:35:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 16:35:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 16:35:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 16:35:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 16:35:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 16:35:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 16:35:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 16:35:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 16:35:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 16:35:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 16:35:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:35:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 16:35:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:35:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 16:35:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:35:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 16:35:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:35:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 16:35:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:35:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 16:35:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:35:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 16:35:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:35:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 16:35:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:35:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:35:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 16:35:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 16:35:00 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 73.713(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:35:00 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:35:00 user cinder-volume[84696]: DEBUG cinder.utils [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 138.919(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:35:00 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:35:00 user cinder-volume[84696]: INFO cinder.message.api [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Creating message record for request_id = req-34802d26-273b-4fa3-a0d4-af12da0d916e Apr 17 16:35:00 user cinder-volume[84696]: DEBUG cinder.utils [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 139.243(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:35:00 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_8b7f1136-3d20-4cf8-a521-6db1d3a0c8a8/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:35:00 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (ea985edf-0b1d-4ffc-ade1-837a9b4be8b3) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 16:35:00 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (ee09d421-0a04-428d-b136-daa84c67a8f4) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:35:00 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (ee09d421-0a04-428d-b136-daa84c67a8f4) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:35:00 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Flow 'volume_create_manager' (29dad5d0-e988-49b6-af92-fc21216548da) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 16:35:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 16:35:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 16:35:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 16:35:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 16:35:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 16:35:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 16:35:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 16:35:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 16:35:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 16:35:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 16:35:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 16:35:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 16:35:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 16:35:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 16:35:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 16:35:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 16:35:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 16:35:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 16:35:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 16:35:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 16:35:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 16:35:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 16:35:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 16:35:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 16:35:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:35:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 16:35:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:35:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 16:35:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:35:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 16:35:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:35:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 16:35:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:35:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 16:35:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:35:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 16:35:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:35:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 16:35:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:35:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:35:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 16:35:00 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 16:35:00 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 16:35:00 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (721e1133-e9db-4d81-8eb1-755d8155c29f) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 16:35:00 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 16:35:00 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:28:03Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesSnapshotListTestJSON-Volume-552939668',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=2159f1f1-2564-4d84-8110-c1f7d843edb1,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='c4b1a8a0074e4798bd385ee96e8ac8b7',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:32:42Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:32:42Z,use_quota=True,user_id='476228814d42467c880dce2b1c5c5113',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 16:35:00 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:28:03Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesSnapshotListTestJSON-Volume-552939668',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=2159f1f1-2564-4d84-8110-c1f7d843edb1,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='c4b1a8a0074e4798bd385ee96e8ac8b7',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:32:42Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:32:42Z,use_quota=True,user_id='476228814d42467c880dce2b1c5c5113',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(2159f1f1-2564-4d84-8110-c1f7d843edb1),volume_id=2159f1f1-2564-4d84-8110-c1f7d843edb1,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': '2159f1f1-2564-4d84-8110-c1f7d843edb1', 'volume_name': 'volume-2159f1f1-2564-4d84-8110-c1f7d843edb1', 'volume_size': 1}} Apr 17 16:35:00 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': '2159f1f1-2564-4d84-8110-c1f7d843edb1', 'volume': {'id': '2159f1f1-2564-4d84-8110-c1f7d843edb1', '_name_id': None, 'ec2_id': None, 'user_id': '476228814d42467c880dce2b1c5c5113', 'project_id': 'c4b1a8a0074e4798bd385ee96e8ac8b7', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-VolumesSnapshotListTestJSON-Volume-552939668', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'admin_metadata': {}, 'glance_metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T16:28:02.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-2159f1f1-2564-4d84-8110-c1f7d843edb1', 'name_id': '2159f1f1-2564-4d84-8110-c1f7d843edb1', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumesSnapshotListTestJSON-Volume-552939668', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': 'c4b1a8a0074e4798bd385ee96e8ac8b7', 'qos_specs': None, 'reservations': ['0c108957-403f-4dc6-adfa-0a0de976576c', 'e72136a8-7407-4f95-a72e-b484cca53b29', 'cc5aa602-fa98-4b80-9896-3829bb01e999', 'ec5862b0-0012-4eba-b940-a35906fa2d44'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': '476228814d42467c880dce2b1c5c5113', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumesSnapshotListTestJSON-Volume-552939668', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': 'c4b1a8a0074e4798bd385ee96e8ac8b7', 'qos_specs': None, 'reservations': ['0c108957-403f-4dc6-adfa-0a0de976576c', 'e72136a8-7407-4f95-a72e-b484cca53b29', 'cc5aa602-fa98-4b80-9896-3829bb01e999', 'ec5862b0-0012-4eba-b940-a35906fa2d44'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': '476228814d42467c880dce2b1c5c5113', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': '476228814d42467c880dce2b1c5c5113', 'metadata': {}, 'qos_specs': None, 'retry': {'num_attempts': 3, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']}}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(2159f1f1-2564-4d84-8110-c1f7d843edb1),volume_id=2159f1f1-2564-4d84-8110-c1f7d843edb1,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:28:03Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesSnapshotListTestJSON-Volume-552939668',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=2159f1f1-2564-4d84-8110-c1f7d843edb1,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='c4b1a8a0074e4798bd385ee96e8ac8b7',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:32:42Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:32:42Z,use_quota=True,user_id='476228814d42467c880dce2b1c5c5113',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 16:35:00 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:28:03Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesSnapshotListTestJSON-Volume-552939668',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=2159f1f1-2564-4d84-8110-c1f7d843edb1,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='c4b1a8a0074e4798bd385ee96e8ac8b7',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:32:42Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:32:42Z,use_quota=True,user_id='476228814d42467c880dce2b1c5c5113',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:28:03Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesSnapshotListTestJSON-Volume-552939668',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=2159f1f1-2564-4d84-8110-c1f7d843edb1,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='c4b1a8a0074e4798bd385ee96e8ac8b7',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:32:42Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:32:42Z,use_quota=True,user_id='476228814d42467c880dce2b1c5c5113',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 16:35:00 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 16:35:00 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 16:35:00 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:35:00 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 16:35:00 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:35:00 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 16:35:00 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:35:00 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 16:35:00 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:35:00 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 16:35:00 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:35:00 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 16:35:00 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:35:00 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 16:35:00 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:35:00 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 16:35:00 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:35:00 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:35:00 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 16:35:00 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 16:35:00 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (721e1133-e9db-4d81-8eb1-755d8155c29f) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:35:00 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (721e1133-e9db-4d81-8eb1-755d8155c29f) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:35:00 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (a114f1d5-edd3-4a2c-acac-5cd6c8fc610a) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:35:00 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (a114f1d5-edd3-4a2c-acac-5cd6c8fc610a) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:35:00 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (4140f006-344c-4130-a620-de9a99f38596) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:35:00 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (4140f006-344c-4130-a620-de9a99f38596) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:35:00 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (66f629a8-6266-48fd-8ac5-26a9a6bec901) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:35:00 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Updating volume 2159f1f1-2564-4d84-8110-c1f7d843edb1 with {'scheduled_at': datetime.datetime(2023, 4, 17, 16, 35, 0, 768264), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 16:35:00 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 16:35:00 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Volume 2159f1f1-2564-4d84-8110-c1f7d843edb1: re-scheduling SchedulerAPI.create_volume attempt 3 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 16:35:00 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (5e72c775-bd35-4c0a-8bb6-234f65b4515a) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 16:35:00 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 16:35:00 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:27:59Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesDeleteCascade-Volume-2055115456',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=8b7f1136-3d20-4cf8-a521-6db1d3a0c8a8,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='abbfd52254f546b0bcfe289884d77f49',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:32:42Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:32:42Z,use_quota=True,user_id='6309fcbea41c471184457d276d32db73',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 16:35:00 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:27:59Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesDeleteCascade-Volume-2055115456',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=8b7f1136-3d20-4cf8-a521-6db1d3a0c8a8,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='abbfd52254f546b0bcfe289884d77f49',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:32:42Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:32:42Z,use_quota=True,user_id='6309fcbea41c471184457d276d32db73',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(8b7f1136-3d20-4cf8-a521-6db1d3a0c8a8),volume_id=8b7f1136-3d20-4cf8-a521-6db1d3a0c8a8,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': '8b7f1136-3d20-4cf8-a521-6db1d3a0c8a8', 'volume_name': 'volume-8b7f1136-3d20-4cf8-a521-6db1d3a0c8a8', 'volume_size': 1}} Apr 17 16:35:00 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': '8b7f1136-3d20-4cf8-a521-6db1d3a0c8a8', 'volume': {'id': '8b7f1136-3d20-4cf8-a521-6db1d3a0c8a8', '_name_id': None, 'ec2_id': None, 'user_id': '6309fcbea41c471184457d276d32db73', 'project_id': 'abbfd52254f546b0bcfe289884d77f49', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-VolumesDeleteCascade-Volume-2055115456', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'admin_metadata': {}, 'glance_metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T16:27:58.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-8b7f1136-3d20-4cf8-a521-6db1d3a0c8a8', 'name_id': '8b7f1136-3d20-4cf8-a521-6db1d3a0c8a8', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumesDeleteCascade-Volume-2055115456', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': 'abbfd52254f546b0bcfe289884d77f49', 'qos_specs': None, 'reservations': ['374a7e05-986c-4abf-a2c9-4761a21cff44', 'b5097575-0972-436f-8ef4-38f9b292ab7b', '56c0c623-13ab-497a-bc50-d552acc88144', 'c6eb5369-b08c-4737-8356-45cc9d705e20'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': '6309fcbea41c471184457d276d32db73', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumesDeleteCascade-Volume-2055115456', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': 'abbfd52254f546b0bcfe289884d77f49', 'qos_specs': None, 'reservations': ['374a7e05-986c-4abf-a2c9-4761a21cff44', 'b5097575-0972-436f-8ef4-38f9b292ab7b', '56c0c623-13ab-497a-bc50-d552acc88144', 'c6eb5369-b08c-4737-8356-45cc9d705e20'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': '6309fcbea41c471184457d276d32db73', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': '6309fcbea41c471184457d276d32db73', 'metadata': {}, 'qos_specs': None, 'retry': {'num_attempts': 3, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']}}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(8b7f1136-3d20-4cf8-a521-6db1d3a0c8a8),volume_id=8b7f1136-3d20-4cf8-a521-6db1d3a0c8a8,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:27:59Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesDeleteCascade-Volume-2055115456',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=8b7f1136-3d20-4cf8-a521-6db1d3a0c8a8,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='abbfd52254f546b0bcfe289884d77f49',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:32:42Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:32:42Z,use_quota=True,user_id='6309fcbea41c471184457d276d32db73',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 16:35:00 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:27:59Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesDeleteCascade-Volume-2055115456',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=8b7f1136-3d20-4cf8-a521-6db1d3a0c8a8,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='abbfd52254f546b0bcfe289884d77f49',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:32:42Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:32:42Z,use_quota=True,user_id='6309fcbea41c471184457d276d32db73',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:27:59Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesDeleteCascade-Volume-2055115456',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=8b7f1136-3d20-4cf8-a521-6db1d3a0c8a8,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='abbfd52254f546b0bcfe289884d77f49',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:32:42Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:32:42Z,use_quota=True,user_id='6309fcbea41c471184457d276d32db73',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 16:35:00 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 16:35:00 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 16:35:00 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:35:00 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 16:35:00 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:35:00 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 16:35:00 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:35:00 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 16:35:00 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:35:00 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 16:35:00 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:35:00 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 16:35:00 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:35:00 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 16:35:00 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:35:00 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 16:35:00 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:35:00 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:35:00 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 16:35:00 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 16:35:00 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (5e72c775-bd35-4c0a-8bb6-234f65b4515a) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:35:00 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (5e72c775-bd35-4c0a-8bb6-234f65b4515a) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:35:00 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (0c4a9744-69dd-471e-9e26-85aafa0d3deb) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:35:00 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (0c4a9744-69dd-471e-9e26-85aafa0d3deb) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:35:00 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (31b2a566-87bb-48fe-965d-d7584cdfa324) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:35:00 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (31b2a566-87bb-48fe-965d-d7584cdfa324) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:35:00 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (39ce3927-4fa9-4393-b9eb-4475175819de) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:35:00 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Updating volume 8b7f1136-3d20-4cf8-a521-6db1d3a0c8a8 with {'scheduled_at': datetime.datetime(2023, 4, 17, 16, 35, 0, 821446), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 16:35:00 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Volume 8b7f1136-3d20-4cf8-a521-6db1d3a0c8a8: re-scheduling SchedulerAPI.create_volume attempt 3 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 16:35:00 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Volume 2159f1f1-2564-4d84-8110-c1f7d843edb1: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 16:35:00 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] deleating volume v_2159f1f1-2564-4d84-8110-c1f7d843edb1 {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 16:35:00 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] get snapshots for volume v_2159f1f1-2564-4d84-8110-c1f7d843edb1 {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 16:35:00 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_2159f1f1-2564-4d84-8110-c1f7d843edb1/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:35:00 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_2159f1f1-2564-4d84-8110-c1f7d843edb1/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:35:01 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Volume 8b7f1136-3d20-4cf8-a521-6db1d3a0c8a8: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 16:35:01 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] deleating volume v_8b7f1136-3d20-4cf8-a521-6db1d3a0c8a8 {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 16:35:01 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] get snapshots for volume v_8b7f1136-3d20-4cf8-a521-6db1d3a0c8a8 {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 16:35:01 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_8b7f1136-3d20-4cf8-a521-6db1d3a0c8a8/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:35:01 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_8b7f1136-3d20-4cf8-a521-6db1d3a0c8a8/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:35:03 user cinder-volume[84696]: DEBUG cinder.utils [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.709(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:35:03 user cinder-volume[84696]: DEBUG cinder.utils [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_8b7f1136-3d20-4cf8-a521-6db1d3a0c8a8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:35:03 user cinder-volume[84696]: DEBUG cinder.utils [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.899(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:35:03 user cinder-volume[84696]: DEBUG cinder.utils [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_2159f1f1-2564-4d84-8110-c1f7d843edb1/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:35:06 user cinder-volume[84696]: DEBUG cinder.utils [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.970(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:35:06 user cinder-volume[84696]: DEBUG cinder.utils [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_2159f1f1-2564-4d84-8110-c1f7d843edb1/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:35:06 user cinder-volume[84696]: DEBUG cinder.utils [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 19.379(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:35:06 user cinder-volume[84696]: DEBUG cinder.utils [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:35:06 user cinder-volume[84696]: DEBUG cinder.utils [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.784(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:35:06 user cinder-volume[84696]: DEBUG cinder.utils [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_8b7f1136-3d20-4cf8-a521-6db1d3a0c8a8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:35:06 user cinder-volume[84696]: DEBUG cinder.utils [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 139.060(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:35:06 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:35:06 user cinder-volume[84696]: INFO cinder.message.api [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Creating message record for request_id = req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d Apr 17 16:35:06 user cinder-volume[84696]: DEBUG cinder.utils [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 139.289(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:35:06 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_f6c0e341-6f00-4e32-bd07-31f954c1ad8a/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:35:06 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (c5686eb2-064c-40f9-8c44-e9ea3b76a86b) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 16:35:06 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (ef826f8e-a283-47ce-92c4-d3525c4caa4a) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:35:06 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (ef826f8e-a283-47ce-92c4-d3525c4caa4a) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:35:06 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Flow 'volume_create_manager' (a8598034-edf8-495f-88e2-8b4c9e8e4fcf) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 16:35:06 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 16:35:06 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 16:35:06 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 16:35:06 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 16:35:06 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 16:35:06 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 16:35:06 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 16:35:06 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 16:35:06 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 16:35:06 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 16:35:06 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 16:35:06 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 16:35:06 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 16:35:06 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 16:35:06 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 16:35:06 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 16:35:06 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 16:35:06 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 16:35:06 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 16:35:06 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 16:35:06 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 16:35:06 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 16:35:06 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 16:35:06 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 16:35:06 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:35:06 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 16:35:06 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:35:06 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 16:35:06 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:35:06 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 16:35:06 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:35:06 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 16:35:06 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:35:06 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 16:35:06 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:35:06 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 16:35:06 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:35:06 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 16:35:06 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:35:06 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:35:06 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 16:35:06 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 16:35:06 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 16:35:06 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (12b6b127-c2dd-448d-9d71-e4010e676eb2) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 16:35:06 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 16:35:06 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:28:10Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumeQuotasAdminTestJSON-Volume-332841798',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=f6c0e341-6f00-4e32-bd07-31f954c1ad8a,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='6e4c2ec9177349a08ce53e07b24ad79c',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:32:48Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:32:48Z,use_quota=True,user_id='ab5017da30ac4d47ba665ff135deb309',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 16:35:06 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:28:10Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumeQuotasAdminTestJSON-Volume-332841798',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=f6c0e341-6f00-4e32-bd07-31f954c1ad8a,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='6e4c2ec9177349a08ce53e07b24ad79c',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:32:48Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:32:48Z,use_quota=True,user_id='ab5017da30ac4d47ba665ff135deb309',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(f6c0e341-6f00-4e32-bd07-31f954c1ad8a),volume_id=f6c0e341-6f00-4e32-bd07-31f954c1ad8a,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': 'f6c0e341-6f00-4e32-bd07-31f954c1ad8a', 'volume_name': 'volume-f6c0e341-6f00-4e32-bd07-31f954c1ad8a', 'volume_size': 1}} Apr 17 16:35:06 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': 'f6c0e341-6f00-4e32-bd07-31f954c1ad8a', 'volume': {'id': 'f6c0e341-6f00-4e32-bd07-31f954c1ad8a', '_name_id': None, 'ec2_id': None, 'user_id': 'ab5017da30ac4d47ba665ff135deb309', 'project_id': '6e4c2ec9177349a08ce53e07b24ad79c', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-VolumeQuotasAdminTestJSON-Volume-332841798', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'admin_metadata': {}, 'glance_metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T16:28:09.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-f6c0e341-6f00-4e32-bd07-31f954c1ad8a', 'name_id': 'f6c0e341-6f00-4e32-bd07-31f954c1ad8a', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumeQuotasAdminTestJSON-Volume-332841798', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '6e4c2ec9177349a08ce53e07b24ad79c', 'qos_specs': None, 'reservations': ['f21078bb-20eb-4029-93a1-5b4cb253f25d', 'bbd80d77-344a-4970-b1bb-2892c08c1c72', 'a38f5177-62fc-4eb7-b072-f15a54689ed0', 'b5e5e1fe-ca2e-48fa-b5e1-b6f6d98fe349'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'ab5017da30ac4d47ba665ff135deb309', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumeQuotasAdminTestJSON-Volume-332841798', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '6e4c2ec9177349a08ce53e07b24ad79c', 'qos_specs': None, 'reservations': ['f21078bb-20eb-4029-93a1-5b4cb253f25d', 'bbd80d77-344a-4970-b1bb-2892c08c1c72', 'a38f5177-62fc-4eb7-b072-f15a54689ed0', 'b5e5e1fe-ca2e-48fa-b5e1-b6f6d98fe349'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'ab5017da30ac4d47ba665ff135deb309', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': 'ab5017da30ac4d47ba665ff135deb309', 'metadata': {}, 'qos_specs': None, 'retry': {'num_attempts': 3, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']}}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(f6c0e341-6f00-4e32-bd07-31f954c1ad8a),volume_id=f6c0e341-6f00-4e32-bd07-31f954c1ad8a,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:28:10Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumeQuotasAdminTestJSON-Volume-332841798',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=f6c0e341-6f00-4e32-bd07-31f954c1ad8a,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='6e4c2ec9177349a08ce53e07b24ad79c',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:32:48Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:32:48Z,use_quota=True,user_id='ab5017da30ac4d47ba665ff135deb309',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 16:35:06 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:28:10Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumeQuotasAdminTestJSON-Volume-332841798',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=f6c0e341-6f00-4e32-bd07-31f954c1ad8a,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='6e4c2ec9177349a08ce53e07b24ad79c',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:32:48Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:32:48Z,use_quota=True,user_id='ab5017da30ac4d47ba665ff135deb309',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:28:10Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumeQuotasAdminTestJSON-Volume-332841798',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=f6c0e341-6f00-4e32-bd07-31f954c1ad8a,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='6e4c2ec9177349a08ce53e07b24ad79c',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:32:48Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:32:48Z,use_quota=True,user_id='ab5017da30ac4d47ba665ff135deb309',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 16:35:06 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 16:35:06 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 16:35:06 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:35:06 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 16:35:06 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:35:06 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 16:35:06 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:35:06 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 16:35:06 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:35:06 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 16:35:06 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:35:06 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 16:35:06 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:35:06 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 16:35:06 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:35:06 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 16:35:06 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:35:06 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:35:06 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 16:35:06 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 16:35:06 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (12b6b127-c2dd-448d-9d71-e4010e676eb2) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:35:06 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (12b6b127-c2dd-448d-9d71-e4010e676eb2) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:35:06 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (228d2dc0-bc93-44c5-9838-57fce29d2f99) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:35:06 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (228d2dc0-bc93-44c5-9838-57fce29d2f99) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:35:06 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (f2fb9d7b-2629-40e2-863b-82c9f30bb379) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:35:06 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (f2fb9d7b-2629-40e2-863b-82c9f30bb379) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:35:06 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (854e74aa-081b-4a34-9787-2c0edccf9773) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:35:06 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Updating volume f6c0e341-6f00-4e32-bd07-31f954c1ad8a with {'scheduled_at': datetime.datetime(2023, 4, 17, 16, 35, 6, 896821), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 16:35:06 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Volume f6c0e341-6f00-4e32-bd07-31f954c1ad8a: re-scheduling SchedulerAPI.create_volume attempt 3 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 16:35:06 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Volume f6c0e341-6f00-4e32-bd07-31f954c1ad8a: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 16:35:06 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] deleating volume v_f6c0e341-6f00-4e32-bd07-31f954c1ad8a {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 16:35:06 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] get snapshots for volume v_f6c0e341-6f00-4e32-bd07-31f954c1ad8a {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 16:35:06 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_f6c0e341-6f00-4e32-bd07-31f954c1ad8a/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:35:06 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_f6c0e341-6f00-4e32-bd07-31f954c1ad8a/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:35:09 user cinder-volume[84696]: DEBUG cinder.utils [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.963(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:35:09 user cinder-volume[84696]: DEBUG cinder.utils [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_f6c0e341-6f00-4e32-bd07-31f954c1ad8a/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:35:12 user cinder-volume[84696]: DEBUG cinder.utils [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 11.926(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:35:12 user cinder-volume[84696]: DEBUG cinder.utils [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_8b7f1136-3d20-4cf8-a521-6db1d3a0c8a8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:35:12 user cinder-volume[84696]: DEBUG cinder.utils [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.117(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:35:12 user cinder-volume[84696]: DEBUG cinder.utils [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_2159f1f1-2564-4d84-8110-c1f7d843edb1/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:35:12 user cinder-volume[84696]: DEBUG cinder.utils [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.039(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:35:12 user cinder-volume[84696]: DEBUG cinder.utils [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_f6c0e341-6f00-4e32-bd07-31f954c1ad8a/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:35:14 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=12,cinder:UPDATE=5,cinder:INSERT=2 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:35:19 user cinder-volume[84696]: DEBUG cinder.utils [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.499(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:35:19 user cinder-volume[84696]: DEBUG cinder.utils [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_f6c0e341-6f00-4e32-bd07-31f954c1ad8a/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:35:22 user cinder-volume[84696]: DEBUG cinder.utils [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.461(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:35:22 user cinder-volume[84696]: DEBUG cinder.utils [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_8b7f1136-3d20-4cf8-a521-6db1d3a0c8a8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:35:22 user cinder-volume[84696]: DEBUG cinder.utils [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.652(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:35:22 user cinder-volume[84696]: DEBUG cinder.utils [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_2159f1f1-2564-4d84-8110-c1f7d843edb1/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:35:25 user cinder-volume[84696]: DEBUG cinder.utils [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 38.130(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:35:25 user cinder-volume[84696]: DEBUG cinder.utils [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:35:28 user cinder-volume[84696]: DEBUG cinder.utils [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.715(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:35:28 user cinder-volume[84696]: DEBUG cinder.utils [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_f6c0e341-6f00-4e32-bd07-31f954c1ad8a/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:35:34 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=13,cinder:INSERT=4,cinder:UPDATE=5 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:35:41 user cinder-volume[84696]: DEBUG cinder.utils [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 40.658(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:35:41 user cinder-volume[84696]: DEBUG cinder.utils [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_2159f1f1-2564-4d84-8110-c1f7d843edb1/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:35:41 user cinder-volume[84696]: DEBUG cinder.utils [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 40.471(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:35:41 user cinder-volume[84696]: DEBUG cinder.utils [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_8b7f1136-3d20-4cf8-a521-6db1d3a0c8a8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:35:47 user cinder-volume[84696]: DEBUG cinder.utils [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 40.723(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:35:47 user cinder-volume[84696]: DEBUG cinder.utils [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_f6c0e341-6f00-4e32-bd07-31f954c1ad8a/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:35:50 user cinder-volume[84696]: DEBUG cinder.utils [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 138.836(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:35:50 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:35:50 user cinder-volume[84696]: INFO cinder.message.api [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Creating message record for request_id = req-208fb94c-86af-4459-a279-1c8be9781cb3 Apr 17 16:35:50 user cinder-volume[84696]: DEBUG cinder.utils [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 139.025(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:35:50 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_bf85ce23-1db3-4881-bd7a-b20b520718b8/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:35:50 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (e593ecf2-32cd-45c5-818c-0bdc7f96fd98) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 16:35:50 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (86c21eaf-9bad-46cc-b34e-900c1110fc18) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:35:50 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (86c21eaf-9bad-46cc-b34e-900c1110fc18) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:35:50 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Flow 'volume_create_manager' (9f70d528-7342-4973-8e87-91c98296c96e) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 16:35:50 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 16:35:50 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 16:35:50 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 16:35:50 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 16:35:50 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 16:35:50 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 16:35:50 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 16:35:50 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 16:35:50 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 16:35:50 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 16:35:50 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 16:35:50 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 16:35:50 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 16:35:50 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 16:35:50 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 16:35:50 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 16:35:50 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 16:35:50 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 16:35:50 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 16:35:50 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 16:35:50 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 16:35:50 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 16:35:50 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 16:35:50 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 16:35:50 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:35:50 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 16:35:50 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:35:50 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 16:35:50 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:35:50 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 16:35:50 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:35:50 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 16:35:50 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:35:50 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 16:35:50 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:35:50 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 16:35:50 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:35:50 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 16:35:50 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:35:50 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:35:50 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 16:35:50 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 16:35:50 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 16:35:50 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=5,cinder:UPDATE=1 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:35:50 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (0fa03cdc-ba54-4ba5-a200-38d38cef5097) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 16:35:50 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 16:35:50 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:31:18Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-CreateVolumesFromSnapshotTest-Volume-2022997390',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=bf85ce23-1db3-4881-bd7a-b20b520718b8,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='93bbb103c036439da2861c7be15c3e7b',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:33:32Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:33:32Z,use_quota=True,user_id='ae7cdf32438c44b6b3f428e9e0142af1',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 16:35:50 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:31:18Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-CreateVolumesFromSnapshotTest-Volume-2022997390',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=bf85ce23-1db3-4881-bd7a-b20b520718b8,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='93bbb103c036439da2861c7be15c3e7b',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:33:32Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:33:32Z,use_quota=True,user_id='ae7cdf32438c44b6b3f428e9e0142af1',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(bf85ce23-1db3-4881-bd7a-b20b520718b8),volume_id=bf85ce23-1db3-4881-bd7a-b20b520718b8,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': 'bf85ce23-1db3-4881-bd7a-b20b520718b8', 'volume_name': 'volume-bf85ce23-1db3-4881-bd7a-b20b520718b8', 'volume_size': 1}} Apr 17 16:35:50 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': 'bf85ce23-1db3-4881-bd7a-b20b520718b8', 'volume': {'id': 'bf85ce23-1db3-4881-bd7a-b20b520718b8', '_name_id': None, 'ec2_id': None, 'user_id': 'ae7cdf32438c44b6b3f428e9e0142af1', 'project_id': '93bbb103c036439da2861c7be15c3e7b', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-CreateVolumesFromSnapshotTest-Volume-2022997390', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'admin_metadata': {}, 'glance_metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T16:31:17.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-bf85ce23-1db3-4881-bd7a-b20b520718b8', 'name_id': 'bf85ce23-1db3-4881-bd7a-b20b520718b8', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-CreateVolumesFromSnapshotTest-Volume-2022997390', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '93bbb103c036439da2861c7be15c3e7b', 'qos_specs': None, 'reservations': ['b69f8abb-8ed7-4741-8211-ad267bc6119b', '37f3f14f-b707-4dd5-868b-e177f7131d19', 'c87fa212-ff4d-42ec-8c41-d50482477074', '51763846-c4e0-4572-a1af-a15208766488'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'ae7cdf32438c44b6b3f428e9e0142af1', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-CreateVolumesFromSnapshotTest-Volume-2022997390', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '93bbb103c036439da2861c7be15c3e7b', 'qos_specs': None, 'reservations': ['b69f8abb-8ed7-4741-8211-ad267bc6119b', '37f3f14f-b707-4dd5-868b-e177f7131d19', 'c87fa212-ff4d-42ec-8c41-d50482477074', '51763846-c4e0-4572-a1af-a15208766488'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'ae7cdf32438c44b6b3f428e9e0142af1', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': 'ae7cdf32438c44b6b3f428e9e0142af1', 'metadata': {}, 'qos_specs': None, 'retry': {'num_attempts': 2, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']}}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(bf85ce23-1db3-4881-bd7a-b20b520718b8),volume_id=bf85ce23-1db3-4881-bd7a-b20b520718b8,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:31:18Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-CreateVolumesFromSnapshotTest-Volume-2022997390',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=bf85ce23-1db3-4881-bd7a-b20b520718b8,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='93bbb103c036439da2861c7be15c3e7b',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:33:32Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:33:32Z,use_quota=True,user_id='ae7cdf32438c44b6b3f428e9e0142af1',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 16:35:50 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:31:18Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-CreateVolumesFromSnapshotTest-Volume-2022997390',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=bf85ce23-1db3-4881-bd7a-b20b520718b8,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='93bbb103c036439da2861c7be15c3e7b',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:33:32Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:33:32Z,use_quota=True,user_id='ae7cdf32438c44b6b3f428e9e0142af1',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:31:18Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-CreateVolumesFromSnapshotTest-Volume-2022997390',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=bf85ce23-1db3-4881-bd7a-b20b520718b8,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='93bbb103c036439da2861c7be15c3e7b',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:33:32Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:33:32Z,use_quota=True,user_id='ae7cdf32438c44b6b3f428e9e0142af1',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 16:35:50 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 16:35:50 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 16:35:50 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:35:50 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 16:35:50 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:35:50 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 16:35:50 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:35:50 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 16:35:50 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:35:50 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 16:35:50 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:35:50 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 16:35:50 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:35:50 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 16:35:50 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:35:50 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 16:35:50 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:35:50 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:35:50 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 16:35:50 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 16:35:50 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (0fa03cdc-ba54-4ba5-a200-38d38cef5097) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:35:50 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (0fa03cdc-ba54-4ba5-a200-38d38cef5097) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:35:50 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (f33a6fe8-09c9-413d-a0eb-39b9da1708c9) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:35:50 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (f33a6fe8-09c9-413d-a0eb-39b9da1708c9) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:35:50 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (c4e52081-a823-46bf-83f9-4909fef6584e) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:35:50 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (c4e52081-a823-46bf-83f9-4909fef6584e) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:35:50 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (d75b44e1-b13c-467d-a901-1faee8a72f76) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:35:50 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Updating volume bf85ce23-1db3-4881-bd7a-b20b520718b8 with {'scheduled_at': datetime.datetime(2023, 4, 17, 16, 35, 50, 800927), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 16:35:50 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Volume bf85ce23-1db3-4881-bd7a-b20b520718b8: re-scheduling SchedulerAPI.create_volume attempt 2 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 16:35:50 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Volume bf85ce23-1db3-4881-bd7a-b20b520718b8: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 16:35:50 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] deleating volume v_bf85ce23-1db3-4881-bd7a-b20b520718b8 {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 16:35:50 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] get snapshots for volume v_bf85ce23-1db3-4881-bd7a-b20b520718b8 {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 16:35:50 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_bf85ce23-1db3-4881-bd7a-b20b520718b8/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:35:50 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_bf85ce23-1db3-4881-bd7a-b20b520718b8/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:35:50 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Volume reschedule parameters: True retry: {'num_attempts': 3, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 16:35:50 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Flow 'volume_create_manager' (096bcc29-7419-4518-b3fd-22ecaaa1421c) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 16:35:50 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (333215a2-6f24-4dd6-ba6b-cb4928fc7188) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:35:50 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (333215a2-6f24-4dd6-ba6b-cb4928fc7188) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:31:18Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-CreateVolumesFromSnapshotTest-Volume-2022997390',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=bf85ce23-1db3-4881-bd7a-b20b520718b8,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='93bbb103c036439da2861c7be15c3e7b',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:35:51Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:35:51Z,use_quota=True,user_id='ae7cdf32438c44b6b3f428e9e0142af1',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:35:50 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (44f4fcbb-0a2d-4361-9c98-290e952f34ed) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:35:50 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (44f4fcbb-0a2d-4361-9c98-290e952f34ed) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:35:50 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (01fed539-9314-424a-9782-b8abccdb4d8a) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:35:50 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (01fed539-9314-424a-9782-b8abccdb4d8a) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': 'bf85ce23-1db3-4881-bd7a-b20b520718b8', 'volume_name': 'volume-bf85ce23-1db3-4881-bd7a-b20b520718b8', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:35:50 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (5328bac4-1da4-4960-906c-e030fc77097b) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:35:51 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (5328bac4-1da4-4960-906c-e030fc77097b) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:35:51 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (bafbf2f1-5d48-4648-a84f-5bf00e566082) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:35:51 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Volume bf85ce23-1db3-4881-bd7a-b20b520718b8: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-bf85ce23-1db3-4881-bd7a-b20b520718b8', 'volume_size': 1} Apr 17 16:35:51 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] creating volume v_bf85ce23-1db3-4881-bd7a-b20b520718b8. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 16:35:51 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] create volume {'name': 'v_bf85ce23-1db3-4881-bd7a-b20b520718b8', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 16:35:51 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:35:51 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:35:53 user cinder-volume[84696]: DEBUG cinder.utils [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.958(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:35:53 user cinder-volume[84696]: DEBUG cinder.utils [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_bf85ce23-1db3-4881-bd7a-b20b520718b8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:35:53 user cinder-volume[84696]: DEBUG cinder.utils [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.779(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:35:53 user cinder-volume[84696]: DEBUG cinder.utils [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:35:56 user cinder-volume[84696]: DEBUG cinder.utils [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.030(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:35:56 user cinder-volume[84696]: DEBUG cinder.utils [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_bf85ce23-1db3-4881-bd7a-b20b520718b8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:35:56 user cinder-volume[84696]: DEBUG cinder.utils [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.851(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:35:56 user cinder-volume[84696]: DEBUG cinder.utils [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:35:59 user cinder-volume[84696]: DEBUG cinder.utils [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 72.498(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:35:59 user cinder-volume[84696]: DEBUG cinder.utils [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:35:59 user cinder-volume[84696]: DEBUG cinder.utils [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 138.787(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:35:59 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:35:59 user cinder-volume[84696]: INFO cinder.message.api [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Creating message record for request_id = req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 Apr 17 16:35:59 user cinder-volume[84696]: DEBUG cinder.utils [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 139.016(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:35:59 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_7384f51d-c3a2-4eea-a968-ec043946989f/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:35:59 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (6d2f64ff-fe0b-4f90-b3bd-5481f65a4834) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 16:35:59 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (3702dcb6-db1c-4ced-a2fa-3f0bb0d53193) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:35:59 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (3702dcb6-db1c-4ced-a2fa-3f0bb0d53193) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:35:59 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Flow 'volume_create_manager' (ce92c14e-8b17-4992-905a-0fe1f43b987c) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 16:35:59 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 16:35:59 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 16:35:59 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 16:35:59 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 16:35:59 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 16:35:59 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 16:35:59 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 16:35:59 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 16:35:59 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 16:35:59 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 16:35:59 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 16:35:59 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 16:35:59 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 16:35:59 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 16:35:59 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 16:35:59 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 16:35:59 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 16:35:59 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 16:35:59 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 16:35:59 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 16:35:59 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 16:35:59 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 16:35:59 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 16:35:59 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 16:35:59 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:35:59 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 16:35:59 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:35:59 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 16:35:59 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:35:59 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 16:35:59 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:35:59 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 16:35:59 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:35:59 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 16:35:59 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:35:59 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 16:35:59 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:35:59 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 16:35:59 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:35:59 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:35:59 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 16:35:59 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 16:35:59 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 16:35:59 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (381d9c70-eeb0-489c-a13e-dc36e3289945) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 16:35:59 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 16:35:59 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:31:21Z,deleted=False,deleted_at=None,display_description=None,display_name='CinderUnicodeTest塵㼗‽',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=7384f51d-c3a2-4eea-a968-ec043946989f,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='9ccd4e98adcb4fb18f651ead3736305a',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:33:41Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:33:41Z,use_quota=True,user_id='d3c58fb72fb443e19315cc15a76767fe',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 16:35:59 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:31:21Z,deleted=False,deleted_at=None,display_description=None,display_name='CinderUnicodeTest塵㼗‽',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=7384f51d-c3a2-4eea-a968-ec043946989f,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='9ccd4e98adcb4fb18f651ead3736305a',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:33:41Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:33:41Z,use_quota=True,user_id='d3c58fb72fb443e19315cc15a76767fe',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(7384f51d-c3a2-4eea-a968-ec043946989f),volume_id=7384f51d-c3a2-4eea-a968-ec043946989f,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': '7384f51d-c3a2-4eea-a968-ec043946989f', 'volume_name': 'volume-7384f51d-c3a2-4eea-a968-ec043946989f', 'volume_size': 1}} Apr 17 16:35:59 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': '7384f51d-c3a2-4eea-a968-ec043946989f', 'volume': {'id': '7384f51d-c3a2-4eea-a968-ec043946989f', '_name_id': None, 'ec2_id': None, 'user_id': 'd3c58fb72fb443e19315cc15a76767fe', 'project_id': '9ccd4e98adcb4fb18f651ead3736305a', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'CinderUnicodeTest塵㼗‽', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'admin_metadata': {}, 'glance_metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T16:31:21.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-7384f51d-c3a2-4eea-a968-ec043946989f', 'name_id': '7384f51d-c3a2-4eea-a968-ec043946989f', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'CinderUnicodeTest塵㼗‽', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '9ccd4e98adcb4fb18f651ead3736305a', 'qos_specs': None, 'reservations': ['e840cac3-58bf-4622-96ed-8d715415019e', '88e95250-2b05-45ff-b59d-697235ec7436', '29eb19dc-9f45-4e12-a7b9-01875910c4ae', 'be37e533-c8a6-464f-942b-ce65599da74b'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'd3c58fb72fb443e19315cc15a76767fe', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'CinderUnicodeTest塵㼗‽', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '9ccd4e98adcb4fb18f651ead3736305a', 'qos_specs': None, 'reservations': ['e840cac3-58bf-4622-96ed-8d715415019e', '88e95250-2b05-45ff-b59d-697235ec7436', '29eb19dc-9f45-4e12-a7b9-01875910c4ae', 'be37e533-c8a6-464f-942b-ce65599da74b'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'd3c58fb72fb443e19315cc15a76767fe', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': 'd3c58fb72fb443e19315cc15a76767fe', 'metadata': {}, 'qos_specs': None, 'retry': {'num_attempts': 2, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']}}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(7384f51d-c3a2-4eea-a968-ec043946989f),volume_id=7384f51d-c3a2-4eea-a968-ec043946989f,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:31:21Z,deleted=False,deleted_at=None,display_description=None,display_name='CinderUnicodeTest塵㼗‽',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=7384f51d-c3a2-4eea-a968-ec043946989f,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='9ccd4e98adcb4fb18f651ead3736305a',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:33:41Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:33:41Z,use_quota=True,user_id='d3c58fb72fb443e19315cc15a76767fe',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 16:35:59 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:31:21Z,deleted=False,deleted_at=None,display_description=None,display_name='CinderUnicodeTest塵㼗‽',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=7384f51d-c3a2-4eea-a968-ec043946989f,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='9ccd4e98adcb4fb18f651ead3736305a',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:33:41Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:33:41Z,use_quota=True,user_id='d3c58fb72fb443e19315cc15a76767fe',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:31:21Z,deleted=False,deleted_at=None,display_description=None,display_name='CinderUnicodeTest塵㼗‽',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=7384f51d-c3a2-4eea-a968-ec043946989f,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='9ccd4e98adcb4fb18f651ead3736305a',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:33:41Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:33:41Z,use_quota=True,user_id='d3c58fb72fb443e19315cc15a76767fe',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 16:35:59 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 16:35:59 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 16:35:59 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:35:59 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 16:35:59 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:35:59 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 16:35:59 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:35:59 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 16:35:59 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:35:59 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 16:35:59 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:35:59 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 16:35:59 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:35:59 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 16:35:59 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:35:59 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 16:35:59 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:35:59 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:35:59 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 16:35:59 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 16:35:59 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (381d9c70-eeb0-489c-a13e-dc36e3289945) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:35:59 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (381d9c70-eeb0-489c-a13e-dc36e3289945) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:35:59 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (1b532189-727b-4106-840c-e32a634bef49) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:35:59 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (1b532189-727b-4106-840c-e32a634bef49) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:36:00 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (5e52c3ca-bbe7-4158-8f40-a24d08d03c17) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:36:00 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (5e52c3ca-bbe7-4158-8f40-a24d08d03c17) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:36:00 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (9b5a87a7-4b89-42c4-8980-3b6a59e23bc8) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:36:00 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Updating volume 7384f51d-c3a2-4eea-a968-ec043946989f with {'scheduled_at': datetime.datetime(2023, 4, 17, 16, 36, 0, 8852), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 16:36:00 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Volume 7384f51d-c3a2-4eea-a968-ec043946989f: re-scheduling SchedulerAPI.create_volume attempt 2 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 16:36:00 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Volume 7384f51d-c3a2-4eea-a968-ec043946989f: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 16:36:00 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] deleating volume v_7384f51d-c3a2-4eea-a968-ec043946989f {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 16:36:00 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] get snapshots for volume v_7384f51d-c3a2-4eea-a968-ec043946989f {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 16:36:00 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_7384f51d-c3a2-4eea-a968-ec043946989f/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:36:00 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_7384f51d-c3a2-4eea-a968-ec043946989f/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:36:00 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Volume reschedule parameters: True retry: {'num_attempts': 3, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 16:36:00 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Flow 'volume_create_manager' (47754890-05fe-49a0-88aa-1d8556959dce) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 16:36:00 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (151796e9-3a94-4863-81cf-e6d865b78c19) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:36:00 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (151796e9-3a94-4863-81cf-e6d865b78c19) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:31:21Z,deleted=False,deleted_at=None,display_description=None,display_name='CinderUnicodeTest塵㼗‽',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=7384f51d-c3a2-4eea-a968-ec043946989f,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='9ccd4e98adcb4fb18f651ead3736305a',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:36:00Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:36:00Z,use_quota=True,user_id='d3c58fb72fb443e19315cc15a76767fe',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:36:00 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (d7f3d394-9859-4e12-aa4a-64ce16c48c98) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:36:00 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (d7f3d394-9859-4e12-aa4a-64ce16c48c98) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:36:00 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (8db470a3-373c-4eb4-b233-49fb5e034f6b) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:36:00 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (8db470a3-373c-4eb4-b233-49fb5e034f6b) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': '7384f51d-c3a2-4eea-a968-ec043946989f', 'volume_name': 'volume-7384f51d-c3a2-4eea-a968-ec043946989f', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:36:00 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (c3e17118-1398-4051-af0a-cb5ea0a53528) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:36:00 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (c3e17118-1398-4051-af0a-cb5ea0a53528) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:36:00 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (a930e0c2-aefa-4592-9dac-015c5f62c62a) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:36:00 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Volume 7384f51d-c3a2-4eea-a968-ec043946989f: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-7384f51d-c3a2-4eea-a968-ec043946989f', 'volume_size': 1} Apr 17 16:36:00 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] creating volume v_7384f51d-c3a2-4eea-a968-ec043946989f. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 16:36:00 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] create volume {'name': 'v_7384f51d-c3a2-4eea-a968-ec043946989f', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 16:36:00 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:36:00 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:36:03 user cinder-volume[84696]: DEBUG cinder.utils [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.969(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:36:03 user cinder-volume[84696]: DEBUG cinder.utils [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_7384f51d-c3a2-4eea-a968-ec043946989f/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:36:03 user cinder-volume[84696]: DEBUG cinder.utils [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.801(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:36:03 user cinder-volume[84696]: DEBUG cinder.utils [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:36:03 user cinder-volume[84696]: DEBUG cinder.utils [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 11.997(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:36:03 user cinder-volume[84696]: DEBUG cinder.utils [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:36:03 user cinder-volume[84696]: DEBUG cinder.utils [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 138.984(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:36:03 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_480a94a0-d1c9-46c0-bf9f-c42f907b4b82/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:36:03 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (8e29d783-ca28-40a8-aa00-4942cc3231e3) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 16:36:03 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (b34b2727-7fbf-4685-8368-a6ec3373855b) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:36:03 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (b34b2727-7fbf-4685-8368-a6ec3373855b) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:36:03 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Flow 'volume_create_manager' (c616e438-ecd0-42ad-8cd3-155c73f9aa09) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 16:36:03 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 16:36:03 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 16:36:03 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 16:36:03 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 16:36:03 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 16:36:03 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 16:36:03 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 16:36:03 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 16:36:03 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 16:36:03 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 16:36:03 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 16:36:03 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 16:36:03 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 16:36:03 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 16:36:03 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 16:36:03 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 16:36:03 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 16:36:03 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 16:36:03 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 16:36:03 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 16:36:03 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 16:36:03 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 16:36:03 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 16:36:03 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 16:36:03 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:36:03 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 16:36:03 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:36:03 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 16:36:03 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:36:03 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 16:36:03 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:36:03 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 16:36:03 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:36:03 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 16:36:03 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:36:03 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 16:36:03 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:36:03 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 16:36:03 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:36:03 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:36:03 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 16:36:03 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 16:36:03 user cinder-volume[84696]: DEBUG cinder.utils [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 138.836(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:36:03 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:36:03 user cinder-volume[84696]: INFO cinder.message.api [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Creating message record for request_id = req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b Apr 17 16:36:03 user cinder-volume[84696]: DEBUG cinder.utils [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.204(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:36:03 user cinder-volume[84696]: DEBUG cinder.utils [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_bf85ce23-1db3-4881-bd7a-b20b520718b8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:36:03 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 16:36:03 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (367ce42b-db7b-40b8-9c29-a43986a55b2d) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 16:36:03 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 16:36:03 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:31:26Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumeQuotasAdminTestJSON-Volume-2080868017',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=480a94a0-d1c9-46c0-bf9f-c42f907b4b82,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='6e4c2ec9177349a08ce53e07b24ad79c',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:33:44Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:33:44Z,use_quota=True,user_id='ab5017da30ac4d47ba665ff135deb309',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 16:36:03 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:31:26Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumeQuotasAdminTestJSON-Volume-2080868017',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=480a94a0-d1c9-46c0-bf9f-c42f907b4b82,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='6e4c2ec9177349a08ce53e07b24ad79c',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:33:44Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:33:44Z,use_quota=True,user_id='ab5017da30ac4d47ba665ff135deb309',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(480a94a0-d1c9-46c0-bf9f-c42f907b4b82),volume_id=480a94a0-d1c9-46c0-bf9f-c42f907b4b82,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': '480a94a0-d1c9-46c0-bf9f-c42f907b4b82', 'volume_name': 'volume-480a94a0-d1c9-46c0-bf9f-c42f907b4b82', 'volume_size': 1}} Apr 17 16:36:03 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': '480a94a0-d1c9-46c0-bf9f-c42f907b4b82', 'volume': {'id': '480a94a0-d1c9-46c0-bf9f-c42f907b4b82', '_name_id': None, 'ec2_id': None, 'user_id': 'ab5017da30ac4d47ba665ff135deb309', 'project_id': '6e4c2ec9177349a08ce53e07b24ad79c', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-VolumeQuotasAdminTestJSON-Volume-2080868017', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'admin_metadata': {}, 'glance_metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T16:31:25.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-480a94a0-d1c9-46c0-bf9f-c42f907b4b82', 'name_id': '480a94a0-d1c9-46c0-bf9f-c42f907b4b82', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumeQuotasAdminTestJSON-Volume-2080868017', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '6e4c2ec9177349a08ce53e07b24ad79c', 'qos_specs': None, 'reservations': ['00e83981-2b5c-4129-be4e-d251babb9099', '4e65b249-f315-4ce2-9106-e50bca02bf14', '166cbb51-86ba-4d39-9450-64b18ab85006', '47198bf3-67c0-44fb-9cca-8e33c9ae31c6'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'ab5017da30ac4d47ba665ff135deb309', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumeQuotasAdminTestJSON-Volume-2080868017', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '6e4c2ec9177349a08ce53e07b24ad79c', 'qos_specs': None, 'reservations': ['00e83981-2b5c-4129-be4e-d251babb9099', '4e65b249-f315-4ce2-9106-e50bca02bf14', '166cbb51-86ba-4d39-9450-64b18ab85006', '47198bf3-67c0-44fb-9cca-8e33c9ae31c6'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'ab5017da30ac4d47ba665ff135deb309', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': 'ab5017da30ac4d47ba665ff135deb309', 'metadata': {}, 'qos_specs': None, 'retry': {'num_attempts': 2, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']}}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(480a94a0-d1c9-46c0-bf9f-c42f907b4b82),volume_id=480a94a0-d1c9-46c0-bf9f-c42f907b4b82,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:31:26Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumeQuotasAdminTestJSON-Volume-2080868017',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=480a94a0-d1c9-46c0-bf9f-c42f907b4b82,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='6e4c2ec9177349a08ce53e07b24ad79c',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:33:44Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:33:44Z,use_quota=True,user_id='ab5017da30ac4d47ba665ff135deb309',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 16:36:03 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:31:26Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumeQuotasAdminTestJSON-Volume-2080868017',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=480a94a0-d1c9-46c0-bf9f-c42f907b4b82,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='6e4c2ec9177349a08ce53e07b24ad79c',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:33:44Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:33:44Z,use_quota=True,user_id='ab5017da30ac4d47ba665ff135deb309',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:31:26Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumeQuotasAdminTestJSON-Volume-2080868017',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=480a94a0-d1c9-46c0-bf9f-c42f907b4b82,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='6e4c2ec9177349a08ce53e07b24ad79c',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:33:44Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:33:44Z,use_quota=True,user_id='ab5017da30ac4d47ba665ff135deb309',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 16:36:03 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 16:36:03 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 16:36:03 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:36:03 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 16:36:03 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:36:03 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 16:36:03 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:36:03 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 16:36:03 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:36:03 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 16:36:03 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:36:03 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 16:36:03 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:36:03 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 16:36:03 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:36:03 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 16:36:03 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:36:03 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:36:03 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 16:36:03 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 16:36:03 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (367ce42b-db7b-40b8-9c29-a43986a55b2d) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:36:03 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (367ce42b-db7b-40b8-9c29-a43986a55b2d) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:36:03 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (7b27e4ea-25e7-48a3-8aeb-408c54c046ed) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:36:03 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (7b27e4ea-25e7-48a3-8aeb-408c54c046ed) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:36:03 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (16ae2e5e-0eab-4600-81d8-ae7d86c5d940) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:36:03 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (16ae2e5e-0eab-4600-81d8-ae7d86c5d940) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:36:03 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (f7be13b7-4f59-4082-8999-dd5515bc049f) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:36:03 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Updating volume 480a94a0-d1c9-46c0-bf9f-c42f907b4b82 with {'scheduled_at': datetime.datetime(2023, 4, 17, 16, 36, 3, 91133), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 16:36:03 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Volume 480a94a0-d1c9-46c0-bf9f-c42f907b4b82: re-scheduling SchedulerAPI.create_volume attempt 2 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 16:36:03 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Volume 480a94a0-d1c9-46c0-bf9f-c42f907b4b82: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 16:36:03 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] deleating volume v_480a94a0-d1c9-46c0-bf9f-c42f907b4b82 {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 16:36:03 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] get snapshots for volume v_480a94a0-d1c9-46c0-bf9f-c42f907b4b82 {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 16:36:03 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_480a94a0-d1c9-46c0-bf9f-c42f907b4b82/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:36:03 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_480a94a0-d1c9-46c0-bf9f-c42f907b4b82/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:36:03 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Volume reschedule parameters: True retry: {'num_attempts': 3, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 16:36:03 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Flow 'volume_create_manager' (c4715c3c-504e-41ce-8a1b-63b6d2dc8538) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 16:36:03 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (35dbda95-7e26-4948-81d5-b79d1b91220d) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:36:03 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (35dbda95-7e26-4948-81d5-b79d1b91220d) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:31:26Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumeQuotasAdminTestJSON-Volume-2080868017',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=480a94a0-d1c9-46c0-bf9f-c42f907b4b82,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='6e4c2ec9177349a08ce53e07b24ad79c',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:36:03Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:36:03Z,use_quota=True,user_id='ab5017da30ac4d47ba665ff135deb309',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:36:03 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (653462dc-cbb2-4a03-b331-cb3dce7448b6) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:36:03 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (653462dc-cbb2-4a03-b331-cb3dce7448b6) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:36:03 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (092baa4c-688b-493d-894e-3ad030e4c933) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:36:03 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (092baa4c-688b-493d-894e-3ad030e4c933) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': '480a94a0-d1c9-46c0-bf9f-c42f907b4b82', 'volume_name': 'volume-480a94a0-d1c9-46c0-bf9f-c42f907b4b82', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:36:03 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (31b1dd3f-e8dc-480d-8a53-9ad63fd0e00a) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:36:03 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (31b1dd3f-e8dc-480d-8a53-9ad63fd0e00a) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:36:03 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (afbcb3fd-a5f9-45a1-bdc1-28a6f5b8d63a) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:36:03 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Volume 480a94a0-d1c9-46c0-bf9f-c42f907b4b82: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-480a94a0-d1c9-46c0-bf9f-c42f907b4b82', 'volume_size': 1} Apr 17 16:36:03 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] creating volume v_480a94a0-d1c9-46c0-bf9f-c42f907b4b82. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 16:36:03 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] create volume {'name': 'v_480a94a0-d1c9-46c0-bf9f-c42f907b4b82', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 16:36:03 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:36:03 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:36:04 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=16,cinder:UPDATE=6,cinder:INSERT=5 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:36:06 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 139.095(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:36:06 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:36:06 user cinder-volume[84696]: ERROR oslo_service.periodic_task [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Error during VolumeManager.publish_service_capabilities: TypeError: 'NoneType' object is not subscriptable Apr 17 16:36:06 user cinder-volume[84696]: ERROR oslo_service.periodic_task Traceback (most recent call last): Apr 17 16:36:06 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/usr/local/lib/python3.10/dist-packages/oslo_service/periodic_task.py", line 216, in run_periodic_tasks Apr 17 16:36:06 user cinder-volume[84696]: ERROR oslo_service.periodic_task task(self, context) Apr 17 16:36:06 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/manager.py", line 2876, in publish_service_capabilities Apr 17 16:36:06 user cinder-volume[84696]: ERROR oslo_service.periodic_task self._report_driver_status(context) Apr 17 16:36:06 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/manager.py", line 2755, in _report_driver_status Apr 17 16:36:06 user cinder-volume[84696]: ERROR oslo_service.periodic_task volume_stats = get_stats() Apr 17 16:36:06 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/usr/local/lib/python3.10/dist-packages/oslo_utils/timeutils.py", line 319, in wrapper Apr 17 16:36:06 user cinder-volume[84696]: ERROR oslo_service.periodic_task result = func(*args, **kwargs) Apr 17 16:36:06 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/manager.py", line 2753, in get_stats Apr 17 16:36:06 user cinder-volume[84696]: ERROR oslo_service.periodic_task return self.driver.get_volume_stats(refresh=True) Apr 17 16:36:06 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/driver.py", line 666, in get_volume_stats Apr 17 16:36:06 user cinder-volume[84696]: ERROR oslo_service.periodic_task self._update_volume_stats() Apr 17 16:36:06 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 726, in _update_volume_stats Apr 17 16:36:06 user cinder-volume[84696]: ERROR oslo_service.periodic_task pool_stats = self.ra.get_pool_stats() Apr 17 16:36:06 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 897, in get_pool_stats Apr 17 16:36:06 user cinder-volume[84696]: ERROR oslo_service.periodic_task if not resp["error"] and resp["code"] == 200: Apr 17 16:36:06 user cinder-volume[84696]: ERROR oslo_service.periodic_task TypeError: 'NoneType' object is not subscriptable Apr 17 16:36:06 user cinder-volume[84696]: ERROR oslo_service.periodic_task Apr 17 16:36:06 user cinder-volume[84696]: DEBUG cinder.utils [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.874(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:36:06 user cinder-volume[84696]: DEBUG cinder.utils [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:36:06 user cinder-volume[84696]: DEBUG cinder.utils [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.044(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:36:06 user cinder-volume[84696]: DEBUG cinder.utils [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_7384f51d-c3a2-4eea-a968-ec043946989f/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:36:06 user cinder-volume[84696]: DEBUG cinder.utils [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.786(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:36:06 user cinder-volume[84696]: DEBUG cinder.utils [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:36:06 user cinder-volume[84696]: DEBUG cinder.utils [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.965(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:36:06 user cinder-volume[84696]: DEBUG cinder.utils [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_480a94a0-d1c9-46c0-bf9f-c42f907b4b82/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:36:06 user cinder-volume[84696]: WARNING oslo.service.loopingcall [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Function 'cinder.service.Service.periodic_tasks' run outlasted interval by 79.10 sec Apr 17 16:36:06 user cinder-volume[84696]: DEBUG oslo_service.periodic_task [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Running periodic task VolumeManager.publish_service_capabilities {{(pid=84696) run_periodic_tasks /usr/local/lib/python3.10/dist-packages/oslo_service/periodic_task.py:210}} Apr 17 16:36:06 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Updating volume stats {{(pid=84696) _update_volume_stats /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:724}} Apr 17 16:36:06 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Get pool Pool-0 fsprops {{(pid=84696) get_pool_stats /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:894}} Apr 17 16:36:06 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:36:06 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:36:09 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 3.063(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:36:09 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:36:09 user cinder-volume[84696]: DEBUG cinder.utils [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.856(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:36:09 user cinder-volume[84696]: DEBUG cinder.utils [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:36:09 user cinder-volume[84696]: DEBUG cinder.utils [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.034(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:36:09 user cinder-volume[84696]: DEBUG cinder.utils [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_480a94a0-d1c9-46c0-bf9f-c42f907b4b82/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:36:12 user cinder-volume[84696]: DEBUG cinder.utils [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.210(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:36:12 user cinder-volume[84696]: DEBUG cinder.utils [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:36:12 user cinder-volume[84696]: DEBUG cinder.utils [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.188(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:36:12 user cinder-volume[84696]: DEBUG cinder.utils [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_7384f51d-c3a2-4eea-a968-ec043946989f/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:36:12 user cinder-volume[84696]: DEBUG cinder.utils [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.020(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:36:12 user cinder-volume[84696]: DEBUG cinder.utils [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:36:12 user cinder-volume[84696]: DEBUG cinder.utils [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.395(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:36:12 user cinder-volume[84696]: DEBUG cinder.utils [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_bf85ce23-1db3-4881-bd7a-b20b520718b8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:36:12 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.141(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:36:12 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:36:15 user cinder-volume[84696]: DEBUG cinder.utils [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 74.450(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:36:15 user cinder-volume[84696]: DEBUG cinder.utils [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_2159f1f1-2564-4d84-8110-c1f7d843edb1/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:36:15 user cinder-volume[84696]: DEBUG cinder.utils [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.001(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:36:15 user cinder-volume[84696]: DEBUG cinder.utils [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:36:15 user cinder-volume[84696]: DEBUG cinder.utils [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.179(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:36:15 user cinder-volume[84696]: DEBUG cinder.utils [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_480a94a0-d1c9-46c0-bf9f-c42f907b4b82/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:36:15 user cinder-volume[84696]: DEBUG cinder.utils [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 74.265(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:36:15 user cinder-volume[84696]: DEBUG cinder.utils [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_8b7f1136-3d20-4cf8-a521-6db1d3a0c8a8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:36:18 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.279(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:36:18 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:36:21 user cinder-volume[84696]: DEBUG cinder.utils [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.401(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:36:21 user cinder-volume[84696]: DEBUG cinder.utils [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_7384f51d-c3a2-4eea-a968-ec043946989f/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:36:21 user cinder-volume[84696]: DEBUG cinder.utils [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 74.516(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:36:21 user cinder-volume[84696]: DEBUG cinder.utils [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_f6c0e341-6f00-4e32-bd07-31f954c1ad8a/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:36:21 user cinder-volume[84696]: DEBUG cinder.utils [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.234(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:36:21 user cinder-volume[84696]: DEBUG cinder.utils [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:36:24 user cinder-volume[84696]: DEBUG cinder.utils [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.215(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:36:24 user cinder-volume[84696]: DEBUG cinder.utils [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:36:24 user cinder-volume[84696]: DEBUG cinder.utils [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.394(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:36:24 user cinder-volume[84696]: DEBUG cinder.utils [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_480a94a0-d1c9-46c0-bf9f-c42f907b4b82/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:36:24 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=20,cinder:UPDATE=6,cinder:INSERT=1 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:36:27 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.495(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:36:27 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:36:30 user cinder-volume[84696]: DEBUG cinder.utils [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.641(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:36:30 user cinder-volume[84696]: DEBUG cinder.utils [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:36:30 user cinder-volume[84696]: DEBUG cinder.utils [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.824(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:36:30 user cinder-volume[84696]: DEBUG cinder.utils [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_bf85ce23-1db3-4881-bd7a-b20b520718b8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:36:34 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=5,cinder:UPDATE=2 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:36:40 user cinder-volume[84696]: DEBUG cinder.utils [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 40.272(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:36:40 user cinder-volume[84696]: DEBUG cinder.utils [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:36:40 user cinder-volume[84696]: DEBUG cinder.utils [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 40.443(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:36:40 user cinder-volume[84696]: DEBUG cinder.utils [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_7384f51d-c3a2-4eea-a968-ec043946989f/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:36:43 user cinder-volume[84696]: DEBUG cinder.utils [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 40.255(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:36:43 user cinder-volume[84696]: DEBUG cinder.utils [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:36:43 user cinder-volume[84696]: DEBUG cinder.utils [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 40.433(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:36:43 user cinder-volume[84696]: DEBUG cinder.utils [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_480a94a0-d1c9-46c0-bf9f-c42f907b4b82/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:36:46 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 40.535(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:36:46 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:36:54 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=5,cinder:UPDATE=1 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:37:03 user cinder-volume[84696]: DEBUG cinder.utils [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 72.719(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:37:03 user cinder-volume[84696]: DEBUG cinder.utils [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_bf85ce23-1db3-4881-bd7a-b20b520718b8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:37:03 user cinder-volume[84696]: DEBUG cinder.utils [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 72.541(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:37:03 user cinder-volume[84696]: DEBUG cinder.utils [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:37:04 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=3,cinder:UPDATE=2 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:37:07 user cinder-volume[84696]: DEBUG cinder.utils [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 139.570(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:37:07 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:37:07 user cinder-volume[84696]: INFO cinder.message.api [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Creating message record for request_id = req-26b6ca51-cba9-4420-975a-eec85c094fd9 Apr 17 16:37:07 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 16:37:07 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (f5ba8a12-e4f8-4a35-a487-af09bc1fce02) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 16:37:07 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 16:37:07 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:34:47Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesMetadataTest-Volume-1367745540',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=e1bb78ac-29e3-46df-9bbe-3b152e39b69a,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='7f9b364e902a445082273ba68da887b4',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:34:47Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:34:47Z,use_quota=True,user_id='ad9fef0ffef84b1783a6075d92bb27c1',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 16:37:07 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:34:47Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesMetadataTest-Volume-1367745540',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=e1bb78ac-29e3-46df-9bbe-3b152e39b69a,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='7f9b364e902a445082273ba68da887b4',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:34:47Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:34:47Z,use_quota=True,user_id='ad9fef0ffef84b1783a6075d92bb27c1',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(e1bb78ac-29e3-46df-9bbe-3b152e39b69a),volume_id=e1bb78ac-29e3-46df-9bbe-3b152e39b69a,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': 'e1bb78ac-29e3-46df-9bbe-3b152e39b69a', 'volume_name': 'volume-e1bb78ac-29e3-46df-9bbe-3b152e39b69a', 'volume_size': 1}} Apr 17 16:37:07 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'retry': {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']}, 'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': 'e1bb78ac-29e3-46df-9bbe-3b152e39b69a', 'volume': {'id': 'e1bb78ac-29e3-46df-9bbe-3b152e39b69a', '_name_id': None, 'ec2_id': None, 'user_id': 'ad9fef0ffef84b1783a6075d92bb27c1', 'project_id': '7f9b364e902a445082273ba68da887b4', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-VolumesMetadataTest-Volume-1367745540', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T16:34:47.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-e1bb78ac-29e3-46df-9bbe-3b152e39b69a', 'name_id': 'e1bb78ac-29e3-46df-9bbe-3b152e39b69a', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumesMetadataTest-Volume-1367745540', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '7f9b364e902a445082273ba68da887b4', 'qos_specs': None, 'reservations': ['534d1a95-6891-4aa6-a81c-55f25671b440', '53d79034-9b12-4c87-ae06-d3cba4398e31', 'a0ad6a16-6f0e-4d9e-b17b-638e4e0563a9', '1856cb28-c4d3-49f0-94e1-95e881d39510'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'ad9fef0ffef84b1783a6075d92bb27c1', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumesMetadataTest-Volume-1367745540', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '7f9b364e902a445082273ba68da887b4', 'qos_specs': None, 'reservations': ['534d1a95-6891-4aa6-a81c-55f25671b440', '53d79034-9b12-4c87-ae06-d3cba4398e31', 'a0ad6a16-6f0e-4d9e-b17b-638e4e0563a9', '1856cb28-c4d3-49f0-94e1-95e881d39510'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'ad9fef0ffef84b1783a6075d92bb27c1', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': 'ad9fef0ffef84b1783a6075d92bb27c1', 'metadata': {}, 'qos_specs': None}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(e1bb78ac-29e3-46df-9bbe-3b152e39b69a),volume_id=e1bb78ac-29e3-46df-9bbe-3b152e39b69a,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:34:47Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesMetadataTest-Volume-1367745540',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=e1bb78ac-29e3-46df-9bbe-3b152e39b69a,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='7f9b364e902a445082273ba68da887b4',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:34:47Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:34:47Z,use_quota=True,user_id='ad9fef0ffef84b1783a6075d92bb27c1',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 16:37:07 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:34:47Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesMetadataTest-Volume-1367745540',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=e1bb78ac-29e3-46df-9bbe-3b152e39b69a,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='7f9b364e902a445082273ba68da887b4',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:34:47Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:34:47Z,use_quota=True,user_id='ad9fef0ffef84b1783a6075d92bb27c1',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:34:47Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesMetadataTest-Volume-1367745540',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=e1bb78ac-29e3-46df-9bbe-3b152e39b69a,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='7f9b364e902a445082273ba68da887b4',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:34:47Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:34:47Z,use_quota=True,user_id='ad9fef0ffef84b1783a6075d92bb27c1',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 16:37:07 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 16:37:07 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 16:37:07 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:37:07 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 16:37:07 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:37:07 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 16:37:07 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:37:07 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 16:37:07 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:37:07 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 16:37:07 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:37:07 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 16:37:07 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:37:07 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 16:37:07 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:37:07 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 16:37:07 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:37:07 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:37:07 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 16:37:07 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 16:37:07 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (f5ba8a12-e4f8-4a35-a487-af09bc1fce02) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:37:07 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (f5ba8a12-e4f8-4a35-a487-af09bc1fce02) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:37:07 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (9fc9f51e-0719-4af4-9c62-1e1b064ac7ee) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:37:07 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (9fc9f51e-0719-4af4-9c62-1e1b064ac7ee) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:37:07 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (c6fb897e-a233-4286-b4e4-260cb051ef6b) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:37:07 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (c6fb897e-a233-4286-b4e4-260cb051ef6b) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:37:07 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (375784c9-e555-42dd-8541-5e77aed44c93) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:37:07 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Updating volume e1bb78ac-29e3-46df-9bbe-3b152e39b69a with {'scheduled_at': datetime.datetime(2023, 4, 17, 16, 37, 7, 64132), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 16:37:07 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Volume e1bb78ac-29e3-46df-9bbe-3b152e39b69a: re-scheduling SchedulerAPI.create_volume attempt 1 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 16:37:07 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Volume e1bb78ac-29e3-46df-9bbe-3b152e39b69a: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 16:37:07 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] deleating volume v_e1bb78ac-29e3-46df-9bbe-3b152e39b69a {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 16:37:07 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] get snapshots for volume v_e1bb78ac-29e3-46df-9bbe-3b152e39b69a {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 16:37:07 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_e1bb78ac-29e3-46df-9bbe-3b152e39b69a/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:37:07 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_e1bb78ac-29e3-46df-9bbe-3b152e39b69a/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:37:07 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Volume reschedule parameters: True retry: {'num_attempts': 2, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 16:37:07 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Flow 'volume_create_manager' (4cbbbbfc-077c-49f9-9e47-ffaefe723479) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 16:37:07 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (4e365595-98e0-4ea9-9930-0ae17e7fa1c8) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:37:07 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (4e365595-98e0-4ea9-9930-0ae17e7fa1c8) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:34:47Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesMetadataTest-Volume-1367745540',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=e1bb78ac-29e3-46df-9bbe-3b152e39b69a,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='7f9b364e902a445082273ba68da887b4',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:37:07Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:37:07Z,use_quota=True,user_id='ad9fef0ffef84b1783a6075d92bb27c1',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:37:07 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (2ed41617-9bca-42e8-b9c3-47cf3fe46f06) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:37:07 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (2ed41617-9bca-42e8-b9c3-47cf3fe46f06) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:37:07 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (ddb7f97d-6392-49bb-a463-a9f6b768e9c5) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:37:07 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (ddb7f97d-6392-49bb-a463-a9f6b768e9c5) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': 'e1bb78ac-29e3-46df-9bbe-3b152e39b69a', 'volume_name': 'volume-e1bb78ac-29e3-46df-9bbe-3b152e39b69a', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:37:07 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (2ff61726-4b32-49da-b668-0417e40a37ba) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:37:07 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (2ff61726-4b32-49da-b668-0417e40a37ba) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:37:07 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (948fcb6d-c840-4e23-b528-b6acbf95c0ab) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:37:07 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Volume e1bb78ac-29e3-46df-9bbe-3b152e39b69a: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-e1bb78ac-29e3-46df-9bbe-3b152e39b69a', 'volume_size': 1} Apr 17 16:37:07 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] creating volume v_e1bb78ac-29e3-46df-9bbe-3b152e39b69a. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 16:37:07 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] create volume {'name': 'v_e1bb78ac-29e3-46df-9bbe-3b152e39b69a', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 16:37:07 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:37:07 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:37:10 user cinder-volume[84696]: DEBUG cinder.utils [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.992(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:37:10 user cinder-volume[84696]: DEBUG cinder.utils [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_e1bb78ac-29e3-46df-9bbe-3b152e39b69a/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:37:10 user cinder-volume[84696]: DEBUG cinder.utils [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.833(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:37:10 user cinder-volume[84696]: DEBUG cinder.utils [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:37:13 user cinder-volume[84696]: DEBUG cinder.utils [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.060(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:37:13 user cinder-volume[84696]: DEBUG cinder.utils [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_e1bb78ac-29e3-46df-9bbe-3b152e39b69a/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:37:13 user cinder-volume[84696]: DEBUG cinder.utils [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 72.945(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:37:13 user cinder-volume[84696]: DEBUG cinder.utils [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:37:13 user cinder-volume[84696]: DEBUG cinder.utils [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.902(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:37:13 user cinder-volume[84696]: DEBUG cinder.utils [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:37:13 user cinder-volume[84696]: DEBUG cinder.utils [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 73.117(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:37:13 user cinder-volume[84696]: DEBUG cinder.utils [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_7384f51d-c3a2-4eea-a968-ec043946989f/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:37:16 user cinder-volume[84696]: DEBUG cinder.utils [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 72.926(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:37:16 user cinder-volume[84696]: DEBUG cinder.utils [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:37:16 user cinder-volume[84696]: DEBUG cinder.utils [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 73.105(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:37:16 user cinder-volume[84696]: DEBUG cinder.utils [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_480a94a0-d1c9-46c0-bf9f-c42f907b4b82/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:37:19 user cinder-volume[84696]: DEBUG cinder.utils [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.267(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:37:19 user cinder-volume[84696]: DEBUG cinder.utils [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:37:19 user cinder-volume[84696]: DEBUG cinder.utils [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.430(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:37:19 user cinder-volume[84696]: DEBUG cinder.utils [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_e1bb78ac-29e3-46df-9bbe-3b152e39b69a/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:37:19 user cinder-volume[84696]: DEBUG cinder.utils [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 138.489(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:37:19 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_8b7f1136-3d20-4cf8-a521-6db1d3a0c8a8/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:37:19 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (39ce3927-4fa9-4393-b9eb-4475175819de) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 16:37:19 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (fdd97590-2600-48c4-8049-02c324ef348b) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:37:19 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (fdd97590-2600-48c4-8049-02c324ef348b) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:37:19 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Flow 'volume_create_manager' (66ae15f6-be27-41c1-a5ca-8479c8adc816) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 16:37:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-ccbb52f2-042b-41e0-9837-623e8a9a2738 tempest-VolumesDeleteCascade-645320261 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 16:37:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 16:37:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 16:37:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 16:37:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 16:37:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 16:37:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 16:37:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 16:37:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 16:37:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 16:37:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 16:37:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 16:37:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 16:37:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 16:37:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 16:37:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 16:37:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 16:37:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 16:37:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 16:37:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 16:37:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 16:37:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 16:37:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 16:37:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 16:37:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:37:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 16:37:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:37:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 16:37:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:37:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 16:37:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:37:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 16:37:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:37:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 16:37:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:37:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 16:37:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:37:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 16:37:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:37:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:37:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 16:37:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 16:37:19 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 73.467(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:37:19 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:37:19 user cinder-volume[84696]: DEBUG cinder.utils [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 138.711(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:37:19 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_2159f1f1-2564-4d84-8110-c1f7d843edb1/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:37:19 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (66f629a8-6266-48fd-8ac5-26a9a6bec901) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 16:37:19 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (433a3676-9d92-4297-a633-daed55cb3767) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:37:19 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (433a3676-9d92-4297-a633-daed55cb3767) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:37:19 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Flow 'volume_create_manager' (7111dffa-ede2-4b36-8afd-1d12c61e9309) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 16:37:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-34802d26-273b-4fa3-a0d4-af12da0d916e tempest-VolumesSnapshotListTestJSON-1099444740 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 16:37:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 16:37:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 16:37:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 16:37:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 16:37:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 16:37:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 16:37:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 16:37:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 16:37:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 16:37:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 16:37:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 16:37:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 16:37:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 16:37:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 16:37:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 16:37:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 16:37:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 16:37:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 16:37:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 16:37:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 16:37:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 16:37:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 16:37:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 16:37:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:37:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 16:37:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:37:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 16:37:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:37:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 16:37:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:37:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 16:37:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:37:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 16:37:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:37:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 16:37:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:37:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 16:37:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:37:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:37:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 16:37:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 16:37:24 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=8,cinder:UPDATE=4 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:37:26 user cinder-volume[84696]: DEBUG cinder.utils [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 139.603(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:37:26 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_f6c0e341-6f00-4e32-bd07-31f954c1ad8a/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:37:26 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (854e74aa-081b-4a34-9787-2c0edccf9773) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 16:37:26 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (5b747472-600c-466e-9baf-9cf308b7082c) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:37:26 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (5b747472-600c-466e-9baf-9cf308b7082c) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:37:26 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Flow 'volume_create_manager' (d2d134d5-e4bd-498d-aff4-08ba0e172157) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 16:37:26 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-4c23c8b0-66e0-4b59-bc55-c45427bf9e5d tempest-VolumeQuotasAdminTestJSON-623147042 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 16:37:26 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 16:37:26 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 16:37:26 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 16:37:26 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 16:37:26 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 16:37:26 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 16:37:26 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 16:37:26 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 16:37:26 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 16:37:26 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 16:37:26 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 16:37:26 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 16:37:26 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 16:37:26 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 16:37:26 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 16:37:26 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 16:37:26 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 16:37:26 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 16:37:26 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 16:37:26 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 16:37:26 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 16:37:26 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 16:37:26 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 16:37:26 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:37:26 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 16:37:26 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:37:26 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 16:37:26 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:37:26 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 16:37:26 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:37:26 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 16:37:26 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:37:26 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 16:37:26 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:37:26 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 16:37:26 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:37:26 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 16:37:26 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:37:26 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:37:26 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 16:37:26 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 16:37:29 user cinder-volume[84696]: DEBUG cinder.utils [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 22.351(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:37:29 user cinder-volume[84696]: DEBUG cinder.utils [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:37:29 user cinder-volume[84696]: DEBUG cinder.utils [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 22.514(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:37:29 user cinder-volume[84696]: DEBUG cinder.utils [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_e1bb78ac-29e3-46df-9bbe-3b152e39b69a/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:37:44 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=1 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:37:44 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=13,cinder:INSERT=2,cinder:UPDATE=3 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:37:46 user cinder-volume[84696]: DEBUG cinder.utils [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.275(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:37:46 user cinder-volume[84696]: DEBUG cinder.utils [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:37:46 user cinder-volume[84696]: DEBUG cinder.utils [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.437(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:37:46 user cinder-volume[84696]: DEBUG cinder.utils [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_e1bb78ac-29e3-46df-9bbe-3b152e39b69a/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:38:04 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=4,cinder:UPDATE=1 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:38:04 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=1,cinder:UPDATE=1 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:38:07 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Volume reschedule parameters: True retry: {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 16:38:07 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Flow 'volume_create_manager' (a603acaf-a9be-4d41-8c24-10a6ca99b673) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 16:38:07 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (9521aa81-709c-4ed8-922f-567003cc6868) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:38:07 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (9521aa81-709c-4ed8-922f-567003cc6868) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:38:07Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesActionsTest-Volume-825907983',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=f920a5f7-106a-4092-b193-c3d340118d77,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='44c0ea9c46d14578a12440ef37ec453e',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:38:08Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:38:08Z,use_quota=True,user_id='e97b56d3796f42fa89023ef8691f08ad',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:38:07 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (79bb0968-9afb-431e-bf9d-34f3714a89de) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:38:07 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (79bb0968-9afb-431e-bf9d-34f3714a89de) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:38:07 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (1aa69d64-a1bf-480e-b138-1b3aad39a823) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:38:07 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (1aa69d64-a1bf-480e-b138-1b3aad39a823) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': 'f920a5f7-106a-4092-b193-c3d340118d77', 'volume_name': 'volume-f920a5f7-106a-4092-b193-c3d340118d77', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:38:07 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (57f333c0-6361-4694-945e-ee4e24342c7f) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:38:07 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (57f333c0-6361-4694-945e-ee4e24342c7f) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:38:07 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (536f62f3-5f49-453b-ba72-abca7ed9533e) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:38:07 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Volume f920a5f7-106a-4092-b193-c3d340118d77: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-f920a5f7-106a-4092-b193-c3d340118d77', 'volume_size': 1} Apr 17 16:38:07 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] creating volume v_f920a5f7-106a-4092-b193-c3d340118d77. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 16:38:07 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] create volume {'name': 'v_f920a5f7-106a-4092-b193-c3d340118d77', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 16:38:07 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:38:07 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:38:10 user cinder-volume[84696]: DEBUG cinder.utils [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 139.794(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:38:10 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_bf85ce23-1db3-4881-bd7a-b20b520718b8/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:38:10 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (d75b44e1-b13c-467d-a901-1faee8a72f76) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 16:38:10 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (b5d69e36-7b38-44e2-bd71-3bbb2648de38) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:38:10 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (b5d69e36-7b38-44e2-bd71-3bbb2648de38) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:38:10 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Flow 'volume_create_manager' (338fa437-d5f6-4108-b918-c18a712406d7) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 16:38:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 16:38:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 16:38:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 16:38:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 16:38:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 16:38:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 16:38:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 16:38:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 16:38:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 16:38:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 16:38:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 16:38:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 16:38:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 16:38:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 16:38:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 16:38:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 16:38:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 16:38:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 16:38:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 16:38:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 16:38:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 16:38:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 16:38:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 16:38:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 16:38:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:38:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 16:38:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:38:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 16:38:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:38:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 16:38:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:38:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 16:38:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:38:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 16:38:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:38:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 16:38:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:38:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 16:38:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:38:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:38:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 16:38:10 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 16:38:10 user cinder-volume[84696]: DEBUG cinder.utils [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.894(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:38:10 user cinder-volume[84696]: DEBUG cinder.utils [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:38:10 user cinder-volume[84696]: DEBUG cinder.utils [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 139.636(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:38:10 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:38:10 user cinder-volume[84696]: INFO cinder.message.api [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Creating message record for request_id = req-208fb94c-86af-4459-a279-1c8be9781cb3 Apr 17 16:38:10 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 16:38:10 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (bafbf2f1-5d48-4648-a84f-5bf00e566082) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 16:38:10 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 16:38:10 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:31:18Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-CreateVolumesFromSnapshotTest-Volume-2022997390',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=bf85ce23-1db3-4881-bd7a-b20b520718b8,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='93bbb103c036439da2861c7be15c3e7b',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:35:51Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:35:51Z,use_quota=True,user_id='ae7cdf32438c44b6b3f428e9e0142af1',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 16:38:10 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:31:18Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-CreateVolumesFromSnapshotTest-Volume-2022997390',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=bf85ce23-1db3-4881-bd7a-b20b520718b8,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='93bbb103c036439da2861c7be15c3e7b',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:35:51Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:35:51Z,use_quota=True,user_id='ae7cdf32438c44b6b3f428e9e0142af1',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(bf85ce23-1db3-4881-bd7a-b20b520718b8),volume_id=bf85ce23-1db3-4881-bd7a-b20b520718b8,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': 'bf85ce23-1db3-4881-bd7a-b20b520718b8', 'volume_name': 'volume-bf85ce23-1db3-4881-bd7a-b20b520718b8', 'volume_size': 1}} Apr 17 16:38:10 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': 'bf85ce23-1db3-4881-bd7a-b20b520718b8', 'volume': {'id': 'bf85ce23-1db3-4881-bd7a-b20b520718b8', '_name_id': None, 'ec2_id': None, 'user_id': 'ae7cdf32438c44b6b3f428e9e0142af1', 'project_id': '93bbb103c036439da2861c7be15c3e7b', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-CreateVolumesFromSnapshotTest-Volume-2022997390', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'admin_metadata': {}, 'glance_metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T16:31:17.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-bf85ce23-1db3-4881-bd7a-b20b520718b8', 'name_id': 'bf85ce23-1db3-4881-bd7a-b20b520718b8', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-CreateVolumesFromSnapshotTest-Volume-2022997390', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '93bbb103c036439da2861c7be15c3e7b', 'qos_specs': None, 'reservations': ['b69f8abb-8ed7-4741-8211-ad267bc6119b', '37f3f14f-b707-4dd5-868b-e177f7131d19', 'c87fa212-ff4d-42ec-8c41-d50482477074', '51763846-c4e0-4572-a1af-a15208766488'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'ae7cdf32438c44b6b3f428e9e0142af1', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-CreateVolumesFromSnapshotTest-Volume-2022997390', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '93bbb103c036439da2861c7be15c3e7b', 'qos_specs': None, 'reservations': ['b69f8abb-8ed7-4741-8211-ad267bc6119b', '37f3f14f-b707-4dd5-868b-e177f7131d19', 'c87fa212-ff4d-42ec-8c41-d50482477074', '51763846-c4e0-4572-a1af-a15208766488'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'ae7cdf32438c44b6b3f428e9e0142af1', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': 'ae7cdf32438c44b6b3f428e9e0142af1', 'metadata': {}, 'qos_specs': None, 'retry': {'num_attempts': 3, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']}}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(bf85ce23-1db3-4881-bd7a-b20b520718b8),volume_id=bf85ce23-1db3-4881-bd7a-b20b520718b8,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:31:18Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-CreateVolumesFromSnapshotTest-Volume-2022997390',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=bf85ce23-1db3-4881-bd7a-b20b520718b8,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='93bbb103c036439da2861c7be15c3e7b',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:35:51Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:35:51Z,use_quota=True,user_id='ae7cdf32438c44b6b3f428e9e0142af1',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 16:38:10 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:31:18Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-CreateVolumesFromSnapshotTest-Volume-2022997390',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=bf85ce23-1db3-4881-bd7a-b20b520718b8,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='93bbb103c036439da2861c7be15c3e7b',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:35:51Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:35:51Z,use_quota=True,user_id='ae7cdf32438c44b6b3f428e9e0142af1',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:31:18Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-CreateVolumesFromSnapshotTest-Volume-2022997390',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=bf85ce23-1db3-4881-bd7a-b20b520718b8,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='93bbb103c036439da2861c7be15c3e7b',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:35:51Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:35:51Z,use_quota=True,user_id='ae7cdf32438c44b6b3f428e9e0142af1',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 16:38:10 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 16:38:10 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 16:38:10 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:38:10 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 16:38:10 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:38:10 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 16:38:10 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:38:10 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 16:38:10 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:38:10 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 16:38:10 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:38:10 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 16:38:10 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:38:10 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 16:38:10 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:38:10 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 16:38:10 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:38:10 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:38:10 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 16:38:10 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 16:38:10 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (bafbf2f1-5d48-4648-a84f-5bf00e566082) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:38:10 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (bafbf2f1-5d48-4648-a84f-5bf00e566082) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:38:10 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (5328bac4-1da4-4960-906c-e030fc77097b) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:38:10 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (5328bac4-1da4-4960-906c-e030fc77097b) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:38:10 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (01fed539-9314-424a-9782-b8abccdb4d8a) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:38:10 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (01fed539-9314-424a-9782-b8abccdb4d8a) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:38:10 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (44f4fcbb-0a2d-4361-9c98-290e952f34ed) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:38:10 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Updating volume bf85ce23-1db3-4881-bd7a-b20b520718b8 with {'scheduled_at': datetime.datetime(2023, 4, 17, 16, 38, 10, 706659), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 16:38:10 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Volume bf85ce23-1db3-4881-bd7a-b20b520718b8: re-scheduling SchedulerAPI.create_volume attempt 3 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 16:38:10 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Volume bf85ce23-1db3-4881-bd7a-b20b520718b8: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 16:38:10 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] deleating volume v_bf85ce23-1db3-4881-bd7a-b20b520718b8 {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 16:38:10 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] get snapshots for volume v_bf85ce23-1db3-4881-bd7a-b20b520718b8 {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 16:38:10 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_bf85ce23-1db3-4881-bd7a-b20b520718b8/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:38:10 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_bf85ce23-1db3-4881-bd7a-b20b520718b8/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:38:13 user cinder-volume[84696]: DEBUG cinder.utils [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.957(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:38:13 user cinder-volume[84696]: DEBUG cinder.utils [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_bf85ce23-1db3-4881-bd7a-b20b520718b8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:38:13 user cinder-volume[84696]: DEBUG cinder.utils [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.949(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:38:13 user cinder-volume[84696]: DEBUG cinder.utils [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:38:16 user cinder-volume[84696]: DEBUG cinder.utils [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.026(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:38:16 user cinder-volume[84696]: DEBUG cinder.utils [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_bf85ce23-1db3-4881-bd7a-b20b520718b8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:38:19 user cinder-volume[84696]: DEBUG cinder.utils [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 139.801(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:38:19 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_7384f51d-c3a2-4eea-a968-ec043946989f/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:38:19 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (9b5a87a7-4b89-42c4-8980-3b6a59e23bc8) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 16:38:19 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (80489580-749f-442e-9351-662f3d5b9906) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:38:19 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (80489580-749f-442e-9351-662f3d5b9906) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:38:19 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Flow 'volume_create_manager' (101546a3-efad-4df1-87b5-2dbcadd98f8e) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 16:38:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 16:38:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 16:38:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 16:38:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 16:38:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 16:38:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 16:38:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 16:38:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 16:38:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 16:38:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 16:38:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 16:38:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 16:38:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 16:38:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 16:38:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 16:38:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 16:38:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 16:38:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 16:38:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 16:38:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 16:38:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 16:38:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 16:38:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 16:38:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 16:38:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:38:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 16:38:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:38:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 16:38:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:38:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 16:38:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:38:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 16:38:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:38:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 16:38:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:38:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 16:38:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:38:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 16:38:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:38:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:38:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 16:38:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 16:38:19 user cinder-volume[84696]: DEBUG cinder.utils [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 139.652(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:38:19 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:38:19 user cinder-volume[84696]: INFO cinder.message.api [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Creating message record for request_id = req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 Apr 17 16:38:19 user cinder-volume[84696]: DEBUG cinder.utils [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 72.774(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:38:19 user cinder-volume[84696]: DEBUG cinder.utils [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_e1bb78ac-29e3-46df-9bbe-3b152e39b69a/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:38:19 user cinder-volume[84696]: DEBUG cinder.utils [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 72.615(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:38:19 user cinder-volume[84696]: DEBUG cinder.utils [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:38:19 user cinder-volume[84696]: DEBUG cinder.utils [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.115(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:38:19 user cinder-volume[84696]: DEBUG cinder.utils [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:38:19 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 16:38:19 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (a930e0c2-aefa-4592-9dac-015c5f62c62a) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 16:38:19 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 16:38:19 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:31:21Z,deleted=False,deleted_at=None,display_description=None,display_name='CinderUnicodeTest塵㼗‽',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=7384f51d-c3a2-4eea-a968-ec043946989f,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='9ccd4e98adcb4fb18f651ead3736305a',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:36:00Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:36:00Z,use_quota=True,user_id='d3c58fb72fb443e19315cc15a76767fe',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 16:38:19 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:31:21Z,deleted=False,deleted_at=None,display_description=None,display_name='CinderUnicodeTest塵㼗‽',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=7384f51d-c3a2-4eea-a968-ec043946989f,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='9ccd4e98adcb4fb18f651ead3736305a',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:36:00Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:36:00Z,use_quota=True,user_id='d3c58fb72fb443e19315cc15a76767fe',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(7384f51d-c3a2-4eea-a968-ec043946989f),volume_id=7384f51d-c3a2-4eea-a968-ec043946989f,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': '7384f51d-c3a2-4eea-a968-ec043946989f', 'volume_name': 'volume-7384f51d-c3a2-4eea-a968-ec043946989f', 'volume_size': 1}} Apr 17 16:38:19 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': '7384f51d-c3a2-4eea-a968-ec043946989f', 'volume': {'id': '7384f51d-c3a2-4eea-a968-ec043946989f', '_name_id': None, 'ec2_id': None, 'user_id': 'd3c58fb72fb443e19315cc15a76767fe', 'project_id': '9ccd4e98adcb4fb18f651ead3736305a', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'CinderUnicodeTest塵㼗‽', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'admin_metadata': {}, 'glance_metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T16:31:21.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-7384f51d-c3a2-4eea-a968-ec043946989f', 'name_id': '7384f51d-c3a2-4eea-a968-ec043946989f', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'CinderUnicodeTest塵㼗‽', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '9ccd4e98adcb4fb18f651ead3736305a', 'qos_specs': None, 'reservations': ['e840cac3-58bf-4622-96ed-8d715415019e', '88e95250-2b05-45ff-b59d-697235ec7436', '29eb19dc-9f45-4e12-a7b9-01875910c4ae', 'be37e533-c8a6-464f-942b-ce65599da74b'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'd3c58fb72fb443e19315cc15a76767fe', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'CinderUnicodeTest塵㼗‽', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '9ccd4e98adcb4fb18f651ead3736305a', 'qos_specs': None, 'reservations': ['e840cac3-58bf-4622-96ed-8d715415019e', '88e95250-2b05-45ff-b59d-697235ec7436', '29eb19dc-9f45-4e12-a7b9-01875910c4ae', 'be37e533-c8a6-464f-942b-ce65599da74b'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'd3c58fb72fb443e19315cc15a76767fe', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': 'd3c58fb72fb443e19315cc15a76767fe', 'metadata': {}, 'qos_specs': None, 'retry': {'num_attempts': 3, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']}}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(7384f51d-c3a2-4eea-a968-ec043946989f),volume_id=7384f51d-c3a2-4eea-a968-ec043946989f,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:31:21Z,deleted=False,deleted_at=None,display_description=None,display_name='CinderUnicodeTest塵㼗‽',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=7384f51d-c3a2-4eea-a968-ec043946989f,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='9ccd4e98adcb4fb18f651ead3736305a',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:36:00Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:36:00Z,use_quota=True,user_id='d3c58fb72fb443e19315cc15a76767fe',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 16:38:19 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:31:21Z,deleted=False,deleted_at=None,display_description=None,display_name='CinderUnicodeTest塵㼗‽',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=7384f51d-c3a2-4eea-a968-ec043946989f,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='9ccd4e98adcb4fb18f651ead3736305a',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:36:00Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:36:00Z,use_quota=True,user_id='d3c58fb72fb443e19315cc15a76767fe',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:31:21Z,deleted=False,deleted_at=None,display_description=None,display_name='CinderUnicodeTest塵㼗‽',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=7384f51d-c3a2-4eea-a968-ec043946989f,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='9ccd4e98adcb4fb18f651ead3736305a',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:36:00Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:36:00Z,use_quota=True,user_id='d3c58fb72fb443e19315cc15a76767fe',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 16:38:19 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 16:38:19 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 16:38:19 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:38:19 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 16:38:19 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:38:19 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 16:38:19 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:38:19 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 16:38:19 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:38:19 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 16:38:19 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:38:19 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 16:38:19 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:38:19 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 16:38:19 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:38:19 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 16:38:19 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:38:19 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:38:19 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 16:38:19 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 16:38:19 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (a930e0c2-aefa-4592-9dac-015c5f62c62a) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:38:19 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (a930e0c2-aefa-4592-9dac-015c5f62c62a) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:38:19 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (c3e17118-1398-4051-af0a-cb5ea0a53528) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:38:19 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (c3e17118-1398-4051-af0a-cb5ea0a53528) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:38:19 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (8db470a3-373c-4eb4-b233-49fb5e034f6b) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:38:19 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (8db470a3-373c-4eb4-b233-49fb5e034f6b) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:38:19 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (d7f3d394-9859-4e12-aa4a-64ce16c48c98) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:38:19 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Updating volume 7384f51d-c3a2-4eea-a968-ec043946989f with {'scheduled_at': datetime.datetime(2023, 4, 17, 16, 38, 19, 930823), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 16:38:19 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Volume 7384f51d-c3a2-4eea-a968-ec043946989f: re-scheduling SchedulerAPI.create_volume attempt 3 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 16:38:19 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Volume 7384f51d-c3a2-4eea-a968-ec043946989f: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 16:38:19 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] deleating volume v_7384f51d-c3a2-4eea-a968-ec043946989f {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 16:38:19 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] get snapshots for volume v_7384f51d-c3a2-4eea-a968-ec043946989f {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 16:38:19 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_7384f51d-c3a2-4eea-a968-ec043946989f/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:38:19 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_7384f51d-c3a2-4eea-a968-ec043946989f/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:38:22 user cinder-volume[84696]: DEBUG cinder.utils [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.169(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:38:22 user cinder-volume[84696]: DEBUG cinder.utils [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_bf85ce23-1db3-4881-bd7a-b20b520718b8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:38:22 user cinder-volume[84696]: DEBUG cinder.utils [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.946(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:38:22 user cinder-volume[84696]: DEBUG cinder.utils [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_7384f51d-c3a2-4eea-a968-ec043946989f/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:38:22 user cinder-volume[84696]: DEBUG cinder.utils [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 139.794(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:38:22 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_480a94a0-d1c9-46c0-bf9f-c42f907b4b82/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:38:22 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (f7be13b7-4f59-4082-8999-dd5515bc049f) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 16:38:22 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (fb48d423-dc2a-4726-8949-0eb0bb594051) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:38:22 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (fb48d423-dc2a-4726-8949-0eb0bb594051) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:38:22 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Flow 'volume_create_manager' (7daf9069-8e3f-4c38-8e02-de0fa1668442) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 16:38:22 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 16:38:22 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 16:38:22 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 16:38:22 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 16:38:22 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 16:38:22 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 16:38:22 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 16:38:22 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 16:38:22 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 16:38:22 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 16:38:22 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 16:38:22 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 16:38:22 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 16:38:22 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 16:38:22 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 16:38:22 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 16:38:22 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 16:38:22 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 16:38:22 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 16:38:22 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 16:38:22 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 16:38:22 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 16:38:22 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 16:38:22 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 16:38:22 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:38:22 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 16:38:22 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:38:22 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 16:38:22 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:38:22 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 16:38:22 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:38:22 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 16:38:22 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:38:22 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 16:38:22 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:38:22 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 16:38:22 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:38:22 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 16:38:22 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:38:22 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:38:22 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 16:38:22 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 16:38:22 user cinder-volume[84696]: DEBUG cinder.utils [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 139.634(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:38:22 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:38:22 user cinder-volume[84696]: INFO cinder.message.api [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Creating message record for request_id = req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b Apr 17 16:38:22 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 16:38:22 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (afbcb3fd-a5f9-45a1-bdc1-28a6f5b8d63a) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 16:38:22 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 16:38:22 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:31:26Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumeQuotasAdminTestJSON-Volume-2080868017',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=480a94a0-d1c9-46c0-bf9f-c42f907b4b82,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='6e4c2ec9177349a08ce53e07b24ad79c',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:36:03Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:36:03Z,use_quota=True,user_id='ab5017da30ac4d47ba665ff135deb309',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 16:38:22 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:31:26Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumeQuotasAdminTestJSON-Volume-2080868017',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=480a94a0-d1c9-46c0-bf9f-c42f907b4b82,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='6e4c2ec9177349a08ce53e07b24ad79c',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:36:03Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:36:03Z,use_quota=True,user_id='ab5017da30ac4d47ba665ff135deb309',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(480a94a0-d1c9-46c0-bf9f-c42f907b4b82),volume_id=480a94a0-d1c9-46c0-bf9f-c42f907b4b82,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': '480a94a0-d1c9-46c0-bf9f-c42f907b4b82', 'volume_name': 'volume-480a94a0-d1c9-46c0-bf9f-c42f907b4b82', 'volume_size': 1}} Apr 17 16:38:22 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': '480a94a0-d1c9-46c0-bf9f-c42f907b4b82', 'volume': {'id': '480a94a0-d1c9-46c0-bf9f-c42f907b4b82', '_name_id': None, 'ec2_id': None, 'user_id': 'ab5017da30ac4d47ba665ff135deb309', 'project_id': '6e4c2ec9177349a08ce53e07b24ad79c', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-VolumeQuotasAdminTestJSON-Volume-2080868017', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'admin_metadata': {}, 'glance_metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T16:31:25.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-480a94a0-d1c9-46c0-bf9f-c42f907b4b82', 'name_id': '480a94a0-d1c9-46c0-bf9f-c42f907b4b82', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumeQuotasAdminTestJSON-Volume-2080868017', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '6e4c2ec9177349a08ce53e07b24ad79c', 'qos_specs': None, 'reservations': ['00e83981-2b5c-4129-be4e-d251babb9099', '4e65b249-f315-4ce2-9106-e50bca02bf14', '166cbb51-86ba-4d39-9450-64b18ab85006', '47198bf3-67c0-44fb-9cca-8e33c9ae31c6'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'ab5017da30ac4d47ba665ff135deb309', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumeQuotasAdminTestJSON-Volume-2080868017', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '6e4c2ec9177349a08ce53e07b24ad79c', 'qos_specs': None, 'reservations': ['00e83981-2b5c-4129-be4e-d251babb9099', '4e65b249-f315-4ce2-9106-e50bca02bf14', '166cbb51-86ba-4d39-9450-64b18ab85006', '47198bf3-67c0-44fb-9cca-8e33c9ae31c6'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'ab5017da30ac4d47ba665ff135deb309', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': 'ab5017da30ac4d47ba665ff135deb309', 'metadata': {}, 'qos_specs': None, 'retry': {'num_attempts': 3, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']}}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(480a94a0-d1c9-46c0-bf9f-c42f907b4b82),volume_id=480a94a0-d1c9-46c0-bf9f-c42f907b4b82,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:31:26Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumeQuotasAdminTestJSON-Volume-2080868017',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=480a94a0-d1c9-46c0-bf9f-c42f907b4b82,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='6e4c2ec9177349a08ce53e07b24ad79c',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:36:03Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:36:03Z,use_quota=True,user_id='ab5017da30ac4d47ba665ff135deb309',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 16:38:22 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:31:26Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumeQuotasAdminTestJSON-Volume-2080868017',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=480a94a0-d1c9-46c0-bf9f-c42f907b4b82,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='6e4c2ec9177349a08ce53e07b24ad79c',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:36:03Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:36:03Z,use_quota=True,user_id='ab5017da30ac4d47ba665ff135deb309',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:31:26Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumeQuotasAdminTestJSON-Volume-2080868017',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=480a94a0-d1c9-46c0-bf9f-c42f907b4b82,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='6e4c2ec9177349a08ce53e07b24ad79c',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:36:03Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:36:03Z,use_quota=True,user_id='ab5017da30ac4d47ba665ff135deb309',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 16:38:22 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 16:38:22 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 16:38:22 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:38:22 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 16:38:22 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:38:22 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 16:38:22 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:38:22 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 16:38:22 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:38:22 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 16:38:22 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:38:22 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 16:38:22 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:38:22 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 16:38:22 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:38:22 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 16:38:22 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:38:22 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:38:22 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 16:38:22 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 16:38:22 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (afbcb3fd-a5f9-45a1-bdc1-28a6f5b8d63a) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:38:22 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (afbcb3fd-a5f9-45a1-bdc1-28a6f5b8d63a) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:38:22 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (31b1dd3f-e8dc-480d-8a53-9ad63fd0e00a) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:38:22 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (31b1dd3f-e8dc-480d-8a53-9ad63fd0e00a) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:38:22 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (092baa4c-688b-493d-894e-3ad030e4c933) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:38:22 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (092baa4c-688b-493d-894e-3ad030e4c933) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:38:22 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (653462dc-cbb2-4a03-b331-cb3dce7448b6) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:38:22 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Updating volume 480a94a0-d1c9-46c0-bf9f-c42f907b4b82 with {'scheduled_at': datetime.datetime(2023, 4, 17, 16, 38, 22, 987980), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 16:38:22 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Volume 480a94a0-d1c9-46c0-bf9f-c42f907b4b82: re-scheduling SchedulerAPI.create_volume attempt 3 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 16:38:23 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Volume 480a94a0-d1c9-46c0-bf9f-c42f907b4b82: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 16:38:23 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] deleating volume v_480a94a0-d1c9-46c0-bf9f-c42f907b4b82 {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 16:38:23 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] get snapshots for volume v_480a94a0-d1c9-46c0-bf9f-c42f907b4b82 {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 16:38:23 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_480a94a0-d1c9-46c0-bf9f-c42f907b4b82/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:38:23 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_480a94a0-d1c9-46c0-bf9f-c42f907b4b82/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:38:25 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 139.895(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:38:25 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:38:25 user cinder-volume[84696]: ERROR oslo_service.periodic_task [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Error during VolumeManager.publish_service_capabilities: TypeError: 'NoneType' object is not subscriptable Apr 17 16:38:25 user cinder-volume[84696]: ERROR oslo_service.periodic_task Traceback (most recent call last): Apr 17 16:38:25 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/usr/local/lib/python3.10/dist-packages/oslo_service/periodic_task.py", line 216, in run_periodic_tasks Apr 17 16:38:25 user cinder-volume[84696]: ERROR oslo_service.periodic_task task(self, context) Apr 17 16:38:25 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/manager.py", line 2876, in publish_service_capabilities Apr 17 16:38:25 user cinder-volume[84696]: ERROR oslo_service.periodic_task self._report_driver_status(context) Apr 17 16:38:25 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/manager.py", line 2755, in _report_driver_status Apr 17 16:38:25 user cinder-volume[84696]: ERROR oslo_service.periodic_task volume_stats = get_stats() Apr 17 16:38:25 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/usr/local/lib/python3.10/dist-packages/oslo_utils/timeutils.py", line 319, in wrapper Apr 17 16:38:25 user cinder-volume[84696]: ERROR oslo_service.periodic_task result = func(*args, **kwargs) Apr 17 16:38:25 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/manager.py", line 2753, in get_stats Apr 17 16:38:25 user cinder-volume[84696]: ERROR oslo_service.periodic_task return self.driver.get_volume_stats(refresh=True) Apr 17 16:38:25 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/driver.py", line 666, in get_volume_stats Apr 17 16:38:25 user cinder-volume[84696]: ERROR oslo_service.periodic_task self._update_volume_stats() Apr 17 16:38:25 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 726, in _update_volume_stats Apr 17 16:38:25 user cinder-volume[84696]: ERROR oslo_service.periodic_task pool_stats = self.ra.get_pool_stats() Apr 17 16:38:25 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 897, in get_pool_stats Apr 17 16:38:25 user cinder-volume[84696]: ERROR oslo_service.periodic_task if not resp["error"] and resp["code"] == 200: Apr 17 16:38:25 user cinder-volume[84696]: ERROR oslo_service.periodic_task TypeError: 'NoneType' object is not subscriptable Apr 17 16:38:25 user cinder-volume[84696]: ERROR oslo_service.periodic_task Apr 17 16:38:25 user cinder-volume[84696]: DEBUG cinder.utils [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.971(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:38:25 user cinder-volume[84696]: DEBUG cinder.utils [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_480a94a0-d1c9-46c0-bf9f-c42f907b4b82/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:38:25 user cinder-volume[84696]: DEBUG cinder.utils [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.026(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:38:25 user cinder-volume[84696]: DEBUG cinder.utils [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_7384f51d-c3a2-4eea-a968-ec043946989f/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:38:25 user cinder-volume[84696]: WARNING oslo.service.loopingcall [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Function 'cinder.service.Service.periodic_tasks' run outlasted interval by 79.91 sec Apr 17 16:38:25 user cinder-volume[84696]: DEBUG oslo_service.periodic_task [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Running periodic task VolumeManager.publish_service_capabilities {{(pid=84696) run_periodic_tasks /usr/local/lib/python3.10/dist-packages/oslo_service/periodic_task.py:210}} Apr 17 16:38:25 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Updating volume stats {{(pid=84696) _update_volume_stats /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:724}} Apr 17 16:38:25 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Get pool Pool-0 fsprops {{(pid=84696) get_pool_stats /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:894}} Apr 17 16:38:25 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:38:25 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:38:29 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 3.054(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:38:29 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:38:29 user cinder-volume[84696]: DEBUG cinder.utils [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.047(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:38:29 user cinder-volume[84696]: DEBUG cinder.utils [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_480a94a0-d1c9-46c0-bf9f-c42f907b4b82/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:38:29 user cinder-volume[84696]: DEBUG cinder.utils [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.308(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:38:29 user cinder-volume[84696]: DEBUG cinder.utils [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:38:32 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.126(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:38:32 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:38:32 user cinder-volume[84696]: DEBUG cinder.utils [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.390(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:38:32 user cinder-volume[84696]: DEBUG cinder.utils [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_bf85ce23-1db3-4881-bd7a-b20b520718b8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:38:32 user cinder-volume[84696]: DEBUG cinder.utils [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.167(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:38:32 user cinder-volume[84696]: DEBUG cinder.utils [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_7384f51d-c3a2-4eea-a968-ec043946989f/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:38:34 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=10,cinder:UPDATE=4,cinder:INSERT=3 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:38:34 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=16,cinder:UPDATE=6,cinder:INSERT=3 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:38:35 user cinder-volume[84696]: DEBUG cinder.utils [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.185(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:38:35 user cinder-volume[84696]: DEBUG cinder.utils [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_480a94a0-d1c9-46c0-bf9f-c42f907b4b82/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:38:38 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.266(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:38:38 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:38:41 user cinder-volume[84696]: DEBUG cinder.utils [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.381(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:38:41 user cinder-volume[84696]: DEBUG cinder.utils [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_7384f51d-c3a2-4eea-a968-ec043946989f/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:38:44 user cinder-volume[84696]: DEBUG cinder.utils [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.401(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:38:44 user cinder-volume[84696]: DEBUG cinder.utils [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_480a94a0-d1c9-46c0-bf9f-c42f907b4b82/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:38:47 user cinder-volume[84696]: DEBUG cinder.utils [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.733(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:38:47 user cinder-volume[84696]: DEBUG cinder.utils [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:38:47 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.484(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:38:47 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:38:50 user cinder-volume[84696]: DEBUG cinder.utils [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.817(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:38:50 user cinder-volume[84696]: DEBUG cinder.utils [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_bf85ce23-1db3-4881-bd7a-b20b520718b8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:38:54 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=1 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:38:57 user cinder-volume[84696]: DEBUG cinder.utils [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 37.601(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:38:57 user cinder-volume[84696]: DEBUG cinder.utils [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_7384f51d-c3a2-4eea-a968-ec043946989f/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:39:00 user cinder-volume[84696]: DEBUG cinder.utils [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 37.625(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:39:00 user cinder-volume[84696]: DEBUG cinder.utils [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_480a94a0-d1c9-46c0-bf9f-c42f907b4b82/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:39:04 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 38.570(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:39:04 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:39:14 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=1 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:39:14 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=11,cinder:UPDATE=4 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:39:21 user cinder-volume[84696]: DEBUG cinder.utils [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 73.909(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:39:21 user cinder-volume[84696]: DEBUG cinder.utils [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:39:24 user cinder-volume[84696]: DEBUG cinder.utils [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 137.644(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:39:24 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_e1bb78ac-29e3-46df-9bbe-3b152e39b69a/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:39:24 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (375784c9-e555-42dd-8541-5e77aed44c93) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 16:39:24 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (a65b7553-8ac5-4db0-ae38-47b3a3f6d21d) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:39:24 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (a65b7553-8ac5-4db0-ae38-47b3a3f6d21d) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:39:24 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Flow 'volume_create_manager' (f7e5f1e4-b757-4576-8d8b-71f8f7d2a5fb) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 16:39:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 16:39:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 16:39:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 16:39:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 16:39:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 16:39:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 16:39:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 16:39:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 16:39:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 16:39:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 16:39:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 16:39:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 16:39:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 16:39:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 16:39:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 16:39:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 16:39:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 16:39:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 16:39:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 16:39:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 16:39:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 16:39:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 16:39:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 16:39:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 16:39:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:39:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 16:39:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:39:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 16:39:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:39:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 16:39:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:39:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 16:39:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:39:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 16:39:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:39:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 16:39:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:39:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 16:39:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:39:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:39:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 16:39:24 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 16:39:24 user cinder-volume[84696]: DEBUG cinder.utils [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 74.012(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:39:24 user cinder-volume[84696]: DEBUG cinder.utils [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_bf85ce23-1db3-4881-bd7a-b20b520718b8/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:39:24 user cinder-volume[84696]: DEBUG cinder.utils [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 137.503(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:39:24 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:39:24 user cinder-volume[84696]: INFO cinder.message.api [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Creating message record for request_id = req-26b6ca51-cba9-4420-975a-eec85c094fd9 Apr 17 16:39:24 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 16:39:24 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (948fcb6d-c840-4e23-b528-b6acbf95c0ab) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 16:39:24 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 16:39:24 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:34:47Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesMetadataTest-Volume-1367745540',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=e1bb78ac-29e3-46df-9bbe-3b152e39b69a,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='7f9b364e902a445082273ba68da887b4',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:37:07Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:37:07Z,use_quota=True,user_id='ad9fef0ffef84b1783a6075d92bb27c1',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 16:39:24 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:34:47Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesMetadataTest-Volume-1367745540',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=e1bb78ac-29e3-46df-9bbe-3b152e39b69a,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='7f9b364e902a445082273ba68da887b4',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:37:07Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:37:07Z,use_quota=True,user_id='ad9fef0ffef84b1783a6075d92bb27c1',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(e1bb78ac-29e3-46df-9bbe-3b152e39b69a),volume_id=e1bb78ac-29e3-46df-9bbe-3b152e39b69a,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': 'e1bb78ac-29e3-46df-9bbe-3b152e39b69a', 'volume_name': 'volume-e1bb78ac-29e3-46df-9bbe-3b152e39b69a', 'volume_size': 1}} Apr 17 16:39:24 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': 'e1bb78ac-29e3-46df-9bbe-3b152e39b69a', 'volume': {'id': 'e1bb78ac-29e3-46df-9bbe-3b152e39b69a', '_name_id': None, 'ec2_id': None, 'user_id': 'ad9fef0ffef84b1783a6075d92bb27c1', 'project_id': '7f9b364e902a445082273ba68da887b4', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-VolumesMetadataTest-Volume-1367745540', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'admin_metadata': {}, 'glance_metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T16:34:47.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-e1bb78ac-29e3-46df-9bbe-3b152e39b69a', 'name_id': 'e1bb78ac-29e3-46df-9bbe-3b152e39b69a', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumesMetadataTest-Volume-1367745540', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '7f9b364e902a445082273ba68da887b4', 'qos_specs': None, 'reservations': ['534d1a95-6891-4aa6-a81c-55f25671b440', '53d79034-9b12-4c87-ae06-d3cba4398e31', 'a0ad6a16-6f0e-4d9e-b17b-638e4e0563a9', '1856cb28-c4d3-49f0-94e1-95e881d39510'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'ad9fef0ffef84b1783a6075d92bb27c1', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumesMetadataTest-Volume-1367745540', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '7f9b364e902a445082273ba68da887b4', 'qos_specs': None, 'reservations': ['534d1a95-6891-4aa6-a81c-55f25671b440', '53d79034-9b12-4c87-ae06-d3cba4398e31', 'a0ad6a16-6f0e-4d9e-b17b-638e4e0563a9', '1856cb28-c4d3-49f0-94e1-95e881d39510'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'ad9fef0ffef84b1783a6075d92bb27c1', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': 'ad9fef0ffef84b1783a6075d92bb27c1', 'metadata': {}, 'qos_specs': None, 'retry': {'num_attempts': 2, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']}}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(e1bb78ac-29e3-46df-9bbe-3b152e39b69a),volume_id=e1bb78ac-29e3-46df-9bbe-3b152e39b69a,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:34:47Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesMetadataTest-Volume-1367745540',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=e1bb78ac-29e3-46df-9bbe-3b152e39b69a,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='7f9b364e902a445082273ba68da887b4',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:37:07Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:37:07Z,use_quota=True,user_id='ad9fef0ffef84b1783a6075d92bb27c1',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 16:39:24 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:34:47Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesMetadataTest-Volume-1367745540',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=e1bb78ac-29e3-46df-9bbe-3b152e39b69a,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='7f9b364e902a445082273ba68da887b4',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:37:07Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:37:07Z,use_quota=True,user_id='ad9fef0ffef84b1783a6075d92bb27c1',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:34:47Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesMetadataTest-Volume-1367745540',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=e1bb78ac-29e3-46df-9bbe-3b152e39b69a,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='7f9b364e902a445082273ba68da887b4',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:37:07Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:37:07Z,use_quota=True,user_id='ad9fef0ffef84b1783a6075d92bb27c1',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 16:39:24 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 16:39:24 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 16:39:24 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:39:24 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 16:39:24 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:39:24 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 16:39:24 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:39:24 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 16:39:24 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:39:24 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 16:39:24 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:39:24 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 16:39:24 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:39:24 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 16:39:24 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:39:24 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 16:39:24 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:39:24 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:39:24 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 16:39:24 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 16:39:24 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (948fcb6d-c840-4e23-b528-b6acbf95c0ab) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:39:24 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (948fcb6d-c840-4e23-b528-b6acbf95c0ab) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:39:24 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (2ff61726-4b32-49da-b668-0417e40a37ba) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:39:24 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (2ff61726-4b32-49da-b668-0417e40a37ba) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:39:24 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (ddb7f97d-6392-49bb-a463-a9f6b768e9c5) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:39:24 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (ddb7f97d-6392-49bb-a463-a9f6b768e9c5) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:39:24 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (2ed41617-9bca-42e8-b9c3-47cf3fe46f06) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:39:24 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Updating volume e1bb78ac-29e3-46df-9bbe-3b152e39b69a with {'scheduled_at': datetime.datetime(2023, 4, 17, 16, 39, 24, 819449), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 16:39:24 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Volume e1bb78ac-29e3-46df-9bbe-3b152e39b69a: re-scheduling SchedulerAPI.create_volume attempt 2 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 16:39:24 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Volume e1bb78ac-29e3-46df-9bbe-3b152e39b69a: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 16:39:24 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] deleating volume v_e1bb78ac-29e3-46df-9bbe-3b152e39b69a {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 16:39:24 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] get snapshots for volume v_e1bb78ac-29e3-46df-9bbe-3b152e39b69a {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 16:39:24 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_e1bb78ac-29e3-46df-9bbe-3b152e39b69a/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:39:24 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_e1bb78ac-29e3-46df-9bbe-3b152e39b69a/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:39:25 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Volume reschedule parameters: True retry: {'num_attempts': 3, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 16:39:25 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Flow 'volume_create_manager' (d9823830-d63b-46a9-9152-d096e7f2e564) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 16:39:25 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (2023bd15-4715-4c95-ad2a-d93c87115526) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:39:25 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (2023bd15-4715-4c95-ad2a-d93c87115526) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:34:47Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesMetadataTest-Volume-1367745540',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=e1bb78ac-29e3-46df-9bbe-3b152e39b69a,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='7f9b364e902a445082273ba68da887b4',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:39:25Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:39:25Z,use_quota=True,user_id='ad9fef0ffef84b1783a6075d92bb27c1',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:39:25 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (f379724d-30b2-4852-80c5-5786ff0b6bc2) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:39:25 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (f379724d-30b2-4852-80c5-5786ff0b6bc2) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:39:25 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (4f917c4f-5d28-49be-9934-e3803367fbe0) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:39:25 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (4f917c4f-5d28-49be-9934-e3803367fbe0) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': 'e1bb78ac-29e3-46df-9bbe-3b152e39b69a', 'volume_name': 'volume-e1bb78ac-29e3-46df-9bbe-3b152e39b69a', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:39:25 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (99b96e09-2b6e-4c5d-b9aa-915a5dc6ea41) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:39:25 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (99b96e09-2b6e-4c5d-b9aa-915a5dc6ea41) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:39:25 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (ac96c869-5ab1-4fc5-a4e5-412bbb4b5255) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:39:25 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Volume e1bb78ac-29e3-46df-9bbe-3b152e39b69a: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-e1bb78ac-29e3-46df-9bbe-3b152e39b69a', 'volume_size': 1} Apr 17 16:39:25 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] creating volume v_e1bb78ac-29e3-46df-9bbe-3b152e39b69a. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 16:39:25 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] create volume {'name': 'v_e1bb78ac-29e3-46df-9bbe-3b152e39b69a', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 16:39:25 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:39:25 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:39:27 user cinder-volume[84696]: DEBUG cinder.utils [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.948(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:39:27 user cinder-volume[84696]: DEBUG cinder.utils [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_e1bb78ac-29e3-46df-9bbe-3b152e39b69a/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:39:27 user cinder-volume[84696]: DEBUG cinder.utils [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.673(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:39:27 user cinder-volume[84696]: DEBUG cinder.utils [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:39:30 user cinder-volume[84696]: DEBUG cinder.utils [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.743(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:39:30 user cinder-volume[84696]: DEBUG cinder.utils [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:39:30 user cinder-volume[84696]: DEBUG cinder.utils [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.022(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:39:30 user cinder-volume[84696]: DEBUG cinder.utils [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_e1bb78ac-29e3-46df-9bbe-3b152e39b69a/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:39:30 user cinder-volume[84696]: DEBUG cinder.utils [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 70.917(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:39:30 user cinder-volume[84696]: DEBUG cinder.utils [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_7384f51d-c3a2-4eea-a968-ec043946989f/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:39:33 user cinder-volume[84696]: DEBUG cinder.utils [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 70.937(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:39:33 user cinder-volume[84696]: DEBUG cinder.utils [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_480a94a0-d1c9-46c0-bf9f-c42f907b4b82/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:39:37 user cinder-volume[84696]: DEBUG cinder.utils [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 11.887(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:39:37 user cinder-volume[84696]: DEBUG cinder.utils [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:39:37 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 71.021(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:39:37 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:39:37 user cinder-volume[84696]: DEBUG cinder.utils [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.168(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:39:37 user cinder-volume[84696]: DEBUG cinder.utils [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_e1bb78ac-29e3-46df-9bbe-3b152e39b69a/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:39:44 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=11,cinder:UPDATE=5,cinder:INSERT=1 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:39:44 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=9,cinder:INSERT=1,cinder:UPDATE=1 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:39:46 user cinder-volume[84696]: DEBUG cinder.utils [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.551(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:39:46 user cinder-volume[84696]: DEBUG cinder.utils [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:39:46 user cinder-volume[84696]: DEBUG cinder.utils [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.830(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:39:46 user cinder-volume[84696]: DEBUG cinder.utils [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_e1bb78ac-29e3-46df-9bbe-3b152e39b69a/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:40:02 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Volume reschedule parameters: True retry: {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 16:40:02 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Flow 'volume_create_manager' (c651d0af-8230-4a3d-8d42-a50cbdec74f4) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 16:40:02 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (b4c06049-69e3-48da-8917-f867b0e37fe6) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:40:02 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (b4c06049-69e3-48da-8917-f867b0e37fe6) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:40:02Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-SnapshotDataIntegrityTests-volume-609702161',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=b3e2b070-7732-4096-b59e-bdd77e430bbf,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='2a9277ba8f5f4f9abcbb70ee01b22d41',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:40:02Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:40:02Z,use_quota=True,user_id='d8c0fb3f72494a489e0230e83a85224a',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:40:02 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (34400a9b-2f05-4ca4-8a6e-2772e530ef0a) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:40:02 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (34400a9b-2f05-4ca4-8a6e-2772e530ef0a) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:40:02 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (8c1eb1dd-4437-49d3-a79c-d7a2218b6cda) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:40:02 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (8c1eb1dd-4437-49d3-a79c-d7a2218b6cda) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': 'b3e2b070-7732-4096-b59e-bdd77e430bbf', 'volume_name': 'volume-b3e2b070-7732-4096-b59e-bdd77e430bbf', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:40:02 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (4dd7213e-dbad-46c0-b2d5-877d4cecb642) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:40:02 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (4dd7213e-dbad-46c0-b2d5-877d4cecb642) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:40:02 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (64cda443-5775-4f2c-8bfd-7c425ef2d84b) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:40:02 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Volume b3e2b070-7732-4096-b59e-bdd77e430bbf: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-b3e2b070-7732-4096-b59e-bdd77e430bbf', 'volume_size': 1} Apr 17 16:40:02 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] creating volume v_b3e2b070-7732-4096-b59e-bdd77e430bbf. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 16:40:02 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] create volume {'name': 'v_b3e2b070-7732-4096-b59e-bdd77e430bbf', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 16:40:02 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:40:02 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:40:03 user cinder-volume[84696]: DEBUG cinder.utils [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 38.479(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:40:03 user cinder-volume[84696]: DEBUG cinder.utils [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:40:03 user cinder-volume[84696]: DEBUG cinder.utils [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 38.757(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:40:03 user cinder-volume[84696]: DEBUG cinder.utils [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_e1bb78ac-29e3-46df-9bbe-3b152e39b69a/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:40:03 user cinder-volume[84696]: DEBUG cinder.utils [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 1.544(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:40:03 user cinder-volume[84696]: DEBUG cinder.utils [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:40:06 user cinder-volume[84696]: DEBUG cinder.utils [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 4.614(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:40:06 user cinder-volume[84696]: DEBUG cinder.utils [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:40:12 user cinder-volume[84696]: DEBUG cinder.utils [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 10.757(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:40:12 user cinder-volume[84696]: DEBUG cinder.utils [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:40:14 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:UPDATE=3,cinder:SELECT=4 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:40:22 user cinder-volume[84696]: DEBUG cinder.utils [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 20.581(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:40:22 user cinder-volume[84696]: DEBUG cinder.utils [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:40:24 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=11,cinder:UPDATE=2 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:40:25 user cinder-volume[84696]: DEBUG cinder.utils [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 137.977(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:40:25 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:40:25 user cinder-volume[84696]: INFO cinder.message.api [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Creating message record for request_id = req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 Apr 17 16:40:25 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 16:40:25 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (536f62f3-5f49-453b-ba72-abca7ed9533e) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 16:40:25 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 16:40:25 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:38:07Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesActionsTest-Volume-825907983',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=f920a5f7-106a-4092-b193-c3d340118d77,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='44c0ea9c46d14578a12440ef37ec453e',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:38:08Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:38:08Z,use_quota=True,user_id='e97b56d3796f42fa89023ef8691f08ad',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 16:40:25 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:38:07Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesActionsTest-Volume-825907983',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=f920a5f7-106a-4092-b193-c3d340118d77,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='44c0ea9c46d14578a12440ef37ec453e',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:38:08Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:38:08Z,use_quota=True,user_id='e97b56d3796f42fa89023ef8691f08ad',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(f920a5f7-106a-4092-b193-c3d340118d77),volume_id=f920a5f7-106a-4092-b193-c3d340118d77,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': 'f920a5f7-106a-4092-b193-c3d340118d77', 'volume_name': 'volume-f920a5f7-106a-4092-b193-c3d340118d77', 'volume_size': 1}} Apr 17 16:40:25 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'retry': {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']}, 'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': 'f920a5f7-106a-4092-b193-c3d340118d77', 'volume': {'id': 'f920a5f7-106a-4092-b193-c3d340118d77', '_name_id': None, 'ec2_id': None, 'user_id': 'e97b56d3796f42fa89023ef8691f08ad', 'project_id': '44c0ea9c46d14578a12440ef37ec453e', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-VolumesActionsTest-Volume-825907983', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T16:38:07.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-f920a5f7-106a-4092-b193-c3d340118d77', 'name_id': 'f920a5f7-106a-4092-b193-c3d340118d77', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumesActionsTest-Volume-825907983', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '44c0ea9c46d14578a12440ef37ec453e', 'qos_specs': None, 'reservations': ['8593249f-303c-48ae-bc1d-dd163db6cbf5', '1ee0e36c-dfea-4a28-90e5-938c2128842b', 'bc62a1da-54b5-4aa4-9b18-530371c05e44', '7a4f51c4-76ae-478d-a8e4-ec1c3749b999'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'e97b56d3796f42fa89023ef8691f08ad', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumesActionsTest-Volume-825907983', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '44c0ea9c46d14578a12440ef37ec453e', 'qos_specs': None, 'reservations': ['8593249f-303c-48ae-bc1d-dd163db6cbf5', '1ee0e36c-dfea-4a28-90e5-938c2128842b', 'bc62a1da-54b5-4aa4-9b18-530371c05e44', '7a4f51c4-76ae-478d-a8e4-ec1c3749b999'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'e97b56d3796f42fa89023ef8691f08ad', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': 'e97b56d3796f42fa89023ef8691f08ad', 'metadata': {}, 'qos_specs': None}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(f920a5f7-106a-4092-b193-c3d340118d77),volume_id=f920a5f7-106a-4092-b193-c3d340118d77,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:38:07Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesActionsTest-Volume-825907983',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=f920a5f7-106a-4092-b193-c3d340118d77,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='44c0ea9c46d14578a12440ef37ec453e',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:38:08Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:38:08Z,use_quota=True,user_id='e97b56d3796f42fa89023ef8691f08ad',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 16:40:25 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:38:07Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesActionsTest-Volume-825907983',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=f920a5f7-106a-4092-b193-c3d340118d77,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='44c0ea9c46d14578a12440ef37ec453e',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:38:08Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:38:08Z,use_quota=True,user_id='e97b56d3796f42fa89023ef8691f08ad',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:38:07Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesActionsTest-Volume-825907983',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=f920a5f7-106a-4092-b193-c3d340118d77,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='44c0ea9c46d14578a12440ef37ec453e',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:38:08Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:38:08Z,use_quota=True,user_id='e97b56d3796f42fa89023ef8691f08ad',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 16:40:25 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 16:40:25 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 16:40:25 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:40:25 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 16:40:25 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:40:25 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 16:40:25 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:40:25 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 16:40:25 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:40:25 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 16:40:25 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:40:25 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 16:40:25 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:40:25 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 16:40:25 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:40:25 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 16:40:25 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:40:25 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:40:25 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 16:40:25 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 16:40:25 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (536f62f3-5f49-453b-ba72-abca7ed9533e) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:40:25 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (536f62f3-5f49-453b-ba72-abca7ed9533e) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:40:25 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (57f333c0-6361-4694-945e-ee4e24342c7f) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:40:25 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (57f333c0-6361-4694-945e-ee4e24342c7f) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:40:25 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (1aa69d64-a1bf-480e-b138-1b3aad39a823) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:40:25 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (1aa69d64-a1bf-480e-b138-1b3aad39a823) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:40:25 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (79bb0968-9afb-431e-bf9d-34f3714a89de) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:40:25 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Updating volume f920a5f7-106a-4092-b193-c3d340118d77 with {'scheduled_at': datetime.datetime(2023, 4, 17, 16, 40, 25, 794521), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 16:40:25 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Volume f920a5f7-106a-4092-b193-c3d340118d77: re-scheduling SchedulerAPI.create_volume attempt 1 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 16:40:25 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Volume f920a5f7-106a-4092-b193-c3d340118d77: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 16:40:25 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] deleating volume v_f920a5f7-106a-4092-b193-c3d340118d77 {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 16:40:25 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] get snapshots for volume v_f920a5f7-106a-4092-b193-c3d340118d77 {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 16:40:25 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_f920a5f7-106a-4092-b193-c3d340118d77/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:40:25 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_f920a5f7-106a-4092-b193-c3d340118d77/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:40:25 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Volume reschedule parameters: True retry: {'num_attempts': 2, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 16:40:25 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Flow 'volume_create_manager' (855299a7-9da1-451d-8700-ffbc82d4dece) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 16:40:25 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (2320cea9-ec63-478e-921b-d41314b2d5b6) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:40:25 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (2320cea9-ec63-478e-921b-d41314b2d5b6) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:38:07Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesActionsTest-Volume-825907983',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=f920a5f7-106a-4092-b193-c3d340118d77,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='44c0ea9c46d14578a12440ef37ec453e',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:40:26Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:40:26Z,use_quota=True,user_id='e97b56d3796f42fa89023ef8691f08ad',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:40:25 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (15105c46-2980-4697-b200-2ce244328fd2) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:40:25 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (15105c46-2980-4697-b200-2ce244328fd2) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:40:25 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (a1c8bf23-b7ce-4c2e-bf59-859797ad48f0) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:40:26 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (a1c8bf23-b7ce-4c2e-bf59-859797ad48f0) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': 'f920a5f7-106a-4092-b193-c3d340118d77', 'volume_name': 'volume-f920a5f7-106a-4092-b193-c3d340118d77', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:40:26 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (cc35a5d8-cee0-4c86-ae1f-3b0b0f83492c) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:40:26 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (cc35a5d8-cee0-4c86-ae1f-3b0b0f83492c) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:40:26 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (6568efaf-57eb-4659-a8a2-3291bfcaa4ac) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:40:26 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Volume f920a5f7-106a-4092-b193-c3d340118d77: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-f920a5f7-106a-4092-b193-c3d340118d77', 'volume_size': 1} Apr 17 16:40:26 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] creating volume v_f920a5f7-106a-4092-b193-c3d340118d77. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 16:40:26 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] create volume {'name': 'v_f920a5f7-106a-4092-b193-c3d340118d77', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 16:40:26 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:40:26 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:40:28 user cinder-volume[84696]: DEBUG cinder.utils [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 3.074(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:40:28 user cinder-volume[84696]: DEBUG cinder.utils [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_f920a5f7-106a-4092-b193-c3d340118d77/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:40:28 user cinder-volume[84696]: DEBUG cinder.utils [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 138.158(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:40:28 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_bf85ce23-1db3-4881-bd7a-b20b520718b8/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:40:28 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (44f4fcbb-0a2d-4361-9c98-290e952f34ed) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 16:40:28 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (333215a2-6f24-4dd6-ba6b-cb4928fc7188) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:40:28 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (333215a2-6f24-4dd6-ba6b-cb4928fc7188) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:40:28 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Flow 'volume_create_manager' (096bcc29-7419-4518-b3fd-22ecaaa1421c) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 16:40:28 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-208fb94c-86af-4459-a279-1c8be9781cb3 tempest-CreateVolumesFromSnapshotTest-1535745892 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 16:40:28 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 16:40:28 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 16:40:28 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 16:40:28 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 16:40:28 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 16:40:28 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 16:40:28 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 16:40:28 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 16:40:28 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 16:40:28 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 16:40:28 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 16:40:28 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 16:40:28 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 16:40:28 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 16:40:28 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 16:40:28 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 16:40:28 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 16:40:28 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 16:40:28 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 16:40:28 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 16:40:28 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 16:40:28 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 16:40:28 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 16:40:28 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:40:28 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 16:40:28 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:40:28 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 16:40:28 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:40:28 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 16:40:28 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:40:28 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 16:40:28 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:40:28 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 16:40:28 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:40:28 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 16:40:28 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:40:28 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 16:40:28 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:40:28 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:40:28 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 16:40:28 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 16:40:28 user cinder-volume[84696]: DEBUG cinder.utils [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.905(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:40:28 user cinder-volume[84696]: DEBUG cinder.utils [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:40:31 user cinder-volume[84696]: DEBUG cinder.utils [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.145(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:40:31 user cinder-volume[84696]: DEBUG cinder.utils [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_f920a5f7-106a-4092-b193-c3d340118d77/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:40:31 user cinder-volume[84696]: DEBUG cinder.utils [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.949(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:40:31 user cinder-volume[84696]: DEBUG cinder.utils [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:40:35 user cinder-volume[84696]: DEBUG cinder.utils [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 135.073(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:40:35 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_7384f51d-c3a2-4eea-a968-ec043946989f/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:40:35 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (d7f3d394-9859-4e12-aa4a-64ce16c48c98) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 16:40:35 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (151796e9-3a94-4863-81cf-e6d865b78c19) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:40:35 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (151796e9-3a94-4863-81cf-e6d865b78c19) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:40:35 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Flow 'volume_create_manager' (47754890-05fe-49a0-88aa-1d8556959dce) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 16:40:35 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-3219fc38-5e4d-4f3e-8600-f93c79e36e17 tempest-CinderUnicodeTest-617908546 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 16:40:35 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 16:40:35 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 16:40:35 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 16:40:35 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 16:40:35 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 16:40:35 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 16:40:35 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 16:40:35 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 16:40:35 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 16:40:35 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 16:40:35 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 16:40:35 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 16:40:35 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 16:40:35 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 16:40:35 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 16:40:35 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 16:40:35 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 16:40:35 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 16:40:35 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 16:40:35 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 16:40:35 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 16:40:35 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 16:40:35 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 16:40:35 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:40:35 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 16:40:35 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:40:35 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 16:40:35 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:40:35 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 16:40:35 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:40:35 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 16:40:35 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:40:35 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 16:40:35 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:40:35 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 16:40:35 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:40:35 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 16:40:35 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:40:35 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:40:35 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 16:40:35 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 16:40:38 user cinder-volume[84696]: DEBUG cinder.utils [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 73.252(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:40:38 user cinder-volume[84696]: DEBUG cinder.utils [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_e1bb78ac-29e3-46df-9bbe-3b152e39b69a/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:40:38 user cinder-volume[84696]: DEBUG cinder.utils [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 72.976(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:40:38 user cinder-volume[84696]: DEBUG cinder.utils [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:40:38 user cinder-volume[84696]: DEBUG cinder.utils [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 135.100(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:40:38 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_480a94a0-d1c9-46c0-bf9f-c42f907b4b82/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:40:38 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (653462dc-cbb2-4a03-b331-cb3dce7448b6) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 16:40:38 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (35dbda95-7e26-4948-81d5-b79d1b91220d) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:40:38 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (35dbda95-7e26-4948-81d5-b79d1b91220d) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:40:38 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Flow 'volume_create_manager' (c4715c3c-504e-41ce-8a1b-63b6d2dc8538) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 16:40:38 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-f9284730-0ae4-46e6-8ee4-ca54c9c1c03b tempest-VolumeQuotasAdminTestJSON-623147042 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 16:40:38 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 16:40:38 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 16:40:38 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 16:40:38 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 16:40:38 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 16:40:38 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 16:40:38 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 16:40:38 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 16:40:38 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 16:40:38 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 16:40:38 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 16:40:38 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 16:40:38 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 16:40:38 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 16:40:38 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 16:40:38 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 16:40:38 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 16:40:38 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 16:40:38 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 16:40:38 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 16:40:38 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 16:40:38 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 16:40:38 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 16:40:38 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:40:38 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 16:40:38 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:40:38 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 16:40:38 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:40:38 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 16:40:38 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:40:38 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 16:40:38 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:40:38 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 16:40:38 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:40:38 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 16:40:38 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:40:38 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 16:40:38 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:40:38 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:40:38 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 16:40:38 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 16:40:38 user cinder-volume[84696]: DEBUG cinder.utils [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.312(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:40:38 user cinder-volume[84696]: DEBUG cinder.utils [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_f920a5f7-106a-4092-b193-c3d340118d77/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:40:38 user cinder-volume[84696]: DEBUG cinder.utils [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.115(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:40:38 user cinder-volume[84696]: DEBUG cinder.utils [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:40:41 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 135.178(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:40:41 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:40:41 user cinder-volume[84696]: ERROR oslo_service.periodic_task [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Error during VolumeManager.publish_service_capabilities: TypeError: 'NoneType' object is not subscriptable Apr 17 16:40:41 user cinder-volume[84696]: ERROR oslo_service.periodic_task Traceback (most recent call last): Apr 17 16:40:41 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/usr/local/lib/python3.10/dist-packages/oslo_service/periodic_task.py", line 216, in run_periodic_tasks Apr 17 16:40:41 user cinder-volume[84696]: ERROR oslo_service.periodic_task task(self, context) Apr 17 16:40:41 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/manager.py", line 2876, in publish_service_capabilities Apr 17 16:40:41 user cinder-volume[84696]: ERROR oslo_service.periodic_task self._report_driver_status(context) Apr 17 16:40:41 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/manager.py", line 2755, in _report_driver_status Apr 17 16:40:41 user cinder-volume[84696]: ERROR oslo_service.periodic_task volume_stats = get_stats() Apr 17 16:40:41 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/usr/local/lib/python3.10/dist-packages/oslo_utils/timeutils.py", line 319, in wrapper Apr 17 16:40:41 user cinder-volume[84696]: ERROR oslo_service.periodic_task result = func(*args, **kwargs) Apr 17 16:40:41 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/manager.py", line 2753, in get_stats Apr 17 16:40:41 user cinder-volume[84696]: ERROR oslo_service.periodic_task return self.driver.get_volume_stats(refresh=True) Apr 17 16:40:41 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/driver.py", line 666, in get_volume_stats Apr 17 16:40:41 user cinder-volume[84696]: ERROR oslo_service.periodic_task self._update_volume_stats() Apr 17 16:40:41 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 726, in _update_volume_stats Apr 17 16:40:41 user cinder-volume[84696]: ERROR oslo_service.periodic_task pool_stats = self.ra.get_pool_stats() Apr 17 16:40:41 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 897, in get_pool_stats Apr 17 16:40:41 user cinder-volume[84696]: ERROR oslo_service.periodic_task if not resp["error"] and resp["code"] == 200: Apr 17 16:40:41 user cinder-volume[84696]: ERROR oslo_service.periodic_task TypeError: 'NoneType' object is not subscriptable Apr 17 16:40:41 user cinder-volume[84696]: ERROR oslo_service.periodic_task Apr 17 16:40:41 user cinder-volume[84696]: DEBUG cinder.utils [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.112(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:40:41 user cinder-volume[84696]: DEBUG cinder.utils [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:40:41 user cinder-volume[84696]: WARNING oslo.service.loopingcall [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Function 'cinder.service.Service.periodic_tasks' run outlasted interval by 75.19 sec Apr 17 16:40:41 user cinder-volume[84696]: DEBUG oslo_service.periodic_task [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Running periodic task VolumeManager.publish_service_capabilities {{(pid=84696) run_periodic_tasks /usr/local/lib/python3.10/dist-packages/oslo_service/periodic_task.py:210}} Apr 17 16:40:41 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Updating volume stats {{(pid=84696) _update_volume_stats /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:724}} Apr 17 16:40:41 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Get pool Pool-0 fsprops {{(pid=84696) get_pool_stats /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:894}} Apr 17 16:40:41 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:40:41 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:40:44 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 3.063(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:40:44 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:40:47 user cinder-volume[84696]: DEBUG cinder.utils [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.308(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:40:47 user cinder-volume[84696]: DEBUG cinder.utils [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:40:47 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.138(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:40:47 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:40:47 user cinder-volume[84696]: DEBUG cinder.utils [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.509(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:40:47 user cinder-volume[84696]: DEBUG cinder.utils [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_f920a5f7-106a-4092-b193-c3d340118d77/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:40:53 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.475(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:40:53 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:40:54 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=8,cinder:UPDATE=3,cinder:INSERT=1 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:40:54 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=10,cinder:INSERT=1,cinder:UPDATE=3 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:41:03 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 22.555(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:41:03 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:41:03 user cinder-volume[84696]: DEBUG cinder.utils [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 37.927(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:41:03 user cinder-volume[84696]: DEBUG cinder.utils [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_f920a5f7-106a-4092-b193-c3d340118d77/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:41:03 user cinder-volume[84696]: DEBUG cinder.utils [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 37.731(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:41:03 user cinder-volume[84696]: DEBUG cinder.utils [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:41:13 user cinder-volume[84696]: DEBUG cinder.utils [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 71.625(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:41:13 user cinder-volume[84696]: DEBUG cinder.utils [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:41:14 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=3,cinder:UPDATE=2 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:41:20 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.511(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:41:20 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:41:34 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=1 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:41:34 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=8,cinder:UPDATE=2 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:41:37 user cinder-volume[84696]: DEBUG cinder.utils [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 71.937(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:41:37 user cinder-volume[84696]: DEBUG cinder.utils [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_f920a5f7-106a-4092-b193-c3d340118d77/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:41:37 user cinder-volume[84696]: DEBUG cinder.utils [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 71.741(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:41:37 user cinder-volume[84696]: DEBUG cinder.utils [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:41:43 user cinder-volume[84696]: DEBUG cinder.utils [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 139.044(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:41:43 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_e1bb78ac-29e3-46df-9bbe-3b152e39b69a/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:41:43 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (2ed41617-9bca-42e8-b9c3-47cf3fe46f06) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 16:41:43 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (4e365595-98e0-4ea9-9930-0ae17e7fa1c8) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:41:43 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (4e365595-98e0-4ea9-9930-0ae17e7fa1c8) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:41:43 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Flow 'volume_create_manager' (4cbbbbfc-077c-49f9-9e47-ffaefe723479) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 16:41:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 16:41:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 16:41:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 16:41:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 16:41:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 16:41:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 16:41:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 16:41:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 16:41:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 16:41:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 16:41:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 16:41:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 16:41:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 16:41:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 16:41:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 16:41:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 16:41:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 16:41:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 16:41:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 16:41:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 16:41:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 16:41:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 16:41:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 16:41:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 16:41:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:41:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 16:41:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:41:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 16:41:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:41:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 16:41:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:41:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 16:41:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:41:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 16:41:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:41:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 16:41:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:41:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 16:41:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:41:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:41:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 16:41:43 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 16:41:43 user cinder-volume[84696]: DEBUG cinder.utils [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 138.789(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:41:43 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:41:43 user cinder-volume[84696]: INFO cinder.message.api [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Creating message record for request_id = req-26b6ca51-cba9-4420-975a-eec85c094fd9 Apr 17 16:41:43 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 16:41:43 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (ac96c869-5ab1-4fc5-a4e5-412bbb4b5255) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 16:41:43 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 16:41:43 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:34:47Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesMetadataTest-Volume-1367745540',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=e1bb78ac-29e3-46df-9bbe-3b152e39b69a,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='7f9b364e902a445082273ba68da887b4',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:39:25Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:39:25Z,use_quota=True,user_id='ad9fef0ffef84b1783a6075d92bb27c1',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 16:41:43 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:34:47Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesMetadataTest-Volume-1367745540',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=e1bb78ac-29e3-46df-9bbe-3b152e39b69a,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='7f9b364e902a445082273ba68da887b4',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:39:25Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:39:25Z,use_quota=True,user_id='ad9fef0ffef84b1783a6075d92bb27c1',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(e1bb78ac-29e3-46df-9bbe-3b152e39b69a),volume_id=e1bb78ac-29e3-46df-9bbe-3b152e39b69a,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': 'e1bb78ac-29e3-46df-9bbe-3b152e39b69a', 'volume_name': 'volume-e1bb78ac-29e3-46df-9bbe-3b152e39b69a', 'volume_size': 1}} Apr 17 16:41:43 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': 'e1bb78ac-29e3-46df-9bbe-3b152e39b69a', 'volume': {'id': 'e1bb78ac-29e3-46df-9bbe-3b152e39b69a', '_name_id': None, 'ec2_id': None, 'user_id': 'ad9fef0ffef84b1783a6075d92bb27c1', 'project_id': '7f9b364e902a445082273ba68da887b4', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-VolumesMetadataTest-Volume-1367745540', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'admin_metadata': {}, 'glance_metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T16:34:47.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-e1bb78ac-29e3-46df-9bbe-3b152e39b69a', 'name_id': 'e1bb78ac-29e3-46df-9bbe-3b152e39b69a', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumesMetadataTest-Volume-1367745540', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '7f9b364e902a445082273ba68da887b4', 'qos_specs': None, 'reservations': ['534d1a95-6891-4aa6-a81c-55f25671b440', '53d79034-9b12-4c87-ae06-d3cba4398e31', 'a0ad6a16-6f0e-4d9e-b17b-638e4e0563a9', '1856cb28-c4d3-49f0-94e1-95e881d39510'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'ad9fef0ffef84b1783a6075d92bb27c1', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumesMetadataTest-Volume-1367745540', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '7f9b364e902a445082273ba68da887b4', 'qos_specs': None, 'reservations': ['534d1a95-6891-4aa6-a81c-55f25671b440', '53d79034-9b12-4c87-ae06-d3cba4398e31', 'a0ad6a16-6f0e-4d9e-b17b-638e4e0563a9', '1856cb28-c4d3-49f0-94e1-95e881d39510'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'ad9fef0ffef84b1783a6075d92bb27c1', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': 'ad9fef0ffef84b1783a6075d92bb27c1', 'metadata': {}, 'qos_specs': None, 'retry': {'num_attempts': 3, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']}}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(e1bb78ac-29e3-46df-9bbe-3b152e39b69a),volume_id=e1bb78ac-29e3-46df-9bbe-3b152e39b69a,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:34:47Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesMetadataTest-Volume-1367745540',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=e1bb78ac-29e3-46df-9bbe-3b152e39b69a,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='7f9b364e902a445082273ba68da887b4',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:39:25Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:39:25Z,use_quota=True,user_id='ad9fef0ffef84b1783a6075d92bb27c1',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 16:41:43 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:34:47Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesMetadataTest-Volume-1367745540',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=e1bb78ac-29e3-46df-9bbe-3b152e39b69a,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='7f9b364e902a445082273ba68da887b4',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:39:25Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:39:25Z,use_quota=True,user_id='ad9fef0ffef84b1783a6075d92bb27c1',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:34:47Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesMetadataTest-Volume-1367745540',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=e1bb78ac-29e3-46df-9bbe-3b152e39b69a,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='7f9b364e902a445082273ba68da887b4',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:39:25Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:39:25Z,use_quota=True,user_id='ad9fef0ffef84b1783a6075d92bb27c1',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 16:41:43 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 16:41:43 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 16:41:43 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:41:43 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 16:41:43 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:41:43 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 16:41:43 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:41:43 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 16:41:43 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:41:43 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 16:41:43 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:41:43 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 16:41:43 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:41:43 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 16:41:43 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:41:43 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 16:41:43 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:41:43 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:41:43 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 16:41:43 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 16:41:43 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (ac96c869-5ab1-4fc5-a4e5-412bbb4b5255) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:41:43 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (ac96c869-5ab1-4fc5-a4e5-412bbb4b5255) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:41:43 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (99b96e09-2b6e-4c5d-b9aa-915a5dc6ea41) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:41:43 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (99b96e09-2b6e-4c5d-b9aa-915a5dc6ea41) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:41:44 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (4f917c4f-5d28-49be-9934-e3803367fbe0) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:41:44 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (4f917c4f-5d28-49be-9934-e3803367fbe0) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:41:44 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (f379724d-30b2-4852-80c5-5786ff0b6bc2) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:41:44 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Updating volume e1bb78ac-29e3-46df-9bbe-3b152e39b69a with {'scheduled_at': datetime.datetime(2023, 4, 17, 16, 41, 44, 12365), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 16:41:44 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Volume e1bb78ac-29e3-46df-9bbe-3b152e39b69a: re-scheduling SchedulerAPI.create_volume attempt 3 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 16:41:44 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Volume e1bb78ac-29e3-46df-9bbe-3b152e39b69a: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 16:41:44 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] deleating volume v_e1bb78ac-29e3-46df-9bbe-3b152e39b69a {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 16:41:44 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] get snapshots for volume v_e1bb78ac-29e3-46df-9bbe-3b152e39b69a {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 16:41:44 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_e1bb78ac-29e3-46df-9bbe-3b152e39b69a/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:41:44 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_e1bb78ac-29e3-46df-9bbe-3b152e39b69a/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:41:46 user cinder-volume[84696]: DEBUG cinder.utils [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.931(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:41:46 user cinder-volume[84696]: DEBUG cinder.utils [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_e1bb78ac-29e3-46df-9bbe-3b152e39b69a/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:41:50 user cinder-volume[84696]: DEBUG cinder.utils [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.007(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:41:50 user cinder-volume[84696]: DEBUG cinder.utils [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_e1bb78ac-29e3-46df-9bbe-3b152e39b69a/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:41:53 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 71.927(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:41:53 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:41:54 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=3,cinder:UPDATE=3,cinder:INSERT=1 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:41:56 user cinder-volume[84696]: DEBUG cinder.utils [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.147(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:41:56 user cinder-volume[84696]: DEBUG cinder.utils [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_e1bb78ac-29e3-46df-9bbe-3b152e39b69a/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:42:04 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=1 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:42:05 user cinder-volume[84696]: DEBUG cinder.utils [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.683(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:42:05 user cinder-volume[84696]: DEBUG cinder.utils [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_e1bb78ac-29e3-46df-9bbe-3b152e39b69a/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:42:14 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=12,cinder:INSERT=1,cinder:UPDATE=3 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:42:19 user cinder-volume[84696]: DEBUG cinder.utils [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 137.669(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:42:19 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:42:19 user cinder-volume[84696]: INFO cinder.message.api [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Creating message record for request_id = req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f Apr 17 16:42:19 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 16:42:19 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (64cda443-5775-4f2c-8bfd-7c425ef2d84b) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 16:42:19 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 16:42:19 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:40:02Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-SnapshotDataIntegrityTests-volume-609702161',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=b3e2b070-7732-4096-b59e-bdd77e430bbf,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='2a9277ba8f5f4f9abcbb70ee01b22d41',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:40:02Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:40:02Z,use_quota=True,user_id='d8c0fb3f72494a489e0230e83a85224a',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 16:42:19 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:40:02Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-SnapshotDataIntegrityTests-volume-609702161',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=b3e2b070-7732-4096-b59e-bdd77e430bbf,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='2a9277ba8f5f4f9abcbb70ee01b22d41',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:40:02Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:40:02Z,use_quota=True,user_id='d8c0fb3f72494a489e0230e83a85224a',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(b3e2b070-7732-4096-b59e-bdd77e430bbf),volume_id=b3e2b070-7732-4096-b59e-bdd77e430bbf,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': 'b3e2b070-7732-4096-b59e-bdd77e430bbf', 'volume_name': 'volume-b3e2b070-7732-4096-b59e-bdd77e430bbf', 'volume_size': 1}} Apr 17 16:42:19 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'retry': {'num_attempts': 1, 'backends': ['user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2']}, 'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': 'b3e2b070-7732-4096-b59e-bdd77e430bbf', 'volume': {'id': 'b3e2b070-7732-4096-b59e-bdd77e430bbf', '_name_id': None, 'ec2_id': None, 'user_id': 'd8c0fb3f72494a489e0230e83a85224a', 'project_id': '2a9277ba8f5f4f9abcbb70ee01b22d41', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-SnapshotDataIntegrityTests-volume-609702161', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T16:40:01.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-b3e2b070-7732-4096-b59e-bdd77e430bbf', 'name_id': 'b3e2b070-7732-4096-b59e-bdd77e430bbf', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-SnapshotDataIntegrityTests-volume-609702161', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '2a9277ba8f5f4f9abcbb70ee01b22d41', 'qos_specs': None, 'reservations': ['9c418056-b486-4dc9-adfa-64bbbb03ad8f', '0276aa33-a2e6-40f5-a1a7-4d1f5e3cc072', '14ca4b2e-e473-45b8-afb7-461c0aa7879f', '446dfc86-c241-4830-b2f2-a41012a6f708'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'd8c0fb3f72494a489e0230e83a85224a', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-SnapshotDataIntegrityTests-volume-609702161', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '2a9277ba8f5f4f9abcbb70ee01b22d41', 'qos_specs': None, 'reservations': ['9c418056-b486-4dc9-adfa-64bbbb03ad8f', '0276aa33-a2e6-40f5-a1a7-4d1f5e3cc072', '14ca4b2e-e473-45b8-afb7-461c0aa7879f', '446dfc86-c241-4830-b2f2-a41012a6f708'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'd8c0fb3f72494a489e0230e83a85224a', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': 'd8c0fb3f72494a489e0230e83a85224a', 'metadata': {}, 'qos_specs': None}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(b3e2b070-7732-4096-b59e-bdd77e430bbf),volume_id=b3e2b070-7732-4096-b59e-bdd77e430bbf,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:40:02Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-SnapshotDataIntegrityTests-volume-609702161',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=b3e2b070-7732-4096-b59e-bdd77e430bbf,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='2a9277ba8f5f4f9abcbb70ee01b22d41',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:40:02Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:40:02Z,use_quota=True,user_id='d8c0fb3f72494a489e0230e83a85224a',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 16:42:19 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:40:02Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-SnapshotDataIntegrityTests-volume-609702161',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=b3e2b070-7732-4096-b59e-bdd77e430bbf,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='2a9277ba8f5f4f9abcbb70ee01b22d41',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:40:02Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:40:02Z,use_quota=True,user_id='d8c0fb3f72494a489e0230e83a85224a',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:40:02Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-SnapshotDataIntegrityTests-volume-609702161',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=b3e2b070-7732-4096-b59e-bdd77e430bbf,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='2a9277ba8f5f4f9abcbb70ee01b22d41',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:40:02Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:40:02Z,use_quota=True,user_id='d8c0fb3f72494a489e0230e83a85224a',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 16:42:19 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 16:42:19 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 16:42:19 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:42:19 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 16:42:19 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:42:19 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 16:42:19 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:42:19 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 16:42:19 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:42:19 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 16:42:19 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:42:19 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 16:42:19 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:42:19 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 16:42:19 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:42:19 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 16:42:19 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:42:19 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:42:19 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 16:42:19 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 16:42:19 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (64cda443-5775-4f2c-8bfd-7c425ef2d84b) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:42:19 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (64cda443-5775-4f2c-8bfd-7c425ef2d84b) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:42:19 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (4dd7213e-dbad-46c0-b2d5-877d4cecb642) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:42:19 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (4dd7213e-dbad-46c0-b2d5-877d4cecb642) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:42:19 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (8c1eb1dd-4437-49d3-a79c-d7a2218b6cda) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:42:19 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (8c1eb1dd-4437-49d3-a79c-d7a2218b6cda) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:42:19 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (34400a9b-2f05-4ca4-8a6e-2772e530ef0a) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:42:19 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Updating volume b3e2b070-7732-4096-b59e-bdd77e430bbf with {'scheduled_at': datetime.datetime(2023, 4, 17, 16, 42, 19, 803653), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 16:42:19 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Volume b3e2b070-7732-4096-b59e-bdd77e430bbf: re-scheduling SchedulerAPI.create_volume attempt 1 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 16:42:19 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Volume b3e2b070-7732-4096-b59e-bdd77e430bbf: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 16:42:19 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] deleating volume v_b3e2b070-7732-4096-b59e-bdd77e430bbf {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 16:42:19 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] get snapshots for volume v_b3e2b070-7732-4096-b59e-bdd77e430bbf {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 16:42:19 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_b3e2b070-7732-4096-b59e-bdd77e430bbf/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:42:19 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_b3e2b070-7732-4096-b59e-bdd77e430bbf/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:42:19 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Volume reschedule parameters: True retry: {'num_attempts': 2, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 16:42:19 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Flow 'volume_create_manager' (97d319ba-952c-4da9-8b89-36a4edc1926f) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 16:42:19 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (ae61ed63-eb54-46d0-ad92-2dee46564540) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:42:19 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (ae61ed63-eb54-46d0-ad92-2dee46564540) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:40:02Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-SnapshotDataIntegrityTests-volume-609702161',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=b3e2b070-7732-4096-b59e-bdd77e430bbf,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='2a9277ba8f5f4f9abcbb70ee01b22d41',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:42:20Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:42:20Z,use_quota=True,user_id='d8c0fb3f72494a489e0230e83a85224a',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:42:20 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (074dded1-3aa3-4be8-9800-ea54911c3cb6) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:42:20 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (074dded1-3aa3-4be8-9800-ea54911c3cb6) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:42:20 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (8314f1b3-6581-4d06-a7cc-7b970b0a7e10) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:42:20 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (8314f1b3-6581-4d06-a7cc-7b970b0a7e10) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': 'b3e2b070-7732-4096-b59e-bdd77e430bbf', 'volume_name': 'volume-b3e2b070-7732-4096-b59e-bdd77e430bbf', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:42:20 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (7b5704ce-896b-47e6-9281-526f07035dbf) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:42:20 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (7b5704ce-896b-47e6-9281-526f07035dbf) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:42:20 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (d537c8cd-a546-48ba-9022-eb6d386d101e) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:42:20 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Volume b3e2b070-7732-4096-b59e-bdd77e430bbf: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-b3e2b070-7732-4096-b59e-bdd77e430bbf', 'volume_size': 1} Apr 17 16:42:20 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] creating volume v_b3e2b070-7732-4096-b59e-bdd77e430bbf. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 16:42:20 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] create volume {'name': 'v_b3e2b070-7732-4096-b59e-bdd77e430bbf', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 16:42:20 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:42:20 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:42:22 user cinder-volume[84696]: DEBUG cinder.utils [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.981(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:42:22 user cinder-volume[84696]: DEBUG cinder.utils [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_b3e2b070-7732-4096-b59e-bdd77e430bbf/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:42:22 user cinder-volume[84696]: DEBUG cinder.utils [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 38.773(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:42:22 user cinder-volume[84696]: DEBUG cinder.utils [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_e1bb78ac-29e3-46df-9bbe-3b152e39b69a/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:42:22 user cinder-volume[84696]: DEBUG cinder.utils [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.788(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:42:22 user cinder-volume[84696]: DEBUG cinder.utils [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:42:25 user cinder-volume[84696]: DEBUG cinder.utils [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.053(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:42:25 user cinder-volume[84696]: DEBUG cinder.utils [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_b3e2b070-7732-4096-b59e-bdd77e430bbf/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:42:25 user cinder-volume[84696]: DEBUG cinder.utils [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.859(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:42:25 user cinder-volume[84696]: DEBUG cinder.utils [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:42:29 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=7,cinder:INSERT=1,cinder:UPDATE=1 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:42:32 user cinder-volume[84696]: DEBUG cinder.utils [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.905(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:42:32 user cinder-volume[84696]: DEBUG cinder.utils [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_b3e2b070-7732-4096-b59e-bdd77e430bbf/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:42:32 user cinder-volume[84696]: DEBUG cinder.utils [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.711(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:42:32 user cinder-volume[84696]: DEBUG cinder.utils [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:42:34 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=8,cinder:UPDATE=4,cinder:INSERT=1 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:42:42 user cinder-volume[84696]: DEBUG cinder.utils [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 22.981(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:42:42 user cinder-volume[84696]: DEBUG cinder.utils [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_b3e2b070-7732-4096-b59e-bdd77e430bbf/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:42:42 user cinder-volume[84696]: DEBUG cinder.utils [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 22.787(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:42:42 user cinder-volume[84696]: DEBUG cinder.utils [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:42:42 user cinder-volume[84696]: DEBUG cinder.utils [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 136.798(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:42:42 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:42:42 user cinder-volume[84696]: INFO cinder.message.api [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Creating message record for request_id = req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 Apr 17 16:42:42 user cinder-volume[84696]: DEBUG cinder.utils [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 136.999(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:42:42 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_f920a5f7-106a-4092-b193-c3d340118d77/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:42:42 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (79bb0968-9afb-431e-bf9d-34f3714a89de) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 16:42:42 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (9521aa81-709c-4ed8-922f-567003cc6868) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:42:42 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (9521aa81-709c-4ed8-922f-567003cc6868) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:42:42 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Flow 'volume_create_manager' (a603acaf-a9be-4d41-8c24-10a6ca99b673) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 16:42:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 16:42:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 16:42:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 16:42:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 16:42:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 16:42:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 16:42:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 16:42:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 16:42:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 16:42:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 16:42:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 16:42:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 16:42:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 16:42:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 16:42:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 16:42:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 16:42:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 16:42:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 16:42:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 16:42:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 16:42:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 16:42:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 16:42:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 16:42:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 16:42:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:42:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 16:42:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:42:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 16:42:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:42:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 16:42:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:42:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 16:42:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:42:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 16:42:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:42:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 16:42:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:42:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 16:42:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:42:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:42:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 16:42:42 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 16:42:42 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 16:42:43 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (6568efaf-57eb-4659-a8a2-3291bfcaa4ac) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 16:42:43 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 16:42:43 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:38:07Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesActionsTest-Volume-825907983',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=f920a5f7-106a-4092-b193-c3d340118d77,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='44c0ea9c46d14578a12440ef37ec453e',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:40:26Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:40:26Z,use_quota=True,user_id='e97b56d3796f42fa89023ef8691f08ad',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 16:42:43 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:38:07Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesActionsTest-Volume-825907983',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=f920a5f7-106a-4092-b193-c3d340118d77,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='44c0ea9c46d14578a12440ef37ec453e',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:40:26Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:40:26Z,use_quota=True,user_id='e97b56d3796f42fa89023ef8691f08ad',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(f920a5f7-106a-4092-b193-c3d340118d77),volume_id=f920a5f7-106a-4092-b193-c3d340118d77,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': 'f920a5f7-106a-4092-b193-c3d340118d77', 'volume_name': 'volume-f920a5f7-106a-4092-b193-c3d340118d77', 'volume_size': 1}} Apr 17 16:42:43 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': 'f920a5f7-106a-4092-b193-c3d340118d77', 'volume': {'id': 'f920a5f7-106a-4092-b193-c3d340118d77', '_name_id': None, 'ec2_id': None, 'user_id': 'e97b56d3796f42fa89023ef8691f08ad', 'project_id': '44c0ea9c46d14578a12440ef37ec453e', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-VolumesActionsTest-Volume-825907983', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'admin_metadata': {}, 'glance_metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T16:38:07.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-f920a5f7-106a-4092-b193-c3d340118d77', 'name_id': 'f920a5f7-106a-4092-b193-c3d340118d77', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumesActionsTest-Volume-825907983', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '44c0ea9c46d14578a12440ef37ec453e', 'qos_specs': None, 'reservations': ['8593249f-303c-48ae-bc1d-dd163db6cbf5', '1ee0e36c-dfea-4a28-90e5-938c2128842b', 'bc62a1da-54b5-4aa4-9b18-530371c05e44', '7a4f51c4-76ae-478d-a8e4-ec1c3749b999'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'e97b56d3796f42fa89023ef8691f08ad', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumesActionsTest-Volume-825907983', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '44c0ea9c46d14578a12440ef37ec453e', 'qos_specs': None, 'reservations': ['8593249f-303c-48ae-bc1d-dd163db6cbf5', '1ee0e36c-dfea-4a28-90e5-938c2128842b', 'bc62a1da-54b5-4aa4-9b18-530371c05e44', '7a4f51c4-76ae-478d-a8e4-ec1c3749b999'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'e97b56d3796f42fa89023ef8691f08ad', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': 'e97b56d3796f42fa89023ef8691f08ad', 'metadata': {}, 'qos_specs': None, 'retry': {'num_attempts': 2, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']}}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(f920a5f7-106a-4092-b193-c3d340118d77),volume_id=f920a5f7-106a-4092-b193-c3d340118d77,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:38:07Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesActionsTest-Volume-825907983',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=f920a5f7-106a-4092-b193-c3d340118d77,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='44c0ea9c46d14578a12440ef37ec453e',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:40:26Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:40:26Z,use_quota=True,user_id='e97b56d3796f42fa89023ef8691f08ad',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 16:42:43 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:38:07Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesActionsTest-Volume-825907983',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=f920a5f7-106a-4092-b193-c3d340118d77,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='44c0ea9c46d14578a12440ef37ec453e',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:40:26Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:40:26Z,use_quota=True,user_id='e97b56d3796f42fa89023ef8691f08ad',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:38:07Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesActionsTest-Volume-825907983',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=f920a5f7-106a-4092-b193-c3d340118d77,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='44c0ea9c46d14578a12440ef37ec453e',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:40:26Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:40:26Z,use_quota=True,user_id='e97b56d3796f42fa89023ef8691f08ad',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 16:42:43 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 16:42:43 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 16:42:43 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:42:43 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 16:42:43 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:42:43 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 16:42:43 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:42:43 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 16:42:43 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:42:43 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 16:42:43 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:42:43 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 16:42:43 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:42:43 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 16:42:43 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:42:43 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 16:42:43 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:42:43 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:42:43 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 16:42:43 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 16:42:43 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (6568efaf-57eb-4659-a8a2-3291bfcaa4ac) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:42:43 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (6568efaf-57eb-4659-a8a2-3291bfcaa4ac) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:42:43 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (cc35a5d8-cee0-4c86-ae1f-3b0b0f83492c) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:42:43 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (cc35a5d8-cee0-4c86-ae1f-3b0b0f83492c) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:42:43 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (a1c8bf23-b7ce-4c2e-bf59-859797ad48f0) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:42:43 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (a1c8bf23-b7ce-4c2e-bf59-859797ad48f0) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:42:43 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (15105c46-2980-4697-b200-2ce244328fd2) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:42:43 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Updating volume f920a5f7-106a-4092-b193-c3d340118d77 with {'scheduled_at': datetime.datetime(2023, 4, 17, 16, 42, 43, 110371), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 16:42:43 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Volume f920a5f7-106a-4092-b193-c3d340118d77: re-scheduling SchedulerAPI.create_volume attempt 2 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 16:42:43 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Volume f920a5f7-106a-4092-b193-c3d340118d77: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 16:42:43 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] deleating volume v_f920a5f7-106a-4092-b193-c3d340118d77 {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 16:42:43 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] get snapshots for volume v_f920a5f7-106a-4092-b193-c3d340118d77 {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 16:42:43 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_f920a5f7-106a-4092-b193-c3d340118d77/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:42:43 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_f920a5f7-106a-4092-b193-c3d340118d77/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:42:43 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Volume reschedule parameters: True retry: {'num_attempts': 3, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 16:42:43 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Flow 'volume_create_manager' (bde4cd1d-b104-4629-990f-5accaf45c8e8) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 16:42:43 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (66f52aa1-87b6-47fd-8d9b-b28646ccc8ce) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:42:43 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (66f52aa1-87b6-47fd-8d9b-b28646ccc8ce) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:38:07Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesActionsTest-Volume-825907983',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=f920a5f7-106a-4092-b193-c3d340118d77,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='44c0ea9c46d14578a12440ef37ec453e',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:42:43Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:42:43Z,use_quota=True,user_id='e97b56d3796f42fa89023ef8691f08ad',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:42:43 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (784d7840-ff0b-426b-aa4d-7634f03894a5) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:42:43 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (784d7840-ff0b-426b-aa4d-7634f03894a5) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:42:43 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (951e8ba9-99a3-4c30-a909-fe02197ae449) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:42:43 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (951e8ba9-99a3-4c30-a909-fe02197ae449) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': 'f920a5f7-106a-4092-b193-c3d340118d77', 'volume_name': 'volume-f920a5f7-106a-4092-b193-c3d340118d77', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:42:43 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (0672b2d8-db76-4065-a5e3-885ffb35f94b) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:42:43 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (0672b2d8-db76-4065-a5e3-885ffb35f94b) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:42:43 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (a8db7261-afdd-48cf-9527-6528ab056a2c) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:42:43 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Volume f920a5f7-106a-4092-b193-c3d340118d77: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-f920a5f7-106a-4092-b193-c3d340118d77', 'volume_size': 1} Apr 17 16:42:43 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] creating volume v_f920a5f7-106a-4092-b193-c3d340118d77. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 16:42:43 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] create volume {'name': 'v_f920a5f7-106a-4092-b193-c3d340118d77', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 16:42:43 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:42:43 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:42:45 user cinder-volume[84696]: DEBUG cinder.utils [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.748(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:42:45 user cinder-volume[84696]: DEBUG cinder.utils [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_f920a5f7-106a-4092-b193-c3d340118d77/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:42:45 user cinder-volume[84696]: DEBUG cinder.utils [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.583(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:42:45 user cinder-volume[84696]: DEBUG cinder.utils [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:42:48 user cinder-volume[84696]: DEBUG cinder.utils [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.820(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:42:48 user cinder-volume[84696]: DEBUG cinder.utils [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_f920a5f7-106a-4092-b193-c3d340118d77/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:42:48 user cinder-volume[84696]: DEBUG cinder.utils [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.654(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:42:48 user cinder-volume[84696]: DEBUG cinder.utils [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:42:55 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:INSERT=2,cinder:UPDATE=1,cinder:SELECT=3 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:42:55 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=11,cinder:UPDATE=4 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:42:55 user cinder-volume[84696]: DEBUG cinder.utils [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 11.964(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:42:55 user cinder-volume[84696]: DEBUG cinder.utils [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_f920a5f7-106a-4092-b193-c3d340118d77/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:42:55 user cinder-volume[84696]: DEBUG cinder.utils [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 71.063(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:42:55 user cinder-volume[84696]: DEBUG cinder.utils [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_e1bb78ac-29e3-46df-9bbe-3b152e39b69a/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:42:55 user cinder-volume[84696]: DEBUG cinder.utils [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 11.802(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:42:55 user cinder-volume[84696]: DEBUG cinder.utils [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:42:58 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 136.983(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:42:58 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:42:58 user cinder-volume[84696]: ERROR oslo_service.periodic_task [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Error during VolumeManager.publish_service_capabilities: TypeError: 'NoneType' object is not subscriptable Apr 17 16:42:58 user cinder-volume[84696]: ERROR oslo_service.periodic_task Traceback (most recent call last): Apr 17 16:42:58 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/usr/local/lib/python3.10/dist-packages/oslo_service/periodic_task.py", line 216, in run_periodic_tasks Apr 17 16:42:58 user cinder-volume[84696]: ERROR oslo_service.periodic_task task(self, context) Apr 17 16:42:58 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/manager.py", line 2876, in publish_service_capabilities Apr 17 16:42:58 user cinder-volume[84696]: ERROR oslo_service.periodic_task self._report_driver_status(context) Apr 17 16:42:58 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/manager.py", line 2755, in _report_driver_status Apr 17 16:42:58 user cinder-volume[84696]: ERROR oslo_service.periodic_task volume_stats = get_stats() Apr 17 16:42:58 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/usr/local/lib/python3.10/dist-packages/oslo_utils/timeutils.py", line 319, in wrapper Apr 17 16:42:58 user cinder-volume[84696]: ERROR oslo_service.periodic_task result = func(*args, **kwargs) Apr 17 16:42:58 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/manager.py", line 2753, in get_stats Apr 17 16:42:58 user cinder-volume[84696]: ERROR oslo_service.periodic_task return self.driver.get_volume_stats(refresh=True) Apr 17 16:42:58 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/driver.py", line 666, in get_volume_stats Apr 17 16:42:58 user cinder-volume[84696]: ERROR oslo_service.periodic_task self._update_volume_stats() Apr 17 16:42:58 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 726, in _update_volume_stats Apr 17 16:42:58 user cinder-volume[84696]: ERROR oslo_service.periodic_task pool_stats = self.ra.get_pool_stats() Apr 17 16:42:58 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 897, in get_pool_stats Apr 17 16:42:58 user cinder-volume[84696]: ERROR oslo_service.periodic_task if not resp["error"] and resp["code"] == 200: Apr 17 16:42:58 user cinder-volume[84696]: ERROR oslo_service.periodic_task TypeError: 'NoneType' object is not subscriptable Apr 17 16:42:58 user cinder-volume[84696]: ERROR oslo_service.periodic_task Apr 17 16:42:58 user cinder-volume[84696]: WARNING oslo.service.loopingcall [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Function 'cinder.service.Service.periodic_tasks' run outlasted interval by 76.99 sec Apr 17 16:42:58 user cinder-volume[84696]: DEBUG oslo_service.periodic_task [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Running periodic task VolumeManager.publish_service_capabilities {{(pid=84696) run_periodic_tasks /usr/local/lib/python3.10/dist-packages/oslo_service/periodic_task.py:210}} Apr 17 16:42:58 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Updating volume stats {{(pid=84696) _update_volume_stats /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:724}} Apr 17 16:42:58 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Get pool Pool-0 fsprops {{(pid=84696) get_pool_stats /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:894}} Apr 17 16:42:58 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:42:58 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:43:01 user cinder-volume[84696]: DEBUG cinder.utils [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 41.413(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:43:01 user cinder-volume[84696]: DEBUG cinder.utils [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_b3e2b070-7732-4096-b59e-bdd77e430bbf/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:43:01 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 3.069(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:43:01 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:43:01 user cinder-volume[84696]: DEBUG cinder.utils [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 41.221(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:43:01 user cinder-volume[84696]: DEBUG cinder.utils [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:43:04 user cinder-volume[84696]: DEBUG cinder.utils [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.180(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:43:04 user cinder-volume[84696]: DEBUG cinder.utils [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_f920a5f7-106a-4092-b193-c3d340118d77/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:43:04 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.140(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:43:04 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:43:04 user cinder-volume[84696]: DEBUG cinder.utils [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.015(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:43:04 user cinder-volume[84696]: DEBUG cinder.utils [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:43:10 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.603(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:43:10 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:43:15 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=1 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:43:20 user cinder-volume[84696]: DEBUG cinder.utils [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 37.556(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:43:20 user cinder-volume[84696]: DEBUG cinder.utils [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:43:20 user cinder-volume[84696]: DEBUG cinder.utils [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 37.725(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:43:20 user cinder-volume[84696]: DEBUG cinder.utils [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_f920a5f7-106a-4092-b193-c3d340118d77/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:43:20 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 22.685(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:43:20 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:43:33 user cinder-volume[84696]: DEBUG cinder.utils [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 74.021(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:43:33 user cinder-volume[84696]: DEBUG cinder.utils [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_b3e2b070-7732-4096-b59e-bdd77e430bbf/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:43:33 user cinder-volume[84696]: DEBUG cinder.utils [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 73.827(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:43:33 user cinder-volume[84696]: DEBUG cinder.utils [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:43:35 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=12,cinder:UPDATE=4 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:43:37 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.611(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:43:37 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:43:54 user cinder-volume[84696]: DEBUG cinder.utils [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 71.573(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:43:54 user cinder-volume[84696]: DEBUG cinder.utils [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:43:54 user cinder-volume[84696]: DEBUG cinder.utils [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 71.741(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:43:54 user cinder-volume[84696]: DEBUG cinder.utils [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_f920a5f7-106a-4092-b193-c3d340118d77/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:43:55 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=4,cinder:UPDATE=1 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:44:01 user cinder-volume[84696]: DEBUG cinder.utils [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 136.983(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:44:01 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_e1bb78ac-29e3-46df-9bbe-3b152e39b69a/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:44:01 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (f379724d-30b2-4852-80c5-5786ff0b6bc2) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 16:44:01 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (2023bd15-4715-4c95-ad2a-d93c87115526) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:44:01 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (2023bd15-4715-4c95-ad2a-d93c87115526) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:44:01 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Flow 'volume_create_manager' (d9823830-d63b-46a9-9152-d096e7f2e564) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 16:44:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-26b6ca51-cba9-4420-975a-eec85c094fd9 tempest-VolumesMetadataTest-779181867 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 16:44:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 16:44:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 16:44:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 16:44:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 16:44:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 16:44:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 16:44:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 16:44:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 16:44:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 16:44:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 16:44:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 16:44:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 16:44:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 16:44:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 16:44:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 16:44:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 16:44:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 16:44:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 16:44:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 16:44:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 16:44:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 16:44:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 16:44:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 16:44:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:44:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 16:44:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:44:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 16:44:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:44:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 16:44:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:44:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 16:44:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:44:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 16:44:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:44:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 16:44:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:44:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 16:44:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:44:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:44:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 16:44:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 16:44:05 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=7,cinder:UPDATE=2 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:44:10 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 72.635(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:44:10 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:44:25 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:UPDATE=1,cinder:SELECT=1 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:44:35 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=4,cinder:UPDATE=1 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:44:38 user cinder-volume[84696]: DEBUG cinder.utils [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 139.013(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:44:38 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_b3e2b070-7732-4096-b59e-bdd77e430bbf/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:44:38 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (34400a9b-2f05-4ca4-8a6e-2772e530ef0a) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 16:44:38 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (b4c06049-69e3-48da-8917-f867b0e37fe6) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:44:38 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (b4c06049-69e3-48da-8917-f867b0e37fe6) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:44:38 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Flow 'volume_create_manager' (c651d0af-8230-4a3d-8d42-a50cbdec74f4) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 16:44:38 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 16:44:38 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 16:44:38 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 16:44:38 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 16:44:38 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 16:44:38 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 16:44:38 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 16:44:38 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 16:44:38 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 16:44:38 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 16:44:38 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 16:44:38 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 16:44:38 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 16:44:38 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 16:44:38 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 16:44:38 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 16:44:38 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 16:44:38 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 16:44:38 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 16:44:38 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 16:44:38 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 16:44:38 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 16:44:38 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 16:44:38 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 16:44:38 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:44:38 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 16:44:38 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:44:38 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 16:44:38 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:44:38 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 16:44:38 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:44:38 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 16:44:38 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:44:38 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 16:44:38 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:44:38 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 16:44:38 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:44:38 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 16:44:38 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:44:38 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:44:38 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 16:44:38 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 16:44:38 user cinder-volume[84696]: DEBUG cinder.utils [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 138.836(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:44:38 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:44:38 user cinder-volume[84696]: INFO cinder.message.api [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Creating message record for request_id = req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f Apr 17 16:44:38 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 16:44:38 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (d537c8cd-a546-48ba-9022-eb6d386d101e) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 16:44:38 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 16:44:38 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:40:02Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-SnapshotDataIntegrityTests-volume-609702161',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=b3e2b070-7732-4096-b59e-bdd77e430bbf,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='2a9277ba8f5f4f9abcbb70ee01b22d41',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:42:20Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:42:20Z,use_quota=True,user_id='d8c0fb3f72494a489e0230e83a85224a',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 16:44:38 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:40:02Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-SnapshotDataIntegrityTests-volume-609702161',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=b3e2b070-7732-4096-b59e-bdd77e430bbf,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='2a9277ba8f5f4f9abcbb70ee01b22d41',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:42:20Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:42:20Z,use_quota=True,user_id='d8c0fb3f72494a489e0230e83a85224a',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(b3e2b070-7732-4096-b59e-bdd77e430bbf),volume_id=b3e2b070-7732-4096-b59e-bdd77e430bbf,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': 'b3e2b070-7732-4096-b59e-bdd77e430bbf', 'volume_name': 'volume-b3e2b070-7732-4096-b59e-bdd77e430bbf', 'volume_size': 1}} Apr 17 16:44:38 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': 'b3e2b070-7732-4096-b59e-bdd77e430bbf', 'volume': {'id': 'b3e2b070-7732-4096-b59e-bdd77e430bbf', '_name_id': None, 'ec2_id': None, 'user_id': 'd8c0fb3f72494a489e0230e83a85224a', 'project_id': '2a9277ba8f5f4f9abcbb70ee01b22d41', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-SnapshotDataIntegrityTests-volume-609702161', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'admin_metadata': {}, 'glance_metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T16:40:01.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-b3e2b070-7732-4096-b59e-bdd77e430bbf', 'name_id': 'b3e2b070-7732-4096-b59e-bdd77e430bbf', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-SnapshotDataIntegrityTests-volume-609702161', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '2a9277ba8f5f4f9abcbb70ee01b22d41', 'qos_specs': None, 'reservations': ['9c418056-b486-4dc9-adfa-64bbbb03ad8f', '0276aa33-a2e6-40f5-a1a7-4d1f5e3cc072', '14ca4b2e-e473-45b8-afb7-461c0aa7879f', '446dfc86-c241-4830-b2f2-a41012a6f708'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'd8c0fb3f72494a489e0230e83a85224a', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-SnapshotDataIntegrityTests-volume-609702161', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '2a9277ba8f5f4f9abcbb70ee01b22d41', 'qos_specs': None, 'reservations': ['9c418056-b486-4dc9-adfa-64bbbb03ad8f', '0276aa33-a2e6-40f5-a1a7-4d1f5e3cc072', '14ca4b2e-e473-45b8-afb7-461c0aa7879f', '446dfc86-c241-4830-b2f2-a41012a6f708'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'd8c0fb3f72494a489e0230e83a85224a', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': 'd8c0fb3f72494a489e0230e83a85224a', 'metadata': {}, 'qos_specs': None, 'retry': {'num_attempts': 2, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']}}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(b3e2b070-7732-4096-b59e-bdd77e430bbf),volume_id=b3e2b070-7732-4096-b59e-bdd77e430bbf,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:40:02Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-SnapshotDataIntegrityTests-volume-609702161',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=b3e2b070-7732-4096-b59e-bdd77e430bbf,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='2a9277ba8f5f4f9abcbb70ee01b22d41',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:42:20Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:42:20Z,use_quota=True,user_id='d8c0fb3f72494a489e0230e83a85224a',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 16:44:38 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:40:02Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-SnapshotDataIntegrityTests-volume-609702161',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=b3e2b070-7732-4096-b59e-bdd77e430bbf,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='2a9277ba8f5f4f9abcbb70ee01b22d41',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:42:20Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:42:20Z,use_quota=True,user_id='d8c0fb3f72494a489e0230e83a85224a',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:40:02Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-SnapshotDataIntegrityTests-volume-609702161',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=b3e2b070-7732-4096-b59e-bdd77e430bbf,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='2a9277ba8f5f4f9abcbb70ee01b22d41',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:42:20Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:42:20Z,use_quota=True,user_id='d8c0fb3f72494a489e0230e83a85224a',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 16:44:38 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 16:44:38 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 16:44:38 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:44:38 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 16:44:38 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:44:38 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 16:44:38 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:44:38 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 16:44:38 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:44:38 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 16:44:38 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:44:38 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 16:44:38 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:44:38 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 16:44:38 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:44:38 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 16:44:38 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:44:38 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:44:38 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 16:44:38 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 16:44:38 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (d537c8cd-a546-48ba-9022-eb6d386d101e) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:44:38 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (d537c8cd-a546-48ba-9022-eb6d386d101e) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:44:38 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (7b5704ce-896b-47e6-9281-526f07035dbf) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:44:38 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (7b5704ce-896b-47e6-9281-526f07035dbf) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:44:38 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (8314f1b3-6581-4d06-a7cc-7b970b0a7e10) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:44:38 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (8314f1b3-6581-4d06-a7cc-7b970b0a7e10) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:44:38 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (074dded1-3aa3-4be8-9800-ea54911c3cb6) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:44:38 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Updating volume b3e2b070-7732-4096-b59e-bdd77e430bbf with {'scheduled_at': datetime.datetime(2023, 4, 17, 16, 44, 38, 931601), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 16:44:38 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Volume b3e2b070-7732-4096-b59e-bdd77e430bbf: re-scheduling SchedulerAPI.create_volume attempt 2 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 16:44:38 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Volume b3e2b070-7732-4096-b59e-bdd77e430bbf: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 16:44:38 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] deleating volume v_b3e2b070-7732-4096-b59e-bdd77e430bbf {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 16:44:38 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] get snapshots for volume v_b3e2b070-7732-4096-b59e-bdd77e430bbf {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 16:44:38 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_b3e2b070-7732-4096-b59e-bdd77e430bbf/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:44:38 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_b3e2b070-7732-4096-b59e-bdd77e430bbf/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:44:39 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Volume reschedule parameters: True retry: {'num_attempts': 3, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']} {{(pid=84696) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}} Apr 17 16:44:39 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Flow 'volume_create_manager' (8576d1dc-69ef-46c6-b1dd-aa6f2a0c77d6) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _flow_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:141}} Apr 17 16:44:39 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (a10e0470-5cb0-431d-a567-76bfc34ab6ab) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:44:39 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (a10e0470-5cb0-431d-a567-76bfc34ab6ab) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:40:02Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-SnapshotDataIntegrityTests-volume-609702161',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=b3e2b070-7732-4096-b59e-bdd77e430bbf,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='2a9277ba8f5f4f9abcbb70ee01b22d41',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:44:39Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:44:39Z,use_quota=True,user_id='d8c0fb3f72494a489e0230e83a85224a',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:44:39 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (4f0b539b-2c26-46f8-8cac-1644024d0987) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:44:39 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (4f0b539b-2c26-46f8-8cac-1644024d0987) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:44:39 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (023b7fd2-9c3d-4a47-b2f8-44b8f531b549) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:44:39 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (023b7fd2-9c3d-4a47-b2f8-44b8f531b549) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': 'b3e2b070-7732-4096-b59e-bdd77e430bbf', 'volume_name': 'volume-b3e2b070-7732-4096-b59e-bdd77e430bbf', 'volume_size': 1}' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:44:39 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (3ff72ea2-fe31-47f5-9bf5-bcaf8b69e633) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:44:39 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (3ff72ea2-fe31-47f5-9bf5-bcaf8b69e633) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:178}} Apr 17 16:44:39 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (d97d05e5-8c0b-41aa-abc4-7a38594ab8cd) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:44:39 user cinder-volume[84696]: INFO cinder.volume.flows.manager.create_volume [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Volume b3e2b070-7732-4096-b59e-bdd77e430bbf: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-b3e2b070-7732-4096-b59e-bdd77e430bbf', 'volume_size': 1} Apr 17 16:44:39 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] creating volume v_b3e2b070-7732-4096-b59e-bdd77e430bbf. {{(pid=84696) create_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:145}} Apr 17 16:44:39 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] create volume {'name': 'v_b3e2b070-7732-4096-b59e-bdd77e430bbf', 'size': '1073741824', 'sparse': True, 'blocksize': '128K'} {{(pid=84696) create_lun /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:135}} Apr 17 16:44:39 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Sending pool request POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:44:39 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Sending POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:44:41 user cinder-volume[84696]: DEBUG cinder.utils [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.768(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:44:41 user cinder-volume[84696]: DEBUG cinder.utils [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:44:41 user cinder-volume[84696]: DEBUG cinder.utils [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.954(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:44:41 user cinder-volume[84696]: DEBUG cinder.utils [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_b3e2b070-7732-4096-b59e-bdd77e430bbf/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:44:44 user cinder-volume[84696]: DEBUG cinder.utils [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 5.840(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:44:44 user cinder-volume[84696]: DEBUG cinder.utils [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:44:44 user cinder-volume[84696]: DEBUG cinder.utils [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.025(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:44:44 user cinder-volume[84696]: DEBUG cinder.utils [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_b3e2b070-7732-4096-b59e-bdd77e430bbf/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:44:51 user cinder-volume[84696]: DEBUG cinder.utils [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.871(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:44:51 user cinder-volume[84696]: DEBUG cinder.utils [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_b3e2b070-7732-4096-b59e-bdd77e430bbf/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:44:51 user cinder-volume[84696]: DEBUG cinder.utils [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.689(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:44:51 user cinder-volume[84696]: DEBUG cinder.utils [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:44:55 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=10,cinder:UPDATE=4,cinder:INSERT=1 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:45:01 user cinder-volume[84696]: DEBUG cinder.utils [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 138.612(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:45:01 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:45:01 user cinder-volume[84696]: INFO cinder.message.api [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Creating message record for request_id = req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 Apr 17 16:45:01 user cinder-volume[84696]: DEBUG cinder.utils [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 22.771(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:45:01 user cinder-volume[84696]: DEBUG cinder.utils [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:45:01 user cinder-volume[84696]: DEBUG cinder.utils [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 22.955(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:45:01 user cinder-volume[84696]: DEBUG cinder.utils [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_b3e2b070-7732-4096-b59e-bdd77e430bbf/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:45:01 user cinder-volume[84696]: DEBUG cinder.utils [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 138.785(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:45:01 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_f920a5f7-106a-4092-b193-c3d340118d77/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:45:01 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (15105c46-2980-4697-b200-2ce244328fd2) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 16:45:01 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (2320cea9-ec63-478e-921b-d41314b2d5b6) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:45:01 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (2320cea9-ec63-478e-921b-d41314b2d5b6) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:45:01 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Flow 'volume_create_manager' (855299a7-9da1-451d-8700-ffbc82d4dece) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 16:45:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 16:45:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 16:45:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 16:45:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 16:45:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 16:45:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 16:45:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 16:45:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 16:45:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 16:45:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 16:45:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 16:45:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 16:45:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 16:45:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 16:45:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 16:45:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 16:45:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 16:45:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 16:45:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 16:45:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 16:45:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 16:45:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 16:45:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 16:45:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 16:45:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:45:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 16:45:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:45:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 16:45:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:45:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 16:45:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:45:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 16:45:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:45:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 16:45:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:45:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 16:45:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:45:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 16:45:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:45:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:45:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 16:45:01 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 16:45:01 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 16:45:01 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (a8db7261-afdd-48cf-9527-6528ab056a2c) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 16:45:01 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 16:45:01 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:38:07Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesActionsTest-Volume-825907983',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=f920a5f7-106a-4092-b193-c3d340118d77,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='44c0ea9c46d14578a12440ef37ec453e',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:42:43Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:42:43Z,use_quota=True,user_id='e97b56d3796f42fa89023ef8691f08ad',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 16:45:01 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:38:07Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesActionsTest-Volume-825907983',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=f920a5f7-106a-4092-b193-c3d340118d77,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='44c0ea9c46d14578a12440ef37ec453e',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:42:43Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:42:43Z,use_quota=True,user_id='e97b56d3796f42fa89023ef8691f08ad',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(f920a5f7-106a-4092-b193-c3d340118d77),volume_id=f920a5f7-106a-4092-b193-c3d340118d77,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': 'f920a5f7-106a-4092-b193-c3d340118d77', 'volume_name': 'volume-f920a5f7-106a-4092-b193-c3d340118d77', 'volume_size': 1}} Apr 17 16:45:01 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': 'f920a5f7-106a-4092-b193-c3d340118d77', 'volume': {'id': 'f920a5f7-106a-4092-b193-c3d340118d77', '_name_id': None, 'ec2_id': None, 'user_id': 'e97b56d3796f42fa89023ef8691f08ad', 'project_id': '44c0ea9c46d14578a12440ef37ec453e', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-VolumesActionsTest-Volume-825907983', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'admin_metadata': {}, 'glance_metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T16:38:07.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-f920a5f7-106a-4092-b193-c3d340118d77', 'name_id': 'f920a5f7-106a-4092-b193-c3d340118d77', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumesActionsTest-Volume-825907983', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '44c0ea9c46d14578a12440ef37ec453e', 'qos_specs': None, 'reservations': ['8593249f-303c-48ae-bc1d-dd163db6cbf5', '1ee0e36c-dfea-4a28-90e5-938c2128842b', 'bc62a1da-54b5-4aa4-9b18-530371c05e44', '7a4f51c4-76ae-478d-a8e4-ec1c3749b999'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'e97b56d3796f42fa89023ef8691f08ad', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-VolumesActionsTest-Volume-825907983', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '44c0ea9c46d14578a12440ef37ec453e', 'qos_specs': None, 'reservations': ['8593249f-303c-48ae-bc1d-dd163db6cbf5', '1ee0e36c-dfea-4a28-90e5-938c2128842b', 'bc62a1da-54b5-4aa4-9b18-530371c05e44', '7a4f51c4-76ae-478d-a8e4-ec1c3749b999'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'e97b56d3796f42fa89023ef8691f08ad', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': 'e97b56d3796f42fa89023ef8691f08ad', 'metadata': {}, 'qos_specs': None, 'retry': {'num_attempts': 3, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']}}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(f920a5f7-106a-4092-b193-c3d340118d77),volume_id=f920a5f7-106a-4092-b193-c3d340118d77,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:38:07Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesActionsTest-Volume-825907983',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=f920a5f7-106a-4092-b193-c3d340118d77,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='44c0ea9c46d14578a12440ef37ec453e',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:42:43Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:42:43Z,use_quota=True,user_id='e97b56d3796f42fa89023ef8691f08ad',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 16:45:01 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:38:07Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesActionsTest-Volume-825907983',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=f920a5f7-106a-4092-b193-c3d340118d77,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='44c0ea9c46d14578a12440ef37ec453e',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:42:43Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:42:43Z,use_quota=True,user_id='e97b56d3796f42fa89023ef8691f08ad',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:38:07Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesActionsTest-Volume-825907983',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=f920a5f7-106a-4092-b193-c3d340118d77,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='44c0ea9c46d14578a12440ef37ec453e',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:42:43Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:42:43Z,use_quota=True,user_id='e97b56d3796f42fa89023ef8691f08ad',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 16:45:01 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 16:45:01 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 16:45:01 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:45:01 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 16:45:01 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:45:01 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 16:45:01 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:45:01 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 16:45:01 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:45:01 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 16:45:01 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:45:01 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 16:45:01 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:45:01 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 16:45:01 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:45:01 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 16:45:01 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:45:01 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:45:01 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 16:45:01 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 16:45:01 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (a8db7261-afdd-48cf-9527-6528ab056a2c) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:45:01 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (a8db7261-afdd-48cf-9527-6528ab056a2c) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:45:01 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (0672b2d8-db76-4065-a5e3-885ffb35f94b) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:45:01 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (0672b2d8-db76-4065-a5e3-885ffb35f94b) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:45:01 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (951e8ba9-99a3-4c30-a909-fe02197ae449) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:45:01 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (951e8ba9-99a3-4c30-a909-fe02197ae449) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:45:02 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (784d7840-ff0b-426b-aa4d-7634f03894a5) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:45:02 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Updating volume f920a5f7-106a-4092-b193-c3d340118d77 with {'scheduled_at': datetime.datetime(2023, 4, 17, 16, 45, 2, 1027), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 16:45:02 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Volume f920a5f7-106a-4092-b193-c3d340118d77: re-scheduling SchedulerAPI.create_volume attempt 3 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 16:45:02 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Volume f920a5f7-106a-4092-b193-c3d340118d77: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 16:45:02 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] deleating volume v_f920a5f7-106a-4092-b193-c3d340118d77 {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 16:45:02 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] get snapshots for volume v_f920a5f7-106a-4092-b193-c3d340118d77 {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 16:45:02 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_f920a5f7-106a-4092-b193-c3d340118d77/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:45:02 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_f920a5f7-106a-4092-b193-c3d340118d77/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:45:04 user cinder-volume[84696]: DEBUG cinder.utils [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.959(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:45:04 user cinder-volume[84696]: DEBUG cinder.utils [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_f920a5f7-106a-4092-b193-c3d340118d77/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:45:05 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:INSERT=3,cinder:UPDATE=4,cinder:SELECT=13 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:45:08 user cinder-volume[84696]: DEBUG cinder.utils [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.031(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:45:08 user cinder-volume[84696]: DEBUG cinder.utils [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_f920a5f7-106a-4092-b193-c3d340118d77/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:45:14 user cinder-volume[84696]: DEBUG cinder.utils [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.175(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:45:14 user cinder-volume[84696]: DEBUG cinder.utils [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_f920a5f7-106a-4092-b193-c3d340118d77/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:45:17 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 139.099(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:45:17 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:45:17 user cinder-volume[84696]: ERROR oslo_service.periodic_task [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Error during VolumeManager.publish_service_capabilities: TypeError: 'NoneType' object is not subscriptable Apr 17 16:45:17 user cinder-volume[84696]: ERROR oslo_service.periodic_task Traceback (most recent call last): Apr 17 16:45:17 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/usr/local/lib/python3.10/dist-packages/oslo_service/periodic_task.py", line 216, in run_periodic_tasks Apr 17 16:45:17 user cinder-volume[84696]: ERROR oslo_service.periodic_task task(self, context) Apr 17 16:45:17 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/manager.py", line 2876, in publish_service_capabilities Apr 17 16:45:17 user cinder-volume[84696]: ERROR oslo_service.periodic_task self._report_driver_status(context) Apr 17 16:45:17 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/manager.py", line 2755, in _report_driver_status Apr 17 16:45:17 user cinder-volume[84696]: ERROR oslo_service.periodic_task volume_stats = get_stats() Apr 17 16:45:17 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/usr/local/lib/python3.10/dist-packages/oslo_utils/timeutils.py", line 319, in wrapper Apr 17 16:45:17 user cinder-volume[84696]: ERROR oslo_service.periodic_task result = func(*args, **kwargs) Apr 17 16:45:17 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/manager.py", line 2753, in get_stats Apr 17 16:45:17 user cinder-volume[84696]: ERROR oslo_service.periodic_task return self.driver.get_volume_stats(refresh=True) Apr 17 16:45:17 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/driver.py", line 666, in get_volume_stats Apr 17 16:45:17 user cinder-volume[84696]: ERROR oslo_service.periodic_task self._update_volume_stats() Apr 17 16:45:17 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 726, in _update_volume_stats Apr 17 16:45:17 user cinder-volume[84696]: ERROR oslo_service.periodic_task pool_stats = self.ra.get_pool_stats() Apr 17 16:45:17 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 897, in get_pool_stats Apr 17 16:45:17 user cinder-volume[84696]: ERROR oslo_service.periodic_task if not resp["error"] and resp["code"] == 200: Apr 17 16:45:17 user cinder-volume[84696]: ERROR oslo_service.periodic_task TypeError: 'NoneType' object is not subscriptable Apr 17 16:45:17 user cinder-volume[84696]: ERROR oslo_service.periodic_task Apr 17 16:45:17 user cinder-volume[84696]: WARNING oslo.service.loopingcall [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Function 'cinder.service.Service.periodic_tasks' run outlasted interval by 79.10 sec Apr 17 16:45:17 user cinder-volume[84696]: DEBUG oslo_service.periodic_task [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Running periodic task VolumeManager.publish_service_capabilities {{(pid=84696) run_periodic_tasks /usr/local/lib/python3.10/dist-packages/oslo_service/periodic_task.py:210}} Apr 17 16:45:17 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Updating volume stats {{(pid=84696) _update_volume_stats /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:724}} Apr 17 16:45:17 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Get pool Pool-0 fsprops {{(pid=84696) get_pool_stats /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:894}} Apr 17 16:45:17 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:45:17 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:45:20 user cinder-volume[84696]: DEBUG cinder.utils [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 41.200(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:45:20 user cinder-volume[84696]: DEBUG cinder.utils [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:45:20 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 3.069(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:45:20 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:45:20 user cinder-volume[84696]: DEBUG cinder.utils [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 41.386(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:45:20 user cinder-volume[84696]: DEBUG cinder.utils [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_b3e2b070-7732-4096-b59e-bdd77e430bbf/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:45:23 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.140(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:45:23 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:45:23 user cinder-volume[84696]: DEBUG cinder.utils [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.393(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:45:23 user cinder-volume[84696]: DEBUG cinder.utils [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_f920a5f7-106a-4092-b193-c3d340118d77/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:45:25 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=1,cinder:UPDATE=1 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:45:29 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.605(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:45:29 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:45:35 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=8,cinder:UPDATE=3 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:45:39 user cinder-volume[84696]: DEBUG cinder.utils [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 37.935(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:45:39 user cinder-volume[84696]: DEBUG cinder.utils [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_f920a5f7-106a-4092-b193-c3d340118d77/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:45:39 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 22.686(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:45:39 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:45:45 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=3,cinder:UPDATE=1 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:45:52 user cinder-volume[84696]: DEBUG cinder.utils [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 73.808(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:45:52 user cinder-volume[84696]: DEBUG cinder.utils [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:45:52 user cinder-volume[84696]: DEBUG cinder.utils [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 73.994(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:45:52 user cinder-volume[84696]: DEBUG cinder.utils [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_b3e2b070-7732-4096-b59e-bdd77e430bbf/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:45:55 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=3,cinder:UPDATE=1 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:45:56 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.612(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:45:56 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:46:13 user cinder-volume[84696]: DEBUG cinder.utils [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 71.951(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:46:13 user cinder-volume[84696]: DEBUG cinder.utils [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_f920a5f7-106a-4092-b193-c3d340118d77/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:46:15 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=4,cinder:UPDATE=1 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:46:25 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=6,cinder:UPDATE=2 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:46:30 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 73.628(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:46:30 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:46:35 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=2,cinder:UPDATE=1 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:46:55 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=5,cinder:UPDATE=1 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:46:58 user cinder-volume[84696]: DEBUG cinder.utils [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 139.792(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:46:58 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Geting None from POST to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:46:58 user cinder-volume[84696]: INFO cinder.message.api [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Creating message record for request_id = req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f Apr 17 16:46:58 user cinder-volume[84696]: DEBUG cinder.utils [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 139.981(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:46:58 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_b3e2b070-7732-4096-b59e-bdd77e430bbf/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:46:58 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (074dded1-3aa3-4be8-9800-ea54911c3cb6) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 16:46:58 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (ae61ed63-eb54-46d0-ad92-2dee46564540) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:46:58 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (ae61ed63-eb54-46d0-ad92-2dee46564540) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:46:58 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Flow 'volume_create_manager' (97d319ba-952c-4da9-8b89-36a4edc1926f) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 16:46:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 16:46:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 16:46:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 16:46:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 16:46:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 16:46:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 16:46:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 16:46:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 16:46:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 16:46:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 16:46:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 16:46:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 16:46:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 16:46:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 16:46:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 16:46:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 16:46:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 16:46:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 16:46:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 16:46:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 16:46:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 16:46:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 16:46:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 16:46:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 16:46:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:46:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 16:46:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:46:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 16:46:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:46:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 16:46:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:46:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 16:46:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:46:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 16:46:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:46:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 16:46:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:46:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 16:46:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:46:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:46:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 16:46:58 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 16:46:58 user cinder-volume[84696]: DEBUG oslo_db.sqlalchemy.engines [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=84696) _check_effective_sql_mode /usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py:335}} Apr 17 16:46:58 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (d97d05e5-8c0b-41aa-abc4-7a38594ab8cd) transitioned into state 'FAILURE' from state 'RUNNING' Apr 17 16:46:58 user cinder-volume[84696]: 5 predecessors (most recent first): Apr 17 16:46:58 user cinder-volume[84696]: Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:40:02Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-SnapshotDataIntegrityTests-volume-609702161',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=b3e2b070-7732-4096-b59e-bdd77e430bbf,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='2a9277ba8f5f4f9abcbb70ee01b22d41',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:44:39Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:44:39Z,use_quota=True,user_id='d8c0fb3f72494a489e0230e83a85224a',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': None} Apr 17 16:46:58 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:40:02Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-SnapshotDataIntegrityTests-volume-609702161',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=b3e2b070-7732-4096-b59e-bdd77e430bbf,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='2a9277ba8f5f4f9abcbb70ee01b22d41',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:44:39Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:44:39Z,use_quota=True,user_id='d8c0fb3f72494a489e0230e83a85224a',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(b3e2b070-7732-4096-b59e-bdd77e430bbf),volume_id=b3e2b070-7732-4096-b59e-bdd77e430bbf,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c))}, 'provides': {'status': 'creating', 'type': 'raw', 'volume_id': 'b3e2b070-7732-4096-b59e-bdd77e430bbf', 'volume_name': 'volume-b3e2b070-7732-4096-b59e-bdd77e430bbf', 'volume_size': 1}} Apr 17 16:46:58 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': None, 'snapshot_id': None, 'source_volid': None, 'volume_id': 'b3e2b070-7732-4096-b59e-bdd77e430bbf', 'volume': {'id': 'b3e2b070-7732-4096-b59e-bdd77e430bbf', '_name_id': None, 'ec2_id': None, 'user_id': 'd8c0fb3f72494a489e0230e83a85224a', 'project_id': '2a9277ba8f5f4f9abcbb70ee01b22d41', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-SnapshotDataIntegrityTests-volume-609702161', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'admin_metadata': {}, 'glance_metadata': {}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2023-04-17T16:40:01.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-b3e2b070-7732-4096-b59e-bdd77e430bbf', 'name_id': 'b3e2b070-7732-4096-b59e-bdd77e430bbf', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c', 'name': 'joviandss-2', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'joviandss-2'}, 'qos_specs_id': None, 'created_at': '2023-04-17T15:19:38.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-SnapshotDataIntegrityTests-volume-609702161', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '2a9277ba8f5f4f9abcbb70ee01b22d41', 'qos_specs': None, 'reservations': ['9c418056-b486-4dc9-adfa-64bbbb03ad8f', '0276aa33-a2e6-40f5-a1a7-4d1f5e3cc072', '14ca4b2e-e473-45b8-afb7-461c0aa7879f', '446dfc86-c241-4830-b2f2-a41012a6f708'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'd8c0fb3f72494a489e0230e83a85224a', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-SnapshotDataIntegrityTests-volume-609702161', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '2a9277ba8f5f4f9abcbb70ee01b22d41', 'qos_specs': None, 'reservations': ['9c418056-b486-4dc9-adfa-64bbbb03ad8f', '0276aa33-a2e6-40f5-a1a7-4d1f5e3cc072', '14ca4b2e-e473-45b8-afb7-461c0aa7879f', '446dfc86-c241-4830-b2f2-a41012a6f708'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': 'd8c0fb3f72494a489e0230e83a85224a', 'volume_type_id': '6051bd20-d4fe-472e-ac09-f3c9b04bd00c'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2023-04-17T15:19:38Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='joviandss-2'},id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c,is_public=True,name='joviandss-2',projects=[],qos_specs=,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': 'd8c0fb3f72494a489e0230e83a85224a', 'metadata': {}, 'qos_specs': None, 'retry': {'num_attempts': 3, 'backends': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2'], 'hosts': ['user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2', 'user@joviandss-2#joviandss-2']}}, 'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(b3e2b070-7732-4096-b59e-bdd77e430bbf),volume_id=b3e2b070-7732-4096-b59e-bdd77e430bbf,volume_properties=VolumeProperties,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c)), 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:40:02Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-SnapshotDataIntegrityTests-volume-609702161',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=b3e2b070-7732-4096-b59e-bdd77e430bbf,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='2a9277ba8f5f4f9abcbb70ee01b22d41',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:44:39Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:44:39Z,use_quota=True,user_id='d8c0fb3f72494a489e0230e83a85224a',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c), 'context': }, 'provides': None} Apr 17 16:46:58 user cinder-volume[84696]: |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': , 'volume': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:40:02Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-SnapshotDataIntegrityTests-volume-609702161',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=b3e2b070-7732-4096-b59e-bdd77e430bbf,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='2a9277ba8f5f4f9abcbb70ee01b22d41',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:44:39Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:44:39Z,use_quota=True,user_id='d8c0fb3f72494a489e0230e83a85224a',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)}, 'provides': Volume(_name_id=None,admin_metadata=,attach_status='detached',availability_zone='nova',bootable=False,cluster=,cluster_name=None,consistencygroup=,consistencygroup_id=None,created_at=2023-04-17T16:40:02Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-SnapshotDataIntegrityTests-volume-609702161',ec2_id=None,encryption_key_id=None,glance_metadata=,group=,group_id=None,host='user@joviandss-2#joviandss-2',id=b3e2b070-7732-4096-b59e-bdd77e430bbf,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='2a9277ba8f5f4f9abcbb70ee01b22d41',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2023-04-17T16:44:39Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=,source_volid=None,status='creating',terminated_at=None,updated_at=2023-04-17T16:44:39Z,use_quota=True,user_id='d8c0fb3f72494a489e0230e83a85224a',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(6051bd20-d4fe-472e-ac09-f3c9b04bd00c),volume_type_id=6051bd20-d4fe-472e-ac09-f3c9b04bd00c)} Apr 17 16:46:58 user cinder-volume[84696]: |__Flow 'volume_create_manager': TypeError: 'NoneType' object is not subscriptable Apr 17 16:46:58 user cinder-volume[84696]: ERROR cinder.volume.manager Traceback (most recent call last): Apr 17 16:46:58 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:46:58 user cinder-volume[84696]: ERROR cinder.volume.manager result = task.execute(**arguments) Apr 17 16:46:58 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:46:58 user cinder-volume[84696]: ERROR cinder.volume.manager model_update = self._create_raw_volume( Apr 17 16:46:58 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:46:58 user cinder-volume[84696]: ERROR cinder.volume.manager with excutils.save_and_reraise_exception(): Apr 17 16:46:58 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:46:58 user cinder-volume[84696]: ERROR cinder.volume.manager self.force_reraise() Apr 17 16:46:59 user cinder-volume[84696]: ERROR cinder.volume.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:46:59 user cinder-volume[84696]: ERROR cinder.volume.manager raise self.value Apr 17 16:46:59 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:46:59 user cinder-volume[84696]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) Apr 17 16:46:59 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:46:59 user cinder-volume[84696]: ERROR cinder.volume.manager self.ra.create_lun(vname, Apr 17 16:46:59 user cinder-volume[84696]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:46:59 user cinder-volume[84696]: ERROR cinder.volume.manager if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:46:59 user cinder-volume[84696]: ERROR cinder.volume.manager TypeError: 'NoneType' object is not subscriptable Apr 17 16:46:59 user cinder-volume[84696]: ERROR cinder.volume.manager Apr 17 16:46:59 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (d97d05e5-8c0b-41aa-abc4-7a38594ab8cd) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:46:59 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (d97d05e5-8c0b-41aa-abc4-7a38594ab8cd) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:46:59 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (3ff72ea2-fe31-47f5-9bf5-bcaf8b69e633) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:46:59 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (3ff72ea2-fe31-47f5-9bf5-bcaf8b69e633) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:46:59 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (023b7fd2-9c3d-4a47-b2f8-44b8f531b549) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:46:59 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (023b7fd2-9c3d-4a47-b2f8-44b8f531b549) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:46:59 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (4f0b539b-2c26-46f8-8cac-1644024d0987) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:46:59 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Updating volume b3e2b070-7732-4096-b59e-bdd77e430bbf with {'scheduled_at': datetime.datetime(2023, 4, 17, 16, 46, 59, 32348), 'host': None}. {{(pid=84696) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}} Apr 17 16:46:59 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Volume b3e2b070-7732-4096-b59e-bdd77e430bbf: re-scheduling SchedulerAPI.create_volume attempt 3 due to 'NoneType' object is not subscriptable {{(pid=84696) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}} Apr 17 16:46:59 user cinder-volume[84696]: DEBUG cinder.volume.flows.manager.create_volume [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Volume b3e2b070-7732-4096-b59e-bdd77e430bbf: re-scheduled {{(pid=84696) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}} Apr 17 16:46:59 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] deleating volume v_b3e2b070-7732-4096-b59e-bdd77e430bbf {{(pid=84696) delete_volume /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:266}} Apr 17 16:46:59 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] get snapshots for volume v_b3e2b070-7732-4096-b59e-bdd77e430bbf {{(pid=84696) get_snapshots /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:816}} Apr 17 16:46:59 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_b3e2b070-7732-4096-b59e-bdd77e430bbf/snapshots {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:46:59 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_b3e2b070-7732-4096-b59e-bdd77e430bbf/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:47:02 user cinder-volume[84696]: DEBUG cinder.utils [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 2.947(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:47:02 user cinder-volume[84696]: DEBUG cinder.utils [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_b3e2b070-7732-4096-b59e-bdd77e430bbf/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:47:05 user cinder-volume[84696]: DEBUG cinder.utils [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.023(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:47:05 user cinder-volume[84696]: DEBUG cinder.utils [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_b3e2b070-7732-4096-b59e-bdd77e430bbf/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:47:11 user cinder-volume[84696]: DEBUG cinder.utils [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.903(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:47:11 user cinder-volume[84696]: DEBUG cinder.utils [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_b3e2b070-7732-4096-b59e-bdd77e430bbf/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:47:15 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=4 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:47:15 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=8,cinder:UPDATE=5,cinder:INSERT=2 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:47:18 user cinder-volume[84696]: DEBUG cinder.utils [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 136.943(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:47:18 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0/volumes/v_f920a5f7-106a-4092-b193-c3d340118d77/snapshots {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:47:18 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (784d7840-ff0b-426b-aa4d-7634f03894a5) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' Apr 17 16:47:18 user cinder-volume[84696]: DEBUG cinder.volume.manager [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (66f52aa1-87b6-47fd-8d9b-b28646ccc8ce) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=84696) _task_receiver /usr/local/lib/python3.10/dist-packages/taskflow/listeners/logging.py:190}} Apr 17 16:47:18 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (66f52aa1-87b6-47fd-8d9b-b28646ccc8ce) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' Apr 17 16:47:19 user cinder-volume[84696]: WARNING cinder.volume.manager [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Flow 'volume_create_manager' (bde4cd1d-b104-4629-990f-5accaf45c8e8) transitioned into state 'REVERTED' from state 'RUNNING' Apr 17 16:47:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server [None req-a237a7e2-7b3e-4cba-90d3-8187cea21e78 tempest-VolumesActionsTest-90490071 None] Exception during message handling: TypeError: 'NoneType' object is not subscriptable Apr 17 16:47:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Apr 17 16:47:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Apr 17 16:47:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Apr 17 16:47:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Apr 17 16:47:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Apr 17 16:47:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Apr 17 16:47:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Apr 17 16:47:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun Apr 17 16:47:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) Apr 17 16:47:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Apr 17 16:47:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Apr 17 16:47:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume Apr 17 16:47:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server _run_flow() Apr 17 16:47:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow Apr 17 16:47:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server flow_engine.run() Apr 17 16:47:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Apr 17 16:47:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): Apr 17 16:47:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter Apr 17 16:47:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) Apr 17 16:47:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any Apr 17 16:47:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server failures[0].reraise() Apr 17 16:47:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/types/failure.py", line 350, in reraise Apr 17 16:47:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise value Apr 17 16:47:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task Apr 17 16:47:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments) Apr 17 16:47:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1228, in execute Apr 17 16:47:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume( Apr 17 16:47:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1191, in _create_raw_volume Apr 17 16:47:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Apr 17 16:47:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Apr 17 16:47:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.force_reraise() Apr 17 16:47:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Apr 17 16:47:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server raise self.value Apr 17 16:47:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1189, in _create_raw_volume Apr 17 16:47:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) Apr 17 16:47:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 151, in create_volume Apr 17 16:47:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server self.ra.create_lun(vname, Apr 17 16:47:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 138, in create_lun Apr 17 16:47:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server if not resp["error"] and resp["code"] in (200, 201): Apr 17 16:47:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable Apr 17 16:47:19 user cinder-volume[84696]: ERROR oslo_messaging.rpc.server Apr 17 16:47:22 user cinder-volume[84696]: DEBUG cinder.utils [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 22.979(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:47:22 user cinder-volume[84696]: DEBUG cinder.utils [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_b3e2b070-7732-4096-b59e-bdd77e430bbf/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:47:25 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=2,cinder:UPDATE=1 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:47:35 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 137.755(s), this was the 7th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:47:35 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Geting None from GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:129}} Apr 17 16:47:35 user cinder-volume[84696]: ERROR oslo_service.periodic_task [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Error during VolumeManager.publish_service_capabilities: TypeError: 'NoneType' object is not subscriptable Apr 17 16:47:35 user cinder-volume[84696]: ERROR oslo_service.periodic_task Traceback (most recent call last): Apr 17 16:47:35 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/usr/local/lib/python3.10/dist-packages/oslo_service/periodic_task.py", line 216, in run_periodic_tasks Apr 17 16:47:35 user cinder-volume[84696]: ERROR oslo_service.periodic_task task(self, context) Apr 17 16:47:35 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/manager.py", line 2876, in publish_service_capabilities Apr 17 16:47:35 user cinder-volume[84696]: ERROR oslo_service.periodic_task self._report_driver_status(context) Apr 17 16:47:35 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/manager.py", line 2755, in _report_driver_status Apr 17 16:47:35 user cinder-volume[84696]: ERROR oslo_service.periodic_task volume_stats = get_stats() Apr 17 16:47:35 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/usr/local/lib/python3.10/dist-packages/oslo_utils/timeutils.py", line 319, in wrapper Apr 17 16:47:35 user cinder-volume[84696]: ERROR oslo_service.periodic_task result = func(*args, **kwargs) Apr 17 16:47:35 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/manager.py", line 2753, in get_stats Apr 17 16:47:35 user cinder-volume[84696]: ERROR oslo_service.periodic_task return self.driver.get_volume_stats(refresh=True) Apr 17 16:47:35 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/driver.py", line 666, in get_volume_stats Apr 17 16:47:35 user cinder-volume[84696]: ERROR oslo_service.periodic_task self._update_volume_stats() Apr 17 16:47:35 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py", line 726, in _update_volume_stats Apr 17 16:47:35 user cinder-volume[84696]: ERROR oslo_service.periodic_task pool_stats = self.ra.get_pool_stats() Apr 17 16:47:35 user cinder-volume[84696]: ERROR oslo_service.periodic_task File "/opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py", line 897, in get_pool_stats Apr 17 16:47:35 user cinder-volume[84696]: ERROR oslo_service.periodic_task if not resp["error"] and resp["code"] == 200: Apr 17 16:47:35 user cinder-volume[84696]: ERROR oslo_service.periodic_task TypeError: 'NoneType' object is not subscriptable Apr 17 16:47:35 user cinder-volume[84696]: ERROR oslo_service.periodic_task Apr 17 16:47:35 user cinder-volume[84696]: WARNING oslo.service.loopingcall [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Function 'cinder.service.Service.periodic_tasks' run outlasted interval by 77.76 sec Apr 17 16:47:35 user cinder-volume[84696]: DEBUG oslo_service.periodic_task [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Running periodic task VolumeManager.publish_service_capabilities {{(pid=84696) run_periodic_tasks /usr/local/lib/python3.10/dist-packages/oslo_service/periodic_task.py:210}} Apr 17 16:47:35 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.iscsi [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Updating volume stats {{(pid=84696) _update_volume_stats /opt/stack/cinder/cinder/volume/drivers/open_e/iscsi.py:724}} Apr 17 16:47:35 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Get pool Pool-0 fsprops {{(pid=84696) get_pool_stats /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest.py:894}} Apr 17 16:47:35 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Sending pool request GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) pool_request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:142}} Apr 17 16:47:35 user cinder-volume[84696]: DEBUG cinder.volume.drivers.open_e.jovian_common.rest_proxy [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Sending GET to https://172.16.0.220:82/api/v3/pools/Pool-0 {{(pid=84696) request /opt/stack/cinder/cinder/volume/drivers/open_e/jovian_common/rest_proxy.py:112}} Apr 17 16:47:35 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=3,cinder:UPDATE=1 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:47:38 user cinder-volume[84696]: DEBUG cinder.utils [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.043(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:47:38 user cinder-volume[84696]: DEBUG cinder.utils [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_b3e2b070-7732-4096-b59e-bdd77e430bbf/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:47:38 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 3.069(s), this was the 1st time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:47:38 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 2.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:47:41 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 6.140(s), this was the 2nd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:47:41 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 4.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:47:47 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 12.284(s), this was the 3rd time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:47:47 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 8.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:47:55 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=1 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:47:55 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=6,cinder:UPDATE=2 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:47:56 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 21.948(s), this was the 4th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:47:56 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 16.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:48:11 user cinder-volume[84696]: DEBUG cinder.utils [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 71.939(s), this was the 6th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:48:11 user cinder-volume[84696]: DEBUG cinder.utils [None req-47547a7c-26c7-42c9-a7cd-c9d95e4ea81f tempest-SnapshotDataIntegrityTests-793909213 None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 64.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0/volumes/v_b3e2b070-7732-4096-b59e-bdd77e430bbf/snapshots (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:48:14 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Finished call to 'cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send' after 39.036(s), this was the 5th time calling it. {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/after.py:30}} Apr 17 16:48:14 user cinder-volume[84696]: DEBUG cinder.utils [None req-5f47039d-c7f2-4061-86af-f2b2e2a3d3c7 None None] Retrying cinder.volume.drivers.open_e.jovian_common.rest_proxy.JovianRESTProxy._send in 32.0 seconds as it raised ConnectionError: HTTPSConnectionPool(host='172.16.0.220', port=82): Max retries exceeded with url: /api/v3/pools/Pool-0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] EHOSTUNREACH')). {{(pid=84696) log_it /usr/local/lib/python3.10/dist-packages/tenacity/before_sleep.py:40}} Apr 17 16:48:15 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=1 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}} Apr 17 16:48:25 user cinder-volume[84696]: DEBUG dbcounter [-] [84696] Writing DB stats cinder:SELECT=8,cinder:UPDATE=3 {{(pid=84696) stat_writer /usr/local/lib/python3.10/dist-packages/dbcounter.py:117}}